Reinforcement learning-based mobile robot navigation control method and system, and storage medium

Abstract

The application discloses a kind of mobile robot navigation control method, system and storage medium based on reinforcement learning, the actual environment of robot movement is discretized as grid map, on this basis, the reinforcement learning model of robot is constructed for navigation;Estimate the change rule of obstacle in environment, in the process of obtaining optimal strategy using reinforcement learning, the probability of each obstacle changes in different directions is obtained, the cumulative return of relearning under the condition of any obstacle combination change at any time and the cumulative return of maintaining original strategy are calculated;The relearning benefit of any time is calculated, and the relearning cost is obtained, the relearning net benefit of any time is obtained, and the best time for relearning is determined according to the obtained relearning net benefit. Avoid each time environment changes to re-perform once reinforcement learning to obtain the optimal strategy after the current change environment, the safety risk brought by relearning at each moment can be reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of reinforcement learning, and relates to a mobile robot navigation control method and system based on reinforcement learning and a storage medium. BACKGROUND

[0002] Reinforcement learning can automatically find an optimal control strategy from active interaction with an environment, and therefore has shown good performance and great potential in the field of mobile robot navigation in recent years. However, reinforcement learning faces a learning safety problem in actual robot deployment, that is, reinforcement learning essentially obtains an optimal control strategy through continuous trial and error of action behavior, that is, when reinforcement learning is actually deployed on a mobile robot system, the robot will explore various action behaviors, including safe and unsafe ones, and the unsafe action behaviors may cause serious consequences to the robot, such as damaging the robot itself. Therefore, the actual deployment of reinforcement learning actually puts higher requirements on the action safety of reinforcement learning.

[0003] Although the safe reinforcement learning method and technology proposed in recent years have gradually begun to improve the learning safety of reinforcement learning (such as patent CN115930968A mobile robot navigation method and system based on small sample reasoning type safe reinforcement learning), most of them consider static or known environments, and the effect on dynamic environments still needs to be improved. Specifically, in long-term navigation in a dynamic environment, some of these methods will be ineffective, and although some can still be used, their performance will be lower than in a static environment, and there is an important default condition for the application of these methods, that is, each time the environment changes, a complete reinforcement learning is performed again to obtain the optimal strategy after the current changed environment. However, as long as reinforcement learning is performed, there is a risk of executing unsafe actions, that is, reinforcement learning actually has a learning safety cost, and if the change of the dynamic environment has little or no effect on the currently used optimal strategy, the unsafe risk brought by relearning far exceeds the possible performance improvement, that is, the disadvantages far outweigh the advantages, and it is actually safer not to perform reinforcement learning again. Therefore, determining whether to perform a new round of reinforcement learning at a certain moment in a dynamic environment and when to perform a new round of reinforcement learning is of great significance to the safe deployment of reinforcement learning in actual scenarios. SUMMARY

[0004] The application aims to provide a mobile robot navigation control method and system based on reinforcement learning and a storage medium, which, in the navigation of a mobile robot in a dynamic environment, considers not only the benefits of learning but also the safety cost of learning when the environment changes, and determines whether to perform a new round of reinforcement learning according to the net benefits of learning.

[0005] The technical solution for achieving the object of the present application is:

[0006] A mobile robot navigation control method based on reinforcement learning, comprising the following steps:

[0007] S01: Discretize the actual environment of robot movement into a grid map, and build a reinforcement learning model of the robot for navigation on this basis;

[0008] S02: Estimate the change rule of obstacles in the environment, obtain the probability of each obstacle changing in different directions in the process of obtaining the optimal strategy by using reinforcement learning, and calculate the cumulative return of relearning and the cumulative return of maintaining the original strategy under the change condition of any obstacle combination at any time;

[0009] S03: Calculate the relearning benefit at any time, and the relearning cost, obtain the relearning net benefit at any time, and determine the best time for relearning according to the obtained relearning net benefit.

[0010] In the preferred technical solution, the probability of each obstacle changing in different directions obtained in step S02 comprises:

[0011] counting the number of changes of each obstacle in different directions in the process of obtaining the optimal strategy by using reinforcement learning ;

[0012] The probability of the ith obstacle changing in the jth direction is:

[0013] ;

[0014] , indicates moving up, indicates moving down, indicates moving left, indicates moving right, indicates not moving, indicates the number of changes of the ith obstacle in the jth direction, is the total time step of this training.

[0015] In the preferred technical solution, step S02 further comprises:

[0016] calculating the new-old cumulative return improvement ratio under the change condition of any obstacle combination at any time ; , , wherein, is the cumulative return in the changed environment after relearning, is the cumulative return in the changed environment while maintaining the original strategy.

[0017] In the preferred technical solution, the relearning benefit at any moment is calculated according to the new and old cumulative return promotion ratio in step S03.

[0018] In the preferred technical solution, the relearning benefit at any moment is calculated as follows in step S03:

[0019] ;

[0020] Wherein, P represents the probability of the i-th obstacle combination change, P represents the probability of the i-th obstacle changing to the j-th direction, , P represents the probability of the i-th obstacle changing to the j-th direction, N represents the total number of obstacles, and the threshold value.

[0021] In the preferred technical solution, the relearning cost at any moment is calculated as follows:

[0022] ;

[0023] Wherein, P represents the probability of the i-th obstacle combination change, P represents the probability of the i-th obstacle changing to the j-th direction, P represents the probability of the i-th obstacle changing to the j-th direction, is a dangerous action set, is a function, when , the value is 1, otherwise the value is 0.

[0024] In the preferred technical solution, the best moment for relearning is determined according to the obtained relearning net benefit as follows:

[0025] ;

[0026] Wherein, P represents the probability of the i-th obstacle combination change, argmax P represents the probability of the i-th obstacle changing to the j-th direction, is a safety threshold value, and are coefficients for adjusting and to the same dimension.

[0027] The application also discloses a mobile robot navigation control system based on reinforcement learning, which is used for realizing the mobile robot navigation control method based on reinforcement learning.

[0028] The reinforcement learning model construction module discretizes the actual environment of robot movement into a grid map, and constructs a reinforcement learning model of the robot for navigation on the basis. ​​​

[0029] The cumulative return calculation module estimates the change rule of the obstacles in the environment, obtains the probability of each obstacle changing in different directions in the process of obtaining the optimal strategy by using reinforcement learning, and calculates the cumulative return of relearning under the change condition of any obstacle combination at any moment and the cumulative return of maintaining the original strategy.

[0030] The optimal relearning time calculation module calculates the relearning benefit at any moment and the relearning cost to obtain the net relearning benefit at any moment, and determines the best moment for relearning according to the obtained net relearning benefit.

[0031] The application further discloses a computer storage medium, which stores a computer program, and the computer program is executed to realize the mobile robot navigation control method based on reinforcement learning.

[0032] The application further discloses an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor runs the computer program stored on the memory, and the computer program is executed to realize the mobile robot navigation control method based on reinforcement learning.

[0033] Compared with the prior art, the application has the following advantages:

[0034] The application considers not only the learning benefit but also the safety cost of learning when the environment changes in the mobile robot navigation based on reinforcement learning in a dynamic environment, determines whether to perform a new round of reinforcement learning and the optimal relearning time according to the net learning benefit, avoids re-performing reinforcement learning to obtain the optimal strategy after the current changed environment each time the environment changes, and reduces the safety risk caused by relearning at each moment. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The application further discloses a computer storage medium, which stores a computer program, and the computer program is executed to realize the mobile robot navigation control method based on reinforcement learning. DETAILED DESCRIPTION

[0036] The principle of the application is that when the environment changes in the mobile robot navigation based on reinforcement learning in a dynamic environment, not only the learning benefit but also the safety cost of learning are considered, whether to perform a new round of reinforcement learning and the optimal relearning time are determined according to the net learning benefit, re-performing reinforcement learning to obtain the optimal strategy after the current changed environment each time the environment changes is avoided, and the safety risk caused by relearning at each moment is reduced.

[0037] Embodiment 1:

[0038] As Figure 1As shown, a mobile robot navigation control method based on reinforcement learning comprises the following steps:

[0039] S01: Discretize the actual environment of robot movement into a grid map, and build a reinforcement learning model of the robot for navigation on this basis;

[0040] S02: Estimate the change rule of obstacles in the environment, obtain the probability of each obstacle changing in different directions in the process of obtaining the optimal strategy by using reinforcement learning, and calculate the cumulative return of relearning and the cumulative return of maintaining the original strategy under the change condition of any obstacle combination at any time;

[0041] S03: Calculate the relearning benefit at any time, and the relearning cost, obtain the relearning net benefit at any time, and determine the best time for relearning according to the obtained relearning net benefit.

[0042] In a preferred embodiment, the probability of each obstacle changing in different directions obtained in step S02 comprises:

[0043] Counting the number of changes of each obstacle in different directions in the process of obtaining the optimal strategy by using reinforcement learning ;

[0044] The probability of the ith obstacle changing in the jth direction is:

[0045] ;

[0046] , indicates moving up, indicates moving down, indicates moving left, indicates moving right, indicates not moving, indicates the number of changes of the ith obstacle in the jth direction, is the total time step of this training.

[0047] In a preferred embodiment, step S02 further comprises:

[0048] calculating the new-old cumulative return improvement ratio under the change condition of any obstacle combination at any time , specifically , , wherein is the cumulative return in the changed environment after relearning, is the cumulative return in the changed environment by maintaining the original strategy.

[0049] ​​In a preferred embodiment, the relearning benefit at any moment is calculated according to the ratio of the new and old cumulative returns in step S03.

[0050] In a preferred technical solution, the relearning benefit at any moment is calculated as follows in step S03:

[0051] ;

[0052] wherein, Pij represents the probability of the i-th obstacle changing to the j-th direction, and specifically, Pij = f (xi, xj). , Pij represents the probability of the i-th obstacle changing to the j-th direction, N represents the total number of obstacles, and is a threshold value.

[0053] In a preferred embodiment, the relearning cost at any moment is calculated as follows:

[0054] ;

[0055] wherein, is the optimal strategy obtained or possessed by the current reinforcement learning, is the action corresponding to the current state, is the set of dangerous actions, is a function, and when , the value is 1, otherwise the value is 0.

[0056] In a preferred embodiment, the optimal moment for relearning is determined according to the obtained relearning net benefit as follows:

[0057] ;

[0058] wherein, argmax is a parameter function returning the maximum value, is a safety threshold value, and are coefficients for adjusting and to the same dimension.

[0059] In another embodiment, a computer storage medium has a computer program stored thereon, and the computer program is executed to implement the above-mentioned method for navigation control of a mobile robot based on reinforcement learning.

[0060] The specific implementation adopts the above method, which will not be described here.

[0061] ​In another embodiment, an electronic device comprises a memory and a processor, the memory storing a computer program, and the processor running the computer program stored on the memory, the computer program being executed to implement the above-mentioned reinforcement learning-based mobile robot navigation control method.

[0062] The specific implementation adopts the above method, which will not be described here.

[0063] In another embodiment, a reinforcement learning-based mobile robot navigation control system is used to implement the above-mentioned reinforcement learning-based mobile robot navigation control method, comprising:

[0064] The reinforcement learning model construction module discretizes the actual environment of the robot movement into a grid map, and constructs a reinforcement learning model of the robot for navigation on this basis.

[0065] The cumulative return calculation module estimates the change rule of the obstacles in the environment, obtains the probability of the change of each obstacle in different directions in the process of obtaining the optimal strategy by reinforcement learning, and calculates the cumulative return of relearning and the cumulative return of maintaining the original strategy under the change condition of any obstacle combination at any time.

[0066] The optimal relearning time calculation module calculates the relearning benefit and the relearning cost at any time, obtains the relearning net benefit at any time, and determines the best time for relearning according to the obtained relearning net benefit.

[0067] Specifically, the working process of the reinforcement learning-based mobile robot navigation control system will be described below with the preferred embodiment as an example, comprising the following steps:

[0068] S1: Grid the actual environment of the robot movement, and construct a reinforcement learning model of the robot.

[0069] For example, the robot moves in a 30m square field, if the grid size is 0.1m , then the environment can be equivalent to a 300 grid map. The specific grid size is selected according to the size of the actual environment, and the map is not too large on the premise of ensuring that the grid division is relatively fine (i.e. ensuring that the robot or the obstacles in the environment can only move 1 grid at a time).

[0070] The reinforcement learning model can be set by using existing reinforcement learning techniques. The reinforcement learning model can be Q-learning, DQN, PPO, DDPG, etc., without limitation.

[0071] For example, if using Q-learning, a kind of reinforcement learning, in a grid environment, the state space is the actual grid coordinates of the robot, the action space contains four actions of up, down, left and right, and the state transition can adopt a deterministic transition mechanism, i.e. if the robot performs the up action, the robot will eventually reach the grid above the present grid. If the robot performs an action that will exceed the range of all existing grids, the robot will keep the coordinates unchanged after performing the action, for example, if the robot continues to perform the up action at the topmost grid, the robot will exceed the grid range after performing the action, and in practice, will be blocked by the boundary and cannot exceed the grid, so the robot will still stay at the present grid position. The reward function can be set to obtain 100 for reaching the specified target position, -500 for hitting the obstacle, and -1 for normal independent steps.

[0072] S2: Calculate the relearning benefit at any time .

[0073] S21: Estimate the change rule of the obstacles in the environment, and the specific method is to count the number of changes of each obstacle in different directions during the process of obtaining the optimal strategy by using reinforcement learning.

[0074] ;

[0075] wherein, Pi,j represents the probability of the i-th obstacle changing in the j-th direction, , N represents the total number of obstacles, , Up represents moving up, Down represents moving down, Left represents moving left, Right represents moving right, NoMove represents not moving, Ni,j represents the number of changes of the i-th obstacle in the j-th direction, Ntotal represents the total time steps of this training. The total time steps and the number of obstacle changes in the same environment (i.e. the grid size, the number of obstacles and the change rule are unchanged) are cumulative.

[0076] For example, the total time steps for obtaining the optimal strategy by using reinforcement learning for the first time is 10,000 steps, and in this process, obstacle 1 moves up 200 times, moves down 300 times, moves left 500 times, moves right 500 times, and it can be inferred that the number of times obstacle 1 does not move is actually 8,500 times, obstacle 2 moves up 500 times, moves down 500 times, moves left 500 times, moves right 500 times, and it can be inferred that the number of times obstacle 2 does not move is actually 8,000 times, then = 0.02, = 0.03, = 0.05, = 0.05, = 0.85, = 0.05, = 0.05, = 0.05, = 0.05, = 0.80.

[0077] Subsequently, in the same environment, the reinforcement learning is enabled again to obtain the latest optimal policy again, and the total time steps used are 10000 steps, in which process, the obstacle 1 moves upward 300 times, downward 200 times, leftward 500 times, rightward 500 times, and it can be inferred that the obstacle 1 does not move 8500 times, the obstacle 2 moves upward 600 times, downward 400 times, leftward 400 times, rightward 600 times, and it can be inferred that the obstacle 2 does not move 8000 times.

[0078] Then the total time steps are accumulated to 20000 steps, the obstacle 1 moves upward 500 times, downward 500 times, leftward 1000 times, rightward 1000 times, and it can be inferred that the obstacle 1 does not move 17000 times, the obstacle 2 moves upward 1100 times, downward 900 times, leftward 900 times, rightward 1100 times, and it can be inferred that the obstacle 2 does not move 16000 times. Therefore, = 0.025, = 0.025, = 0.05, = 0.05, = 0.85, = 0.055, = 0.045, = 0.055, = 0.045, = 0.80.

[0079] S22: calculating any time The probability of each obstacle combination change is .

[0080] In the upper continuous multiplication formula, and Each time only one value can be taken from the corresponding set.

[0081] For example, the probability of the obstacle 1 moving upward and the obstacle 2 moving upward is , the probability of the obstacle 1 moving downward and the obstacle 2 moving upward is , the probability of obstacle 1 not moving and obstacle 2 moving upward is , the total number of combinations.

[0082] S23: Calculate the new-old cumulative return improvement ratio under any obstacle combination change condition at any time, specifically , wherein, , , is the cumulative return in the changed environment after relearning, is the cumulative return in the changed environment while maintaining the original strategy (i.e., still using the current strategy or becoming the last time strategy).

[0083] Cumulative return, also known as long-term return, refers to the sum of all rewards that the agent can obtain in all future time steps starting from the current time. Cumulative return can be obtained through conventional reinforcement learning methods (such as Q-learning), which will not be described in detail here.

[0084] S24: Calculate the relearning benefit at any time, specifically:

[0085] ;

[0086] The specific meaning is that when the maximum value of all possible new-old cumulative return improvement ratios is greater than or equal to the threshold , preferably 0.5, take the maximum value of all obstacle combination change probability and new-old cumulative return improvement ratio product. That is, when the new-old cumulative return improvement ratio is obvious, it is more inclined to consider only the maximum improvement and more inclined to relearn. When the maximum value of all possible new-old cumulative return improvement ratios is less than the threshold , take the mean of all obstacle combination change probability and new-old cumulative return improvement ratio product. That is, when the new-old cumulative return improvement ratio is not particularly obvious, consider the better improvement and the worse improvement, and adopt a more conservative relearning strategy.

[0087] S3: Calculate the relearning cost at any time :

[0088] ;

[0089] wherein, is the optimal strategy that the current reinforcement learning has obtained or possesses, which can be directly calculated according to the current state (i.e., the grid) , and the corresponding action is obtained, which is the knowledge of conventional reinforcement learning and will not be described in detail here. refers to the set of dangerous actions, specifically the actions of the states (i.e., the grid) around each obstacle (up, down, left, and right) towards the obstacle. For a function, when , i.e. the action selected by the optimal policy belongs to the action in the dangerous action set, its value is 1, otherwise its value is 0.

[0090] For example, the action of the grid above the obstacle moving downward, the action of the grid below the obstacle moving upward, the action of the grid left to the obstacle moving right, and the action of the grid right to the obstacle moving left are all dangerous actions.

[0091] S4: Calculate the net relearning benefit at any time and determine the optimal time to perform relearning , specifically:

[0092] ;

[0093] Wherein, is a safety threshold customized by a user according to different scenes and different tasks, and are coefficients for adjusting and to the same dimension, which are also customized by the user according to different scenes and different tasks.

[0094] This step can be interpreted from two levels: on the one hand, for any time t, the method of the present application can be used to determine whether relearning is needed; on the other hand, if only one learning is allowed within a specified time range, the method of the present application can determine the optimal time to learn.

[0095] The above embodiment is a preferred embodiment of the present application, but the embodiments of the present application are not limited by the above embodiment, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A mobile robot navigation control method based on reinforcement learning, characterized in that, Includes the following steps: S01: Discretize the actual environment of the robot's movement into a grid map, and build a reinforcement learning model for the robot for navigation based on this map; S02: Estimate the changing patterns of obstacles in the environment, obtain the probability of each obstacle changing in different directions during the process of obtaining the optimal policy using reinforcement learning, and calculate the cumulative reward of relearning and the cumulative reward of maintaining the original policy under the condition of any combination of obstacles changing at any time. S03: Calculate the relearning gain and relearning cost at any given time to obtain the net relearning gain at that time. Determine the optimal time to perform relearning based on the obtained net relearning gain. The relearning gain at any given time is calculated as follows: in, Indicates the first The probability of different combinations of obstacles. , This represents the probability that the i-th obstacle changes direction to the j-th direction. The total number of obstacles. For threshold; To calculate the cumulative reward in a changed environment after relearning, To maintain the cumulative returns of the original strategy in the changed environment; Calculate the relearning cost at any time step for: in, The optimal policy that reinforcement learning has already obtained or possesses is based on the current state. Find the corresponding action. It is a collection of dangerous actions. For a function, when When the value is 1, the value is 1; otherwise, the value is 0.

2. The mobile robot navigation control method based on reinforcement learning according to claim 1, characterized in that, The probabilities of each obstacle changing in different directions obtained in step S02 include: In the process of obtaining the optimal policy using reinforcement learning, the number of times each obstacle changes in different directions is counted. ; The probability that the i-th obstacle changes to the j-th direction for: , Indicates moving upwards. Indicates moving downwards. Indicates moving to the left. Indicates moving to the right. Indicates no movement. This represents the number of changes in the i-th obstacle towards the j-th direction. This represents the total time step for this training session.

3. The mobile robot navigation control method based on reinforcement learning according to claim 1, characterized in that, Step S02 is followed by: Calculate any time Any combination of obstacles changing conditions The cumulative return increase ratio between the old and new is as follows: , 4. The mobile robot navigation control method based on reinforcement learning according to claim 1, characterized in that, Determine the optimal time to perform relearning based on the net relearning benefit obtained. for: in, argmax ( ) is the function that returns the maximum value. It is a safety threshold. and To adjust and Coefficients of the same dimension.

5. A mobile robot navigation control system based on reinforcement learning, characterized in that, The method for implementing the reinforcement learning-based mobile robot navigation control method as described in any one of claims 1-4 includes: The reinforcement learning model building module discretizes the actual environment of the robot's movement into a grid map, and builds a reinforcement learning model for the robot for navigation based on this map. The cumulative reward calculation module estimates the changing patterns of obstacles in the environment. During the process of obtaining the optimal policy using reinforcement learning, it obtains the probability of each obstacle changing in different directions and calculates the cumulative reward for relearning and maintaining the original policy under any combination of obstacle changes at any time. The optimal relearning time calculation module calculates the relearning benefit and relearning cost at any given time, obtains the net relearning benefit at any given time, and determines the best time to perform relearning based on the obtained net relearning benefit.

6. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the mobile robot navigation control method based on reinforcement learning as described in any one of claims 1-4.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor runs the computer program stored in the memory. When the computer program is executed, it implements the mobile robot navigation control method based on reinforcement learning as described in any one of claims 1-4.

Images