An indoor patrol robot autonomous navigation method based on an improved DQN algorithm

Abstract

This invention relates to the field of DQN algorithm technology, and particularly to an autonomous navigation method for an indoor patrol robot based on an improved DQN algorithm. The method includes: S1, the patrol robot perceives its surrounding environment and combines this with its own position information and the target points it intends to reach to form a state space. Several target points are sequentially set in the state space and used as inputs to the DQN algorithm; S2, the objective function of the DQN algorithm is improved so that the target points continuously change from starting points until the last target point; S3, the convergence speed of the DQN algorithm is improved by modifying the reward / penalty function r. This invention improves the DQN algorithm to avoid the problem of the algorithm getting stuck in an infinite loop; it segments the robot's path, greatly reducing the number of iterations of the DQN algorithm; and it accelerates the convergence speed of the DQN algorithm by improving the reward / penalty function, thereby completing the task of autonomous navigation for the indoor patrol robot.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of DQN algorithms, in particular to an indoor patrol robot autonomous navigation method based on an improved DQN algorithm. BACKGROUND

[0002] The traditional security system is realized by "people defense + physical defense". With the problems of population aging, soaring labor costs, security personnel loss rate and the like, it is difficult to adapt to the modern security needs, and the security patrol robot industry is ushering in a new development opportunity. The security patrol robot is still in its infancy, but under the huge security market demand, its development potential and future prospects are broad. Since the deep Q network (DQN) algorithm in deep reinforcement learning uses neural networks to integrate on the basis of Q-learning, thereby overcoming the defects of "dimension disaster" caused by the large memory consumption of Q-learning data storage, the search process is constantly convergent, and the optimal solution is finally approached, which has been widely applied to solve the problem of autonomous navigation of robots. Indoor patrol robots are different from ordinary robots, their patrol paths are relatively unified, and obstacles in the path should be avoided first.

[0003] Mnih et al. proposed the first deep reinforcement learning model, namely deep Q network (DQN), which combines neural networks and Q-learning, and uses neural networks to replace the Q value table to solve the dimension disaster problem in Q-learning, but the convergence speed is slow during network training; TaiL et al. applied DQN to model-free obstacle avoidance path planning, but there is a state-action value overestimation problem, which causes the sparse rewards and punishments obtained by the mobile robot, and the planned path is not optimal.

[0004] The traditional path planning algorithm has A* algorithm, artificial potential field method and rapid expansion random tree method. The A* algorithm has the characteristics that when the target point is not reachable, a large amount of performance consumption is caused; the path of the artificial potential field method is generally smooth and safe, but the method has a local optimal point problem, is easy to fall into a local minimum point, cannot find a path between similar obstacles, and after detecting a new environmental obstacle, the path planned by the artificial potential field method may oscillate and swing in a narrow channel. A weakness of the rapid expansion random tree method is that it is difficult to find a path in an environment with a narrow channel. SUMMARY

[0005] In order to overcome the defects of the prior art, the technical scheme adopted by the application is that an indoor patrol robot autonomous navigation method based on an improved DQN algorithm comprises the following steps:

[0006] S1, the patrol robot perceives the environmental information around itself, and combines the position information and the target point to be reached to form a state space, a plurality of target points are set in the state space in turn, and the patrol robot starts from the starting point;

[0007] The patrol robot patrols according to the specified circular path, and when an obstacle suddenly appears in the specified circular path, the patrol robot must give priority to obstacle avoidance, and then should patrol according to the specified path;

[0008] S2, by improving the objective function of the DQN algorithm, the patrol robot takes the current position and speed as the output of the next action, and when the first target point is reached, the next target point is found with the first target point as the starting point, and when the robot reaches the next target point, it completes this segment of path walking, and the cycle is continuously repeated until the last target point is found, and the last target point is the starting point;

[0009] The patrol robot runs in the process of the specified route under the limited condition of the environment and finally returns to the starting point, thereby completing the autonomous navigation task of the mobile robot;

[0010] The DQN algorithm combines neural networks and Q-learning. The neural network needs to model the Q table, and the Q-learning uses Markov decision to model. The current state, action, reward, strategy and next action in Markov decision are used to represent; In order to improve the sample relevance of the robot and solve the efficiency problem of the robot, the experience replay mechanism is introduced, and the uniqueness of the target Q value is used to improve the stability of the action update; DQN includes three steps of establishing an objective function, a target network and introducing experience replay:

[0011] Objective function: the objective function of DQN is constructed by Q-learning, and the formula is as follows:

[0012] Q'(s,a)←Q(s,a)+α[r+γmax a′ Q(s',a')-Q(s,a)] (1)

[0013] Wherein, (s,a) represents the current state and action, (s',a') represents the next state and action, Q(s,a) represents the current action-state value, Q'(s,a) represents the updated state-action value; α represents the learning rate, that is, how much error in the current state will be updated and learned, the value range is 0-1; γ represents the decay value of future rewards, the value range is 0-1; Because in the DQN algorithm, the value of Q is random, therefore, in order to facilitate calculation, the maximum value max in random number is used,a′ Q(s',a') represents the maximum value of Q under the next step state-action value. By breaking down the expression of Q value, the expression shown below can be obtained:

[0014] Q(s1) = r2 + γQ(s2) = r2 + γ[r3 + γQ(s3)] = r2 + γ{[r3 + γ[r4 + γQ(s4)]} = … (2)

[0015] That is: Q(s1) = r2 + γ·r3 + γ 2 ·r4 + γ 3 ·r5 + γ 4 ·r6 + … + γ n ·r(n+2) + … (3)

[0016] The Q value is associated with the rewards and punishments of each subsequent step, but these associated rewards and punishments will decay over time, and the further away from the state s1, the more serious the state decay;

[0017] The target state-action value function can be represented by the Bellman equation:

[0018] y' = r + γmaxQ(s',a',θ) (4)

[0019] Where y' represents the target Q value, θ is the weight parameter trained in the neural network structure model, and maxQ(s',a',θ) represents the maximum value of Q under the next state-action value and θ;

[0020] The loss function is the mean square error loss function, and the formula is shown below:

[0021] L(θ) = E[(y' - Q(s,a,θ)) 2 ] (5)

[0022] Target network: DQN evaluates the current state-action value function through the target network and the prediction network; the target network obtains the target Q value based on the neural network, and uses the target Q value to estimate the Q value of the next action; the prediction network updates the network weight Δθ using the stochastic gradient descent method, and the formula of the gradient descent algorithm is shown below:

[0023]

[0024] Where, is the Hamiltonian operator;

[0025] Experience replay: the experience replay mechanism improves the sample relevance of the robot and solves the problem of efficient utilization of the robot; when the robot and the environment interact information, the robot can obtain the sample database; the robot stores the sample database into the established experience pool and randomly extracts a small part of the data for training samples, and then sends the training samples into the neural network for training; the experience replay mechanism uses the repeatability of the sample itself to improve the learning efficiency;

[0026] Improved objective function: the target point is the position coordinate that the patrol robot needs to reach in the motion state, representing the final position of the robot. For the patrol robot, it needs to travel on a fixed route and return to the initial position after completing the patrol. Placing the target point at the starting position will cause the algorithm to not iterate and directly jump to the end, thereby skipping the patrol step. Even if the function is designed to make the algorithm stop iterating when it reaches the initial position for the second time, due to the inherent characteristics of the DQN algorithm, it will quickly stop iterating, and the final walking path may only be a small grid.

[0027] In a circular route, the final walking route is from the starting point to the second time walking to the starting point, and the robot's walking distance is longer. When encountering obstacles, the DQN algorithm will recalculate from the starting point again, resulting in an excessively long calculation period for the entire circle.

[0028] The target function can be improved to enable the patrol robot to walk along the predetermined route and return to the starting point. After the patrol robot reaches the first target point position and completes the first leg of the journey, a second target point will be generated on the robot's forward route, the first target point will become the initial point for the next leg of the journey, and the point before the first target point on the route will be changed to an obstacle point. By segmenting the iteration, the patrol robot only needs to perform individual iterations for each segment to find the target point, thereby reducing the number of iterations and preventing the robot from taking a "return route" during each iteration. The target point of the last stage is set to the starting point of the initial stage, ensuring that the patrol robot completes the entire route.

[0029] S3, improve the reward function r of the DQN algorithm to improve the convergence speed of the DQN algorithm and optimize the reward sparsity of the environment state space.

[0030] The patrol robot will inevitably encounter obstacles on its path, and the patrol robot must first avoid obstacles before following the set route for patrol.

[0031] By improving the reward function, the DQN algorithm's calculation time is reduced. The reward function r optimizes the target Q value of formula (4), thereby speeding up the gradient descent of formula (6) and reducing the number of iterations.

[0032] The reward function r is shown in formula 7:

[0033]

[0034] Wherein, (i,j) is the horizontal and vertical coordinates in the grid;

[0035] r of the special reward and punishment point (m,n) The characteristics are that the 9 points including the point as the center and the surrounding 8 points are not obstacle points;

[0036] Special reward and punishment value mu, the value range is 10 -6 ~0, a changed mu value is given, namely initial value mu1=10 -6 , with the completion of each target point path, the mu value gradually decreases, and finally becomes 0, mu k =[mu1*(N+1-K)] / N, wherein N is the total number of target points, K is the path segment between two target points, and 1=<K=<N+1.

[0037] The beneficial effects of the present application are:

[0038] 1. The DQN algorithm is improved in the present application, the problem that the algorithm falls into a dead loop is avoided, meanwhile, the path of the robot is segmented, the iteration number of the DQN algorithm is greatly reduced, finally, the DQN algorithm convergence speed is accelerated through improving the reward and punishment function, so that the task of indoor patrol robot autonomous navigation is completed. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is the flow chart of the indoor patrol robot autonomous navigation method based on the improved DQN algorithm of the present application;

[0040] Figure 2 It is the starting target point state diagram of the model one single cycle of the present application;

[0041] Figure 3 It is the final target point state diagram of the model one single cycle of the present application;

[0042] Figure 4 It is the starting target point state diagram of the model two single cycle of the present application;

[0043] Figure 5 It is the final target point state diagram of the model two single cycle of the present application;

[0044] Figure 6 It is the explanation diagram of the special reward and punishment point of the first embodiment of the present application;

[0045] Figure 7 It is the training number-loss function change curve obtained by running the DQN algorithm before improvement;

[0046] Figure 8is a training number-loss function change curve (reward and punishment value μ=0.5) obtained by running the improved DQN algorithm of the application;

[0047] Figure 9 is a training number-loss function change curve (reward and punishment value μ=10 -6 ) obtained by running the improved DQN algorithm of the application;

[0048] Figure 10 is a training number-loss function change curve (reward and punishment value μ changes from 10 -6 to 0) obtained by running the improved DQN algorithm of the application;

[0049] Figure 11 is a running path of the patrol robot under the improved DQN algorithm of the application. DETAILED DESCRIPTION

[0050] The application will be further described below in conjunction with the accompanying drawings and examples, which are simplified schematic diagrams and only schematically show the basic structure of the application, and thus only show the configurations related to the application.

[0051] As shown in Figure 1 , an indoor patrol robot autonomous navigation method based on an improved DQN algorithm comprises the following steps:

[0052] S1, the patrol robot perceives the environmental information around itself, combines the position information of itself and the target points to be reached to form a state space, sets a plurality of target points in the state space in sequence and takes the target points as the input of the improved DQN algorithm, and the patrol robot starts from the starting point;

[0053] S2, by improving the objective function of the DQN algorithm, the patrol robot takes the current position and speed as the output of the next action, finds the next target point after reaching the first target point, continuously loops in sequence, and finds the last target point, which is the starting point;

[0054] As shown in Figure 2The diagram shows Model 1. The patrol robot's path is a circular single-loop path. In a 30*30 grid, the track boundaries are set to black to represent obstacle points; red dots represent the starting point, i.e., the starting position of the patrol robot; yellow dots represent target points. The circular path is segmented. The yellow dots in the initial state are the robot's first-stage target points. After the robot achieves the first stage, a second target point will be generated on the robot's path. At the same time, the first target point becomes the new algorithm iteration starting point, and the white grid point closest to the first target point on the previous path is turned black, making it an obstacle point. This ensures that the patrol robot's efficiency is improved in subsequent iterations and also prevents the robot from "backtracking" during each iteration. The target point of the last stage is set as the starting point of the initial stage, thus ensuring that the robot completes the entire circular path. The state of the target point in the last stage is as follows. Figure 2 As shown.

[0055] like Figure 3 The image shows Model 2, where the patrol robot follows a circular single-loop path, but the white area in the middle of the two black loops is magnified; as shown... Figure 4 As shown, multiple black obstacles are randomly placed in the passageway. The patrol robot must avoid these obstacles first. The red dot represents the starting point, i.e., the starting position of the patrol robot, and the yellow dot represents the target point. Since the patrol robot's walking space is 3 times larger than that in Model 1, the degree of freedom is greatly improved. Therefore, the calculation time of the DQN algorithm in Model 2 will be longer than that in Model 1.

[0056] S3. Improve the reward and penalty function r of the DQN algorithm to increase the convergence speed of the DQN algorithm and optimize the reward and penalty sparsity of the environment state space;

[0057] The computation time of the DQN algorithm is reduced by improving the reward and penalty function. The target Q value of formula (4) is optimized by the reward and penalty function r, thereby accelerating the gradient descent speed of formula (6) and reducing the number of iterations.

[0058] The reward / penalty function r is shown in Equation 7:

[0059]

[0060] Where (i,j) are the horizontal and vertical coordinates in the grid.

[0061] Special reward and punishment points r (m,n) The characteristic is that, centered on this point, all nine points, including the eight surrounding points, are not obstacle points; for example... Figure 6 As shown in the diagram, the small squares are special reward / penalty points, while the small circles are not, because the point to the lower left of the small circles is an obstacle point.

[0062] Special reward / penalty value μ, ranging from 0 to 10. -6Assign a variable value μ, i.e., an initial value of 10. -6 As each target point is completed along the path, the value of μ decreases evenly according to its initial value, eventually becoming 0; Figure 5 For example, there are a total of 7 target points (excluding the last target point, i.e., the starting point). When a target point is reached, if the special reward / penalty conditions are met during the journey, a special reward / penalty value is applied. Figure 5 The special penalty values ​​for the 8 paths are: μ1 = 10 -6 μ2=(6×10 -6 ) / 7、μ3=(5×10 -6 ) / 7、μ4=(4×10 -6 ) / 7、μ5=(3×10 -6 ) / 7、μ6=(2×10 -6 ) / 7、μ7=(10 -6 ) / 7、μ8=0.

[0063] This invention compares the training iterations versus loss function values ​​of the DQN algorithm before and after the improvement; the parameters are shown in Table 1 when using the same parameters:

[0064] Table 1: Parameter Settings

[0065]

[0066] The improved DQN converges approximately 30% faster than the original DQN algorithm, and its average loss function is reduced by about 25% compared to DQN for the same number of training iterations. Figure 7 This represents the curve showing the change in the loss function based on the number of training iterations obtained from running the DQN algorithm before the improvement. Figure 8 The curves represent the changes obtained from running the improved DQN algorithm. The special reward / penalty value μ = 0.5. Before 500 training iterations, it is in the early stage of training, when the robot is just beginning to explore and learn. It has not been able to make correct judgments about obstacles, and the loss value is relatively large. After 500 training iterations, the robot is still in the exploration and learning obstacle avoidance stage. This means that the robot has begun to recognize obstacles and can correctly avoid some obstacles. However, due to insufficient training iterations, it is still interacting with the environment to learn and further adjust its actions to avoid more obstacles in order to reduce the loss. Figure 8 The improved DQN algorithm is more stable. When the number of training iterations is between 500 and 1500, the robot is unstable under both the DQN algorithm and the improved DQN algorithm. When the number of training iterations reaches about 1800, the loss function of the improved DQN algorithm tends to balance. In contrast, the original DQN algorithm requires about 2600 training iterations for the loss function to reach balance.

[0067] The above test stage is to further verify the effectiveness of the network by using the trained result model to test in the same environment. The target function and the reward and punishment value function of the test and the training are consistent. Therefore, the improved DQN algorithm can shorten the training time and make the robot plan a shorter path.

[0068] Although the improved target function and reward and punishment function accelerate the convergence speed of the algorithm, the corresponding loss function value is slightly increased. The analysis result shows that, due to the special reward and punishment value 0.5 being too large, the robot reward and punishment value accumulates too much, which seriously interferes with the judgment of the robot when the improved DQN algorithm tends to be balanced, Figure 9 The special reward and punishment value 0.5 is modified to 10 -6 , and the special reward and punishment value only has an impact at the beginning of learning, and the special reward and punishment value is set to the general reward and punishment value point 0 after the training is completed. The training frequency-loss function generated by the improved DQN algorithm is as follows Figure 9 , which indicates that the value of μ is always 10 -6 The training frequency-loss function change curve obtained by running is as follows Figure 10 The value of μ is 10 -6 at the beginning of the experiment, and the corresponding change curve obtained by running at the later stage of the experiment is as follows: compared with the algorithm before the improvement, the training frequency of the two algorithms is increased, which is caused by the decrease of the special reward and punishment value. It is found that the convergence speed is reduced by about 40% before and after adjusting the μ value of the improved DQN algorithm Figure 9 and Figure 10 When the training frequency is less than 500 times, it belongs to the initial training stage, the robot is in the initial exploration and learning stage, and cannot make correct judgments on obstacles, so the loss value is still large, but the loss function of the algorithm with the μ value changing from 10 -6 to 0 decreases earlier; when the training frequency reaches 500 times, the two algorithms have tended to be balanced, and the training frequency of the algorithm with the changed μ value is less, which means that the improved DQN algorithm with the changed μ value can complete the training faster, and the robot can avoid obstacles and reach the target point faster. In table 2, μ=0 is the DQN algorithm before the improvement, the special reward and punishment value μ=0.5, 10 -6 and changeable μ (10 -6 decreased to 0) are used in the improved DQN algorithm. Through comparison, the method of the application is obviously effective in running time, convergence step number and loss function.

[0069] Table 2: Comparison of four algorithms

[0070]

[0071] After multiple rounds of learning according to the improved DQN algorithm, the patrol robot plans a path according to Figure 11The route shown is patrolled; when the DQN algorithm is used, the patrol robot is prone to infinite loop at the yellow circular target point in the lower left corner and cannot complete the patrol task. Figure 10 The yellow circular target point in the lower left corner is in infinite loop and cannot complete the patrol task.

[0072] In addition to the black ring-shaped obstacle point simulation track setting, the target point, the starting point and the intermediate obstacle point in the application are randomly set, so that the algorithm of the application has strong universality; the stability and the calculation speed of the DQN algorithm before improvement are obviously insufficient; and the effect of the application is optimal running time on the basis of ensuring stability.

[0073] Based on the above ideal embodiments according to the application, through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the application. The technical scope of the application is not limited to the contents in the specification, and must be determined according to the scope of claims.

Claims

1. An improved DQN algorithm for autonomous navigation of indoor patrol robots, characterized in that, Includes the following steps: S1. The patrol robot senses the environmental information around itself and combines its own position information with the target point to be reached to form a state space. Several target points are set in the state space in sequence and used as inputs to the DQN algorithm. S2. By improving the objective function of the DQN algorithm, the patrol robot uses its current position and speed as the output of the next action. After reaching the first target point, it uses the first target point as the starting point to find the next target point, and so on, until the last target point is found. After the robot reaches the first target point, a second target point is generated on the robot's forward path. At the same time, the first target point is made into a new iteration starting point, and the white grid point that is closest to the first target point on the previous forward path is turned black, making it an obstacle point. S3. Improve the reward and penalty function r of the DQN algorithm to increase the convergence speed of the DQN algorithm and optimize the reward and penalty sparsity of the environment state space; The formula for the reward / penalty function r is: (7) Where (i,j) are the horizontal and vertical coordinates in the grid. It is a special reward / penalty value; r is a special reward / penalty point. (m,n) The characteristic is that, with this point as the center, none of the nine points, including the eight surrounding points, are obstacles. Special reward and punishment value , Initial value 10 -6 , The number of paths corresponding to two consecutive target points is divided equally and then decreased until it becomes 0. K =[ 1×(N+1-K)] / N, where N is the total number of target points, K is the path segment between two target points, and 1= <K<=N+1。

Images