A robot active navigation method based on reinforcement learning in unknown environments

Abstract

The invention discloses a robot active navigation method based on reinforcement learning in an unknown environment. The method comprises: creating a simulation robot model; the simulation robot model uses a cartographer algorithm to complete map construction; constructing a reinforcement learning decision network, and designing an objective function of the reinforcement learning decision network; training the reinforcement learning decision network on the simulation robot model, and when the objective function value is minimum, stopping the training to obtain a trained reinforcement learning decision network; transplanting the network parameters of the trained reinforcement learning decision network and the cartographer algorithm parameters into a physical robot, placing the physical robot in an unknown environment, setting a walking task, and the robot autonomously navigating and avoiding obstacles. The invention has the advantages that it is applicable to uneven areas of unmanned aerial vehicles and ground vehicles, consumes less computing resources and memory for large-scale mapping, and has high training efficiency and algorithm accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of Simultaneous Localization and Mapping (SLAM) processing technology and mobile robot navigation technology, and more specifically to a robot active navigation method based on reinforcement learning in an unknown environment. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, mobile robots have been widely used, especially in complex environments where the assistance of mobile robots is more needed. At present, the application scenarios of mobile robots are mainly some simple environments, such as: sweeping robots, handling robots, etc. Most of the robots in these scenarios require human intervention. However, in actual applications, people expect mobile robots to be able to achieve autonomous navigation and obstacle avoidance in complex and unknown environments, and to establish a complete environmental map.

[0003] SLAM technology refers to: in an unknown environment, the mobile robot uses the data generated by the movement and the real-time data obtained by its own sensors to locate itself and gradually build an environmental map. SLAM technology has very important theoretical significance and application value. For example, in the field of driverless cars, SLAM can build a 3D environmental model and positioning navigation through laser radar. In the military, SLAM can allow mobile robots to reach many harsh environments that humans cannot reach. It can help realize the intelligent reconnaissance and combat of robots. It can also be used to search and remove dangerous explosives.

[0004] According to papers published at home and abroad, most of the current SLAM technologies are based on a single sensor, such as LiDAR SLAM and visual SLAM. LiDAR SLAM already has mature algorithms and solutions. With the development of hardware equipment, and in order to meet the requirements of different scenarios as much as possible, visual SLAM has also developed rapidly. Therefore, the current mature SLAM technology is mainly divided into two categories, LiDAR SLAM (LIDAR SLAM) and visual SLAM (VSLAM). Compared with visual SLAM, LiDAR has high reliability, mature technology, wide visibility range, can detect the angle and distance of obstacles with high precision, achieve obstacle avoidance, and is not affected by ambient light, and can work normally even in the dark.

[0005] Active SLAM means that in an unknown environment, the robot can autonomously plan paths and avoid obstacles while building a map of the surroundings. Due to the complexity and uncertainty of the environment, the primary task of active SLAM is to avoid obstacles, including static and dynamic obstacles. Therefore, the two core issues are obstacle avoidance and map building. A good obstacle avoidance and navigation algorithm can enable the robot to adapt to more complex work scenarios and a wider range of fields, remove the limitations of the environment and space on the robot's capabilities, and greatly improve the operability of the mobile robot.

[0006] Reinforcement learning (RL) is one of the three machine learning models besides supervised learning and unsupervised learning. It uses an agent (the agent in reinforcement learning is actually a behavior determiner that can interact with the environment to decide the next action to perform, and is replaced by agent below) as a human expert in the field to take action. RL does not require labeled data; instead, it learns from experience by interacting with the environment, observing, and responding to the results.

[0007] Reinforcement learning can be represented by a Markov decision process (MDP), such as Figure 1 As shown in Figure 1. Each environment is represented by a state that reflects what is happening in the environment. The reinforcement learning agent takes actions in the environment that cause the current state of the environment to change, generates new states, and receives rewards based on the results. The agent receives positive rewards for good behavior and negative rewards for bad behavior, which helps the agent evaluate the actions it has performed in a given state and learn from experience.

[0008] Although most reinforcement learning algorithms use deep neural networks, different algorithms are suitable for different types of environments. There are many types of reinforcement learning, including model-free reinforcement learning and model-based reinforcement learning. There is also a classification that can be divided into policy-based reinforcement learning and value-based reinforcement learning, etc. Figure 2 As shown in the figure. Understanding the model is the environment in reinforcement learning. It is classified according to whether to learn the environment. Model-free reinforcement learning does not learn and understand the environment. Whatever information the environment gives is the information. Common methods include policy optimization and Q-learning.

[0009] In May 2019, Yuan Xiao of Yanshan University published a master's thesis entitled "Research on Active SLAM Algorithms Based on Deep Reinforcement Learning in Complex Environments". He proposed an active SLAM method that integrates the Dueling DQN path planning algorithm with FastSLAM to solve the problem of autonomous navigation of mobile robots in complex environments. Based on the FCRN algorithm, the depth prediction image of obstacles is performed, the Dueling DQN algorithm is used for path planning, and the path planning method is integrated with FastSLAM. Experiments show that the proposed algorithm can avoid different numbers of static and dynamic obstacles, realize autonomous navigation and collision-free walking of mobile robots, and complete the construction of maps for complex environments. However, this document uses FastSLAM for mapping. FastSLAM uses particle filtering, which cannot be applied to uneven areas of drones and ground cars, has no loop detection, and consumes more computing resources and memory for large-scale mapping. The Dueling DQN path planning algorithm inputs the depth information predicted by deep learning. The prediction accuracy directly affects the training efficiency of the entire model and the results of path planning, resulting in insufficient algorithm accuracy. Summary of the invention

[0010] The technical problem to be solved by the present invention is that the existing robot active navigation method is not applicable to uneven areas of drones and ground vehicles, large-scale mapping consumes more computing resources and memory, and the training efficiency and algorithm accuracy are not high.

[0011] The present invention solves the above technical problems through the following technical means: a robot active navigation method based on reinforcement learning in an unknown environment, the method comprising:

[0012] Step 1: Create a simulation robot model;

[0013] Step 2: The simulation robot model uses the cartographer algorithm to complete map construction;

[0014] Step 3: Construct a reinforcement learning decision network and design the objective function of the reinforcement learning decision network. The input of the reinforcement learning decision network is the data of the robot's laser radar scanning the surrounding environment, the relative distance between the robot and the target position, and the robot's forward speed and rotation speed.

[0015] Step 4: Perform reinforcement learning decision network training on the simulation robot model. When the objective function value is the minimum, stop training to obtain a trained reinforcement learning decision network.

[0016] Step 5: Transplant the network parameters of the trained reinforcement learning decision network and the cartographer algorithm parameters into the physical robot, place the physical robot in an unknown environment, set a walking task, and let the robot autonomously navigate and avoid obstacles.

[0017] The present invention adopts the cartographer algorithm to complete map construction, which is suitable for uneven areas of drones and ground cars. The algorithm itself has loop detection, and large-scale mapping consumes less computing resources and memory. The robot avoids obstacles by constructing a reinforcement learning decision network and training the network. The input values ​​of the reinforcement learning decision network are real-time collected environmental data, distance, speed, etc., rather than predicted data. The input data has high accuracy and model precision. In addition, the present invention first uses a simulation robot model to train model parameters, and after the training is completed, the parameters are transplanted to the physical robot, which effectively improves the training efficiency and solves the problem of long training time of existing reinforcement learning.

[0018] Furthermore, the simulation robot model is Turtlebot3-Burger.

[0019] Furthermore, the reinforcement learning decision network includes a 4-layer network, the input of the first convolutional neural network is the laser data of three sampling times, the input of the second convolutional neural network is connected to the output of the first convolutional neural network, the third and fourth networks are fully connected layers, the input of the third network is connected to the output of the second convolutional neural network, the input of the fourth network includes the output of the third network, the relative target position and the current speed of the robot, and the output of the fourth network is the output action a of the robot. t .

[0020] Furthermore, the laser data size of the three sampling times is 512*3, the first layer of convolutional neural network includes 32 one-dimensional filters with a kernel size of 5 and a step size of 2, the second layer of convolutional neural network includes 32 one-dimensional filters with a kernel size of 3 and a step size of 2, the first layer of convolutional neural network and the second layer of convolutional neural network both use the ReLU function as the activation function, the fourth layer of the network uses two activation functions, namely the sigmoid function and the tanh function, which respectively limit the average linear velocity and the rotation angular velocity to between (0.0, 1.0) and between (-1.0, 1.0), and the output action a of the robot t From the normal distribution We randomly sample from , where represents the mean speed calculated by the neural network, is the vector of logarithmic standard deviations, a set of separate parameters.

[0021] Furthermore, the process of designing the objective function of the reinforcement learning decision network is:

[0022] By formula

[0023]

[0024] Construct the objective function of the reinforcement learning decision network;

[0025] Among them, s t represents the state of the reinforcement learning decision network and Represents the lidar data observed in three units of time; Indicates the relative position of the target point in the robot's coordinate system; Represents the observed current speed information of the robot, v is the linear speed, ω is the angular speed; Represents π θ′ From t , a t ) and substitute it into the expected value obtained by the min function, p θ (a t |s t ) indicates state s t and action a t The probability of sampling with θ, p θ′ (a t |s t ) indicates state s t and action a t The probability obtained by sampling with θ′ is Represents the advantage function, which is used to compare the current state s t Next, perform action a t Compared to other actions, the clip function can It is limited to the range of (1-ε, 1+ε). ε is the first hyperparameter and is set to 0.2.

[0026] Furthermore, the advantage function Where α is the second hyperparameter, x represents sampled Gaussian noise and x~N(0,1), A θ′ Yes A θ′ (a t |s t ), A θ′ (a t |s t ) is the advantage function before improvement, A θ′ (a t |s t )=E θ′ [R t |s t , a t ]-V θ′ (s t ), E θ′ [R t |s t , a t ] indicates that in (s t, a t ) after the cumulative expected discounted reward, R t Represents the reward function value at the current moment, V θ′ (s t ) indicates that from the current state s t The cumulative expected discount reward to start with.

[0027] Furthermore, the reward function is R = R g +R c +R ω , where R g represents the target reward, R c represents the obstacle avoidance reward, R ω Indicates stable reward.

[0028] Furthermore, the calculation process of the target reward is:

[0029] By formula Calculate the target reward, where dis curr Indicates the distance from the mobile robot to the target position. pre Indicates the distance that the mobile robot has previously reached the target position. The distance is calculated as

[0030] Indicates the x-axis coordinate of the target position, Indicates the y-axis coordinate of the target position, represents the x-axis coordinate of the robot, Indicates the y-axis coordinate of the robot.

[0031] Furthermore, the calculation process of the obstacle avoidance reward is:

[0032] By formula Calculate the obstacle avoidance reward, where ifCollision indicates when the robot collides with an obstacle.

[0033] Furthermore, the calculation process of the stable reward is:

[0034] By formula Calculate the stable reward, where ω represents the robot's rotation angular velocity, r t is the reward for the time taken to reach the target point, w t is the time weight, w t =0.01.

[0035] The advantages of the present invention are:

[0036] (1) The present invention uses the cartographer algorithm to complete map construction, which is suitable for uneven areas of drones and ground cars. The algorithm itself has loop detection, and large-scale mapping consumes less computing resources and memory. By constructing a reinforcement learning decision network and training the network, the robot avoids obstacles. The input value of the reinforcement learning decision network is real-time collected environmental data, distance, speed, etc., rather than predicted data. The input data has high accuracy and model precision. In addition, the present invention first uses a simulation robot model to train model parameters, and after the training is completed, the parameters are transplanted to the physical robot, which effectively improves the training efficiency and solves the problem of long training time of existing reinforcement learning.

[0037] (2) The present invention has an advantage function A θ′ Gaussian noise is added to interfere with the advantage value, and the weight α is introduced to compare with the advantage function A θ′ The mixed interference advantage value can prevent the reinforcement learning algorithm from falling into the state of overfitting, improve the algorithm accuracy, and thus improve the accuracy of the robot's active navigation and obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of reinforcement learning using a Markov decision process to represent the existing technology;

[0039] Figure 2 A schematic diagram for classifying existing reinforcement learning algorithms;

[0040] Figure 3 A flowchart of a robot active navigation method based on reinforcement learning in an unknown environment disclosed in an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of the overall structure of a robot in a robot active navigation method based on reinforcement learning in an unknown environment disclosed in an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of the structure of a reinforcement learning decision network in a robot active navigation method based on reinforcement learning in an unknown environment disclosed in an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of a training environment of a robot in Gazebo in a robot active navigation method based on reinforcement learning in an unknown environment disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] like Figure 3 As shown, a robot active navigation method based on reinforcement learning in an unknown environment, the method comprising:

[0046] S1: Create a simulation robot model; the simulation robot model can use the existing Turtlebot3-Burger, whose PC is installed with Ubuntu16.04 and ROS Kinetic operating system, and a laser radar is installed on the top of the robot. Turtlebot3-Burger is another kinetic-based software and hardware learning platform officially built by ROS after Turtlebot2. It is smaller in size and more cost-effective. The maximum movement speed is 0.22m / s, the maximum angular velocity is 2.84rad / s, and the maximum load is 15kg. It should be noted that in addition to the above-mentioned Turtlebot3-Burger, the simulation robot model can also use a self-designed robot car with walking function. The robot car is installed with a PCB circuit board. The main control module in the PCB circuit board records the cartographer algorithm and reinforcement learning algorithm that the robot needs to execute. Figure 4 As shown, a robot car structure provided by the present invention has a walking function, and a circuit board is arranged inside the car, and various algorithms are loaded on the circuit board. In the early stage, the robot car can be placed in the Gazebo environment for reinforcement learning obstacle avoidance training, and in the later stage, the robot car can be made into a physical structure, and by loading the trained algorithm or model, SLAM navigation and obstacle avoidance can be realized in the real scene.

[0047] S2: The simulated robot model uses the cartographer algorithm to complete map construction. The cartographer algorithm is an existing mature algorithm and an open source lidar SLAM algorithm from Google. It uses a graph optimization method and is mainly divided into two parts: the front end and the back end: the front end is mainly responsible for creating sub-graphs and local loop detection. Whenever the lidar data is scanned, it is convenient to match the most recently established sub-graph, so that this frame of lidar scan data is inserted into the optimal position of the sub-graph. The sub-graph is updated while new data frames are continuously inserted. When no new scan data is inserted, the sub-graph is successfully created.

[0048] The accuracy of the sub-graphs created in a short time is reliable, but there will be cumulative errors in the long run. In order to eliminate the cumulative errors, loop closure detection is needed to build loops and finally optimize the poses of all sub-graphs. Local loop closure detection is also performed. The backend is mainly responsible for pose optimization and uses the branch and bound method to achieve global loop closure detection and reduce errors.

[0049] S3: At the beginning of the robot SLAM navigation task, the first step is to conduct a reinforcement learning obstacle avoidance test. If there is an obstacle, the PPO algorithm based on the interference advantage value is used to avoid the obstacle; if there is no obstacle, it continues to move forward. The cartographer algorithm is used for positioning and composition until the task is completed and the entire environment map is constructed.

[0050] S3.1: Reinforcement learning obstacle avoidance test, controlled by the reinforcement learning module, mainly includes the reinforcement learning decision network, using the PPO algorithm of interference advantage value, input s t It consists of three parts: data scanned by LiDAR Relative target position and the robot's current speed These three parts are combined into a whole as the state s of the robot in the reinforcement learning module t , taking this state as input, after the reinforcement learning decision network, the output action is the linear velocity and angular velocity of the robot.

[0051] S3.2: Construct a reinforcement learning decision network, which consists of 4 layers of networks to approximate the policy network π θ , architecture such as Figure 5 As shown, the input of the first convolutional network is 512*3 laser data of three sampling times. The network contains 32 one-dimensional filters with kernelsize of 5 and stride of 2. The second convolutional network contains 32 one-dimensional filters with kernelsize of 3 and stride of 2. The first two layers use ReLU function as activation function. The third layer is a fully connected layer, and the fourth layer is a fully connected layer. The input includes the output of the third layer, the relative target position and the current speed of the robot. The fourth layer uses two activation functions, namely sigmoid function and tanh function, which limit the average linear speed and rotational speed to between (0.0, 1.0) and (-1.0, 1.0) respectively. The final output action a t From the normal distribution It is obtained by random sampling. represents the mean speed calculated by the neural network, is the vector of logarithmic standard deviations, a set of separate parameters.

[0052] S3.2.1: The algorithm for the reinforcement learning decision network is the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm proposes a new objective function that can achieve small batch updates in multiple training steps, solving the problem of difficult step size determination in the Policy Gradient (PG) algorithm. It is an on-policy algorithm. If the step size is too small, the training time will be too long. If the step size is too large, useful information will be obscured by noise, or the performance will be catastrophically reduced, making it difficult to converge.

[0053] The robot uses the PPO algorithm to interact with the environment, and obtains observation data, action labels, and reward functions as a trajectory sequence stored in the experience buffer. After accessing a certain number of trajectory sequences, the objective function is calculated. The gradient is obtained and used to update the policy network to maximize the expected return. The objective function is as follows:

[0054]

[0055] Among them, s t represents the state of the reinforcement learning decision network and Represents the lidar data observed in three units of time; Indicates the relative position of the target point in the robot's coordinate system; Represents the observed current speed information of the robot, v is the linear speed, ω is the angular speed; Represents π θ′ From t , a t ) and substitute it into the expected value obtained by the min function, p θ (a t |s t ) indicates state s t and action a t The probability of sampling with θ, p θ′ (a t |s t ) indicates state s t and action a t The probability of action a is obtained by sampling with θ′ t =[v, ω], where v and ω are continuous values, where v∈[0.00, 0.20] and ω∈[-0.60, 0.60]. min is mainly used to select the smaller of the two items. The clip function is a comparison function. The calculation rule is: there are three items in the brackets. If the first item is less than the second item, 1-ε is output; if the first item is greater than the third item, 1+ε is output. ε is the first hyperparameter, which is generally set to 0.2.

[0056] S3.2.2: A in the above objective function θ′ is the advantage function, used to evaluate t Next, perform action a t The advantage relative to the average, from a quantitative point of view, is the deviation of the random variable relative to the mean. The significance of using the advantage function is to improve the learning efficiency of the algorithm, make the learning more stable, and help reduce the variance and prevent overfitting. θ′ Yes A θ′ (a t |s t ), A θ′ (a t |s t ) The calculation formula is: θ′ (a t |s t )=E θ ,[R t |s t , a t ]-V θ′ (s t ), E θ′ [R t |s t , a t ] indicates that in (s t , a t ) after the cumulative expected discounted reward, R t Represents the reward function value at the current moment, which is a parameter in reinforcement learning. When the agent interacts with the environment, the environment will give the agent a reward, namely R t , which may be positive or negative. If it is positive, it will encourage the agent to take more actions. If it is a negative value, it will encourage the agent to take fewer actions. V θ′ (s t ) indicates that from the current state s t The cumulative expected discount reward to start with.

[0057] S3.2.3: The reward function in S3.2.2 above is R = R g +R c +R ω , R g represents the target reward, R c represents the obstacle avoidance reward, R ω In this invention, the robot aims to reach the target position (P x , P y). In order to reach the target position, it is necessary to avoid obstacles through reinforcement learning without colliding with obstacles and stay within the performance limit. Therefore, this part of the reward must also be taken into account. The total reward function is the sum of these three reward functions. Among them,

[0058] 1) R g For the goal reward, if the mobile robot reaches the goal position, the agent will receive a high reward of 10. In addition, when moving to the goal position, if the distance to the goal is shorter than before, then rewards are also given when the robot moves in the right direction.

[0059]

[0060] discurr represents the distance from the mobile robot to the target position. pre Indicates the distance that the mobile robot has previously reached the target position. The distance is calculated as

[0061]

[0062] Indicates the x-axis coordinate of the target position, Indicates the y-axis coordinate of the target position, represents the x-axis coordinate of the robot, Indicates the y-axis coordinate of the robot.

[0063] 2) R c For the obstacle avoidance reward, when the robot collides with an obstacle, the robot will receive a huge penalty of -10. This reward will greatly encourage the robot to avoid collisions with obstacles.

[0064]

[0065] Among them, ifCollision means when the robot collides with an obstacle.

[0066] 3) R ω In order to stabilize the reward and encourage the robot to move smoothly, a stable reward function R is introduced ω To suppress high rotation speed.

[0067]

[0068] Where, ω represents the angular velocity of the robot, r t The time taken to reach the target point is rewarded. The reward value increases by 1 for every minute that passes. t is the time weight, w t =0.01.

[0069] S3.2.4: The advantage function A has been calculated above θ′ In the case of advantage function A θ′ Gaussian noise is added to interfere with the advantage value. Gaussian noise is sampled for the agent, x~N(0,1), and the weight α is introduced to interfere with the advantage function A θ′ Mixed interference advantage value. α is the second hyperparameter, and its value is 0.975. Then the new advantage function is:

[0070]

[0071] So the corresponding new objective function is:

[0072]

[0073] The objective function used in the training process of the reinforcement learning algorithm of the present invention is the above-mentioned new objective function.

[0074] S4: The advantage of reinforcement learning is that it can encode quite complex behaviors by collecting and learning experiential information through interaction with the environment. However, reinforcement learning training takes a long time, converges slowly, and has sparse rewards. It requires constant interaction with the environment, so it is not realistic to train the robot in a real environment. In order to avoid equipment consumption and reduce training time, the robot model is first placed in the Gazebo simulation environment for training, and then transplanted to the physical robot after the training is completed. The Gazebo simulation environment takes into account dynamic factors, such as inertia and friction. Therefore, the present invention first performs reinforcement learning decision network training on the simulation robot model. When the new objective function value in S3.2.4 is the smallest, the training is stopped to obtain a trained reinforcement learning decision network. Figure 6 Shown is the robot's training environment in Gazebo.

[0075] S5: Transplant the network parameters of the trained reinforcement learning decision network and the cartographer algorithm parameters into the physical robot, place the physical robot in an unknown environment, set a walking task, and let the robot autonomously navigate and avoid obstacles.

[0076] Through the above technical scheme, the present invention realizes autonomous navigation of the robot in an unknown environment, and studies the reinforcement learning obstacle avoidance algorithm based on the interference advantage value PPO for obstacle avoidance and the map construction of the active cartographer algorithm. The cartographer algorithm is used to complete the map construction, which is suitable for uneven areas of drones and ground cars. The algorithm itself has loop detection, and large-scale mapping consumes less computing resources and memory. The robot avoids obstacles by constructing a reinforcement learning decision network and training the network. The input value of the reinforcement learning decision network is the real-time collected environmental data, distance and speed, etc., not the predicted data. The input data has high accuracy and model precision. In addition, the present invention first uses a simulation robot model to train the model parameters, and after the training is completed, the parameters are transplanted to the physical robot, which effectively improves the training efficiency and solves the problem of long training time of existing reinforcement learning.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot active navigation method based on reinforcement learning in an unknown environment, characterized in that: The method comprises: Step 1: Create a simulation robot model; Step 2: The simulation robot model uses the cartographer algorithm to complete map construction; Step 3: Construct a reinforcement learning decision network and design the objective function of the reinforcement learning decision network. The input of the reinforcement learning decision network is the data of the robot's laser radar scanning the surrounding environment, the relative distance from the robot to the target position, the forward speed and rotation speed of the robot; the reinforcement learning decision network includes 4 layers of networks. The input of the first layer of convolutional neural network is the laser data of three sampling times. The input of the second layer of convolutional neural network is connected to the output of the first layer of convolutional neural network. The third layer and the fourth layer are both fully connected layers. The input of the third layer is connected to the output of the second layer of convolutional neural network. The input of the fourth layer includes the output of the third layer, the relative target position and the current speed of the robot. The output of the fourth layer is the output action a of the robot. t ; The process of designing the objective function of the reinforcement learning decision network is: By formula Construct the objective function of the reinforcement learning decision network; Among them, s t represents the state of the reinforcement learning decision network and Represents the lidar data observed in three units of time; Indicates the relative position of the target point in the robot's coordinate system; Represents the observed current speed information of the robot, v is the linear speed, ω is the angular speed; Represents π θ′ From t ,a t ) and substitute it into the expected value obtained by the min function, p θ (a t |s t ) indicates state s t and action a t The probability obtained by sampling with θ, represents the advantage function, clip represents the cutting function, Restricted to the range of (1-ε, 1+ε), ε is the first hyperparameter; Step 4: Perform reinforcement learning decision network training on the simulation robot model. When the objective function value is the minimum, stop training to obtain a trained reinforcement learning decision network. Step 5: Transplant the network parameters of the trained reinforcement learning decision network and the cartographer algorithm parameters into the physical robot, place the physical robot in an unknown environment, set a walking task, and let the robot autonomously navigate and avoid obstacles.

2. According to claim 1, a robot active navigation method based on reinforcement learning in an unknown environment is characterized in that: The simulation robot model is Turtlebot3-Burger.

3. The robot active navigation method based on reinforcement learning in an unknown environment according to claim 1 is characterized in that: The laser data size of the three sampling times is 512*3. The first convolutional neural network contains 32 one-dimensional filters with a kernel size of 5 and a step size of 2. The second convolutional neural network contains 32 one-dimensional filters with a kernel size of 3 and a step size of 2. The first and second convolutional neural networks both use the ReLU function as the activation function. The fourth network uses two activation functions, namely the sigmoid function and the tanh function, which limit the average linear velocity and the rotational angular velocity to between (0.0, 1.0) and between (-1.0, 1.0), respectively. The output action a of the robot t From the normal distribution We randomly sample from , where represents the mean speed, is the vector of log standard deviations.

4. The method for active robot navigation based on reinforcement learning in an unknown environment according to claim 1, characterized in that: The advantage function Where α is the second hyperparameter, x represents sampled Gaussian noise and x~N(0,1), A θ′ Yes A θ′ (a t |s t ), A θ′ (a t |s t ) is the advantage function before improvement, A θ′ (a t |s t )=E θ′ [R t |s t , a t ]-V θ′ (s t ), E θ′ [R t |s t , a t ] indicates that in (s t , a t ) after the cumulative expected discounted reward, R t Represents the reward function value at the current moment, V θ′ (s t ) indicates that from the current state s t The cumulative expected discount reward to start with.

5. The method for active robot navigation based on reinforcement learning in an unknown environment according to claim 4, characterized in that: The reward function is R = R g +R c +R ω , where R g represents the target reward, R c represents the obstacle avoidance reward, R ω Indicates stable rewards.

6. The method for active robot navigation based on reinforcement learning in an unknown environment according to claim 5, characterized in that: The calculation process of the target reward is: By formula Calculate the target reward, where dis curr Indicates the distance from the mobile robot to the target position. pre Indicates the distance that the mobile robot has previously reached the target position. The distance is calculated as Indicates the x-axis coordinate of the target position, Indicates the y-axis coordinate of the target position, represents the x-axis coordinate of the robot, Indicates the y-axis coordinate of the robot.

7. The robot active navigation method based on reinforcement learning in an unknown environment according to claim 5, characterized in that: The calculation process of the obstacle avoidance reward is: By formula Calculate the obstacle avoidance reward, where if Collision means when the robot collides with an obstacle.

8. The robot active navigation method based on reinforcement learning in an unknown environment according to claim 5, characterized in that: The calculation process of the stable reward is as follows: By formula Calculate the stable reward, where ω represents the robot's rotation angular velocity, r t is the reward for the time taken to reach the target point, w t is the time weight, w t =0.01.

Images