Exploration method, device and intelligent device based on reinforcement learning

Abstract

The present application provides an exploration method, apparatus and intelligent device based on reinforcement learning. After constructing the map information of the space to be explored based on the obtained radar data, a local map is constructed according to the map information, and local boundary points are obtained. The next target boundary point is obtained based on the optimized reinforcement learning model, the robot is controlled to move to the target boundary point, and the map information and the local map are updated until the local map has no local boundary points available for exploration, and the unexplored global boundary points are determined. The robot is controlled to explore the unexplored global boundary points until the exploration is completed when all global boundary points are traversed. In this scheme, the exploration task is divided into two parts, local and global. The local part uses reinforcement learning to plan the target point, which has small computational complexity, accurate decision-making and easy convergence, while the global part maintains a map of unvisited areas. When the local embedding converges, global planning is enabled to avoid the problem of incomplete exploration.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to an exploration method, device and intelligent equipment based on reinforcement learning. Background Art

[0002] The development trend of robots is intelligence, and they can have human-like thinking and decision-making capabilities to a certain extent. Therefore, whether it is urban search and rescue or unmanned inspection, new requirements are put forward for the autonomous exploration capabilities of mobile robots. Robots need to understand two questions: "Where should I go" and "How should I go". In general, mobile robots need to have the ability to go from their current location to another unknown environment, and be able to complete autonomous exploration of unknown areas, and finally generate 3D point cloud maps and 2D occupancy grid maps for robot navigation or high-definition environmental maps for human reading and viewing.

[0003] At present, many robots still need human assistance to explore the environment, that is, the robot is first manipulated to explore a circle in an environment, and then the environmental map is obtained through the simultaneous localization and mapping technology (SLAM). This kind of robot that is not "fully autonomous" cannot complete the corresponding tasks in completely unmanned environments such as underground exploration and outer space exploration, and does not meet the requirements of intelligent robots. In addition, the spatial localization of the environment may change over time, and the original map will lose its effectiveness. Without autonomous exploration, the robot cannot play its due role. Autonomous exploration robots can also replace humans to perform tasks such as data collection and environmental mapping in various complex and harsh scenarios, which can not only reduce dependence on labor, but also complete tasks better and more accurately.

[0004] The purpose of robot autonomous exploration is to actively build a map of the entire environment without any prior knowledge. It can be mainly divided into three modules. First, the robot uses the sensors it carries to build an environmental map. Second, the autonomous decision-making module extracts map information and decides the next target point. Finally, the navigation module navigates the robot to the target point. Therefore, the decision-making method in autonomous exploration has been one of the hot topics in robot research in the past 20 years. There are two main mainstream decision-making ideas: the first is to make decisions based on boundary point information; the second is to make decisions based on the perspective sampling method of information theory.

[0005] The autonomous exploration solution based on boundary information detects the cross-borders between known and unknown areas in the environment and clusters these boundary points as the robot's next target point. Many people have optimized this, such as changing the search method for boundary points; making the selection of boundary points more diverse, and comprehensively considering the information gain and path cost of each boundary point.

[0006] The method based on perspective sampling uses the uncertainty of map information (map entropy) as the optimization target for the next decision. Cyrill Stachniss et al. proposed using Rao-Blackwellized particle filter to calculate the possible information gain of the robot's next step, and verified it in simulation and reality. Some people also proposed that the 2D map is a binary random variable field, and the robot's actions and the mutual information of Bayesian filter reasoning are associated with the data measured by sensors.

[0007] Currently, there are several ways to use reinforcement learning to solve robot exploration: 1) Using the robot as the sampling center, use reinforcement learning to sample in the occupancy grid map. If the sampled target point is reasonable, it will be the next target point, otherwise resample; 2) Exploration schemes based on boundary points all calculate the potential information gain of each boundary point in various ways. Reinforcement learning can replace this calculation method, and the information gain of each point can be estimated using the Q value based on the value function.

[0008] The methods based on boundary information and map entropy are mainly based on the greedy principle. It is difficult to plan a globally optimal exploration path, and the exploration trajectory is prone to a large number of return traces. Moreover, as the environment grows dynamically, the amount of calculation also increases, making it difficult to explore a complete environment with complex topological structures. Finally, these methods also need to modify the parameters of the target optimization function when facing different environments, and cannot be adjusted adaptively. Summary of the invention

[0009] The purpose of the present invention includes, for example, providing an exploration method, apparatus and intelligent device based on reinforcement learning, which can avoid the problem of exploration falling into local optimality and incomplete exploration.

[0010] The embodiments of the present invention can be implemented as follows:

[0011] In a first aspect, the present invention provides an exploration method based on reinforcement learning, the method comprising:

[0012] Construct map information of the space to be explored based on the acquired radar data;

[0013] According to the map information, a local map within a preset range is constructed with the current position of the robot as the center, and local boundary points of the local map are obtained;

[0014] Importing the local map and the local boundary point into the reinforcement learning model to optimize the reinforcement learning model, and outputting the next target boundary point based on the optimized reinforcement learning model;

[0015] Controlling the robot to move to the next target boundary point, and updating the map information and the local map, until the local map has no local boundary points available for exploration, and determining the unexplored global boundary points;

[0016] The robot is controlled to explore unexplored global boundary points until the exploration is completed when all global boundary points are traversed.

[0017] In an optional embodiment, the map information includes a 2D grid map, and the 2D grid map includes a plurality of grid points;

[0018] The step of obtaining local boundary points of the local map includes:

[0019] For each grid point in the local map, obtaining a plurality of neighborhood points of the grid point;

[0020] For each of the neighborhood points, if the neighborhood point is in a known area and the neighborhood point has a neighborhood point in an unknown area, the neighborhood point is marked as a boundary point, wherein the known area is the area commonly occupied by all current local maps, and the unknown area is the area outside all current local maps.

[0021] In an optional embodiment, the reinforcement learning model is a two-layer network architecture, including network one and network two;

[0022] The step of importing the local map and the local boundary points into the reinforcement learning model to optimize the reinforcement learning model includes:

[0023] The current observation state is obtained based on the local map, local boundary points and the current posture information of the robot;

[0024] According to the current observation state and the reinforcement learning model fitting, a plurality of execution actions are obtained, the next target boundary point of the robot under each execution action is obtained, and a new observation state at each next target boundary point is obtained;

[0025] Obtaining a current score of the target execution action based on the current observation state obtained by the network 1, wherein the target execution action is the execution action with the largest corresponding score;

[0026] Obtaining a next score of the execution action obtained by the network 2 under the new observation state obtained under the target execution action;

[0027] A loss function is constructed based on the current score and the next score, and the reinforcement learning model is optimized based on the loss function.

[0028] In an optional implementation, the loss function also includes a reward function, the reward function includes information gain, and the information gain is obtained in the following manner:

[0029] Based on the probability information of each grid point being occupied in all the current local maps, the map entropy of all the current local maps is calculated;

[0030] The difference information of the map entropy of all the current local maps at adjacent moments is calculated as the information gain.

[0031] In an optional implementation, the loss function also includes a reward function, the reward function includes information gain, and the information gain is obtained in the following manner:

[0032] According to the number of grid points in free space and the number of grid points in occupied state in all the current local maps, the area of ​​the known area formed by all the current local maps is calculated;

[0033] The difference information between the areas of the known regions at adjacent moments is calculated as the information gain.

[0034] In an optional embodiment, the step of controlling the robot to move to the next target boundary point includes:

[0035] A travel path is obtained based on the current position of the robot and the next target boundary point planning;

[0036] Dividing the travel path into a plurality of landmarks;

[0037] The robot is controlled to move to each landmark point in sequence according to the order of the multiple landmark points, so as to move to the next target boundary point.

[0038] In an optional implementation, the map information further includes 3D point cloud data, and after the step of determining the global boundary point, the method further includes:

[0039] Determine, based on the 3D point cloud data, global boundary points covered by obstacles among the global boundary points;

[0040] The determined global boundary points covered by obstacles are filtered.

[0041] In an optional embodiment, the step of controlling the robot to explore unexplored global boundary points includes:

[0042] Construct a distance matrix between all unexplored global boundary points and the robot's current position;

[0043] According to the distance matrix corresponding to all global boundary points, determine the order of exploring each global boundary point;

[0044] Start exploring each global boundary point in sequence, and update the global map when the robot explores each global boundary point.

[0045] In a second aspect, the present invention provides an exploration device based on reinforcement learning, the device comprising:

[0046] A first construction module is used to construct map information of the space to be explored based on the acquired radar data;

[0047] A second construction module is used to construct a local map within a preset range based on the map information and with the current position of the robot as the center, and obtain local boundary points of the local map;

[0048] An optimization module, used for importing the local map and the local boundary point into the reinforcement learning model to optimize the reinforcement learning model, and outputting the next target boundary point based on the optimized reinforcement learning model;

[0049] A control module, used for controlling the robot to move to the next target boundary point, and updating the map information and the local map, until the local map has no local boundary points available for exploration, and determining the unexplored global boundary points;

[0050] The exploration module is used to control the robot to explore the determined global boundary points until the exploration is completed when all global boundary points are traversed.

[0051] In a third aspect, the present invention provides an intelligent device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the intelligent device is running, the processor executes the machine executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0052] The beneficial effects of the embodiments of the present invention include, for example:

[0053] The present application provides an exploration method, apparatus and intelligent device based on reinforcement learning. After constructing the map information of the space to be explored based on the acquired radar data, a local map within a preset range is constructed based on the new map and with the current position of the robot as the center, and the local boundary points of the local map are obtained. The next target boundary point is obtained based on the optimized reinforcement learning model, the robot is controlled to move to the next target boundary point, and the map information and local information are updated until the local map has no layout boundary points available for exploration, and the unexplored global boundary points are determined. The robot is controlled to explore the unexplored global boundary points until the exploration is completed when all global boundary points are traversed. In this scheme, the exploration task is divided into two parts, local and global. The local part uses reinforcement learning to plan the target point, which has a small amount of calculation, accurate decision-making and easy convergence. The global part is a map of an unvisited area. When the layout embedding converges, global planning is enabled to avoid the problem of incomplete exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 A flowchart of an exploration method based on reinforcement learning provided in an embodiment of the present application;

[0056] Figure 2 A local map and a local boundary schematic diagram provided for the embodiments of the present application;

[0057] Figure 3 for Figure 1 Flow chart of sub-steps included in step S102;

[0058] Figure 4 for Figure 1 Flow chart of sub-steps included in step S103;

[0059] Figure 5 This is a schematic diagram of the DQN network architecture;

[0060] Figure 6 A schematic diagram of a pseudo code of a reward function provided in an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the network architecture of the reinforcement learning model provided in the embodiment of the present application;

[0062] Figure 8 A schematic diagram of a local sliding window provided in an embodiment of the present application;

[0063] Fig. 9 for Figure 1 Flow chart of sub-steps included in step S104;

[0064] Fig.10 One of the schematic diagrams of rapid boundary detection provided in the embodiment of the present application;

[0065] Fig.11 The second schematic diagram of rapid boundary detection provided in the embodiment of the present application;

[0066] Fig.12 This is a schematic diagram of 2D grid map error;

[0067] Fig.13 A flowchart of a filtering method provided in an embodiment of the present application;

[0068] Fig.14 Filter schematic diagram for 3D point cloud framework;

[0069] Fig.15 for Figure 1 Flow chart of sub-steps included in step S105;

[0070] Fig.16 To explore the curve diagram of area changing over time;

[0071] Fig.17 A schematic diagram of the curve to explore how the distance changes over time;

[0072] Fig.18 3D and 2D images of the experimental results of the embodiments of the present application;

[0073] Fig.19 A structural block diagram of the smart device provided in the embodiment of the present application;

[0074] Fig. 20 A block diagram of the functional modules of the exploration device based on reinforcement learning provided in an embodiment of the present application.

[0075] Icons: 110 - storage medium; 120 - processor; 130 - exploration device based on reinforcement learning; 131 - first building module; 132 - second building module; 133 - optimization module; 134 - control module; 135 - exploration module; 140 - communication interface. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0077] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0079] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0080] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0081] See also Figure 1 , is a flow chart of an exploration method based on reinforcement learning provided in an embodiment of the present application. The method steps defined in the process related to the exploration method based on reinforcement learning can be implemented by an intelligent device with data processing function, such as a robot. Figure 1 The specific process shown is described in detail.

[0082] S101, constructing map information of the space to be explored based on the acquired radar data.

[0083] S102, constructing a local map within a preset range based on the map information and taking the current position of the robot as the center, and obtaining local boundary points of the local map.

[0084] S103, importing the local map and the local boundary point into the reinforcement learning model to optimize the reinforcement learning model, and outputting the next target boundary point based on the optimized reinforcement learning model.

[0085] S104, controlling the robot to move to the next target boundary point, and updating the map information and the local map, until the local map has no local boundary points available for exploration, and determining the unexplored global boundary points.

[0086] S105, controlling the robot to explore unexplored global boundary points until the exploration is completed when all global boundary points are traversed.

[0087] In this embodiment, the space to be explored is a closed space, which may have obstacles, etc. The robot is placed in the space to be explored, and based on the robot's autonomous exploration capability, the space is explored. The robot may be equipped with a radar device, such as a single-line laser radar, a multi-line laser radar, etc. The radar device on the robot can work continuously, thereby achieving detection within a certain range.

[0088] In this embodiment, the map information of the space to be explored can be constructed based on the radar data continuously collected by the radar device. It should be understood that the overall spatial range of the space to be explored may be very large, and the detection distance of the radar device is limited. Therefore, when the robot is at a certain position, the detected radar data is only the information of a part of the area in the space to be explored with the position as the center point. As the robot continues to move, the total detected area information is also constantly changing and increasing.

[0089] In this embodiment, after obtaining radar data from the radar device each time, map information of the space to be explored may be constructed based on the radar data. The map information is map information of a portion of the area within the space to be explored.

[0090] In this embodiment, it is assumed that the robot is initially located at a certain point X(x, y, θ) in the environment, where x and y represent 2D coordinate information and θ represents the orientation of the robot. At this time, the data collected by the radar device is Z0, and the robot needs to obtain its own positioning information and map information based on the current posture information and radar measurement data. The core idea of ​​​​simultaneous localization and mapping (SLAM) technology is to estimate the joint posterior probability density function based on radar observations and odometer measurement information, that is, p(x 1:t ,m|z 1:t ,u 0:t )=p(m|x 1:t ,z 1:t )p(x 1:t |z 1:t ,u 0:t ). where x 1:t represents the robot's historical pose sequence, m represents the map to be solved, z 1:t represents the historical radar observation sequence, u 0:tis a historical sensor (odometer, inertial navigation) data sequence. SLAM technology enables the robot to have the ability to perceive the environment, such as its own posture, distance information of obstacles, etc. The map it generates will be used for navigation and decision-making modules in autonomous exploration. In this embodiment, the latest 2D laser cartographer algorithm based on graph optimization and lightweight 3D point cloud lego-loam algorithm can be used to obtain the 2D occupancy grid map and 3D point cloud map of the environment.

[0091] In this embodiment, the map information obtained based on radar data may include a 2D grid map, and the resolution of the 2D grid map is easy to adjust and the search space is small. In this embodiment, the construction of the local map and the search and clustering of boundary points are mainly performed based on the 2D grid map.

[0092] In this embodiment, the local map can be constructed with the current position of the robot as the center and within a preset range. For example, a square with a side length of L and a robot posture X(x, y, θ) as the center is the local 2D map. Figure 2 As shown in the figure, the local map is the local field of view centered on the robot. The multiple dense points on the boundary are the boundary points detected by the current map. The black represents the obstacle outline after expansion, the gray-white represents the explored free area (no obstacles), the gray-black represents the unexplored area, and the large points represent the points after local boundary clustering.

[0093] In this embodiment, the boundary is defined as the intersection of the known area and the unknown area, that is, the intersection is a set of unknown area points, and each point has a free space neighboring point. Among them, the known area can be the area occupied by all current local maps, and the unknown area is the area outside all current local maps.

[0094] Traditional boundary detection algorithms will search the entire 2D map in a breadth-first manner, add boundary points to a queue, and then use this point as the search starting point to continue to traverse the entire map in a breadth-first manner, and put the interconnected boundary points into the same queue. In this embodiment, a wavefront boundary detection algorithm (WFD) is used for boundary search and clustering. Its main improvement is that it only searches the known area each time instead of traversing the entire 2D map, which greatly reduces the amount of calculation.

[0095] In this embodiment, after constructing the local map, the local boundary points of the local map are determined. Figure 3 , the local boundary points can be determined as follows:

[0096] S1021, for each grid point in the local map, obtain a plurality of neighborhood points of the grid point.

[0097] S1022: For each of the neighborhood points, if the neighborhood point is in a known area and the neighborhood point has a neighborhood point in an unknown area, mark the neighborhood point as a boundary point.

[0098] In this embodiment, assume that the currently occupied 2D grid map is m, the boundary point set is F, and F can be considered empty at the initial moment. The robot starting point is c0∈m. Among them, c0 is any grid point in the local map. Take c0 as the search starting point and add it to the search queue queue. Take a point c from the queue each time. a . For c a If c is in a known area and has a neighboring point in an unknown area, mark c as a boundary point f, and then continue searching with c as the starting point, add the boundary points connected to it to a queue f′, and add f′ to the point set F. If c does not meet the above conditions, add c to the search queue queue. Perform detection and testing in the above manner until the queue is empty and the search is completed.

[0099] The reinforcement learning model in this embodiment adopts a reinforcement learning algorithm based on a value function, the Deep-Q-learning (DQN) algorithm. The Q-learning algorithm uses a Q-table to store the Q value of each state, which is difficult to use in a high-dimensional state space. Among them, the Q value can be understood as the score after performing a certain action in a certain state. DQN combines deep learning and Q-learning to fit a function that estimates the Q value with a neural network. It should be noted that in this embodiment, in addition to the DQN algorithm, other reinforcement learning algorithms that can be used in discrete action spaces can also be used, such as the Proximal Policy Optimization (PPO) algorithm, ACKTR, etc.

[0100] The reinforcement learning model uses a two-layer network architecture: Network 1 (Local Network) and Network 2 (Target Network). When optimizing the reinforcement learning model based on local maps and local boundary points, it can be achieved in the following ways, see Figure 4 :

[0101] S1031, obtaining the current observation state based on the local map, local boundary points and the current position information of the robot.

[0102] S1032, obtaining multiple execution actions according to the current observation state and the reinforcement learning model fitting, obtaining the next target boundary point of the robot under each execution action, and obtaining a new observation state at each next target boundary point.

[0103] S1033, obtaining a current score of the target execution action based on the current observation state obtained by the network 1, wherein the target execution action is the execution action with the largest corresponding score.

[0104] S1034, obtaining a next score of the execution action obtained by the network 2 under the new observation state under the target execution action.

[0105] S1035, constructing a loss function based on the current score and the next score, and optimizing the reinforcement learning model based on the loss function.

[0106] In this embodiment, the input of the reinforcement learning model is the local boundary points P{p1,p2,p3,…,p n}、The current position P of the robot c and local map m local , based on the output, we can get the next boundary point to explore {p i |i∈n}.

[0107] like Figure 5 As shown in , the local boundary points, current pose information and local map constitute the current observation state s(m local ,P c ,P), outputs the execution action a(p i ), the robot selects the target boundary point for the next exploration according to the execution action a, and then obtains the new observation state s_, and repeats this process until the exploration is completed. The update of the neural network is mainly through experience playback. The experience pool stores the tuple (s, a, r, s _ ), where r represents the reward item.

[0108] In this embodiment, the network 1 in the reinforcement learning model is used to estimate the current Q value, that is, the current score of the action executed under the current observation state, and the network 2 is used to estimate the next state Q value, that is, to obtain the next score of the action executed under the new observation state.

[0109] In this embodiment, network 1 and network 2 may be a local network and a target network, respectively. When the tuples stored in the experience pool reach a certain threshold, the cross entropy loss of the local network and the target network is calculated to update the local network, and the parameters between the two networks are synchronized every n steps.

[0110] The major improvement of DQN is the use of two networks for estimating Q values ​​and experience replay, in which experience replay uses random sampling to solve the problems of data correlation and non-static distribution.

[0111] Based on the above ideas, the constructed loss function can be shown as follows:

[0112] Loss=[r+γmaxQ(s′,a′)-Q(s,a)] 2

[0113] Reinforcement learning generally requires the assumption that the environment conforms to the Markov property, that is, the state observed by the robot from the environment is comprehensive, and the next state is only related to the current state and is not affected by the historical state. c ,P,n local ), the environment map, boundary points and robot pose are aggregated into a vector s, which contains all the information that the robot can obtain from the environment, which conforms to the Markov property. According to the Bellman equation, the following formula can be obtained:

[0114] Q(s,a)←r+γmaxQ(s′,a′)(γ∈[0,1])

[0115] Among them, Q(s,a) represents the Q value obtained by the model taking action a in state s, which is a measure of action a. The main goal of reinforcement learning is to converge the Q value to the maximum value. From the above formula, it can be seen that the Q value is not only related to the reward r (reward item) obtained at the current moment, but also to the Q value of the subsequent state. γ is a decay coefficient. The closer γ is to 1, the more the model values ​​the cumulative reward, and vice versa, it only focuses on the current reward. Environmental feedback r has a great influence on reinforcement learning, and reasonable reward design can accelerate model convergence.

[0116] As can be seen from the above, in this embodiment, a reward function (i.e., reward item) is added to the loss function, and the design of the reward function comprehensively considers information gain, moving distance, boundary accessibility, and whether to trigger global rescue. Figure 6 In this embodiment, according to Figure 6 In the way shown, the parameters of the Local and Target networks can be updated based on the above loss function. When the tuples stored in the experience pool of the neural network reach a certain threshold, the cross entropy loss of the Local and Target networks is calculated, and then the Local network is updated, and the parameters between the two networks are synchronized every n steps.

[0117] The major improvement of DQN is the use of two networks for estimating Q values ​​and experience replay, in which experience replay uses random sampling to solve the problems of data correlation and non-static distribution.

[0118] Among them, information gain is a key factor in setting reward. Information gain can generally be calculated in two ways.

[0119] In one possible implementation, information gain can be calculated based on the change in map entropy as follows:

[0120] Based on the probability information of each grid point being occupied in all the current local maps, the map entropy of all the current local maps is calculated, and the difference information of the map entropy of all the current local maps at adjacent moments is calculated as the information gain.

[0121] In this embodiment, the calculation formula of map entropy can be specifically as follows:

[0122]

[0123] H(m) is a measure of map entropy, p(m i,j ) represents the probability of whether the grid point in the i-th column and j-th row in the current 2D grid image is occupied.

[0124] Entropy reflects the degree of disorder in a system. The entropy of a stable system is relatively small. The decrease in entropy of a map indicates that the map is being explored, so the calculation of information gain can be as follows:

[0125] Information=H(m t-1 )-H(m t )

[0126] In addition, in another possible implementation, the information gain may be calculated according to the change in the area of ​​the exploration region as follows:

[0127] According to the number of grid points in free space and the number of grid points in occupied state in all current local maps, the area of ​​the known area constituted by all current local maps is calculated, and the difference information between the areas of the known areas at adjacent moments is calculated as the information gain.

[0128] In this embodiment, the area of ​​the known region in the current 2D grid map can be calculated according to the following formula:

[0129] S(m)=Count(m free )+Count(m occupied )

[0130] Among them, Count(m free ) represents the number of grid points in the free space in the grid map, Count(m occupied ) represents the number of grid points in the grid map that are in an occupied state.

[0131] The calculation formula of information gain is as follows:

[0132] Information=S(m t )-S(m t-1 )

[0133] Convolutional neural networks are good at processing images. They can not only effectively reduce the dimension of large-scale images into small-scale images, but also effectively retain the image features, which conforms to the basic principles of image processing. In this embodiment, a multi-layer fully convolutional neural network is used to extract features, and all Q(s, a) values ​​are output through a fully connected layer. The DQN network architecture that can be adopted in this embodiment is as follows: Figure 7 As shown in the figure, the observation state s enters the network model as input, the features are extracted through a three-layer fully convolutional neural network, the features are calculated after passing through two fully connected layers, and finally the maximum Q value is sampled and output through the softmax function.

[0134] Based on the maximum Q value of the output, the next target boundary point below the maximum Q value can be determined.

[0135] In this embodiment, the local exploration process is analogized to a continuously sliding window, such as Figure 8 As shown in the figure, as long as there are boundary points in the local map, the reinforcement learning model will drive the robot to explore continuously. The advantage of this is that the map is rich in information and convenient as network input. Secondly, the number of local boundary points is small, and the dimension of the reinforcement learning action space is low, which is easy to converge.

[0136] Traditional methods generally estimate the information gain of each boundary by constructing an information gain function. Such greedy search is prone to fall into local optimality and multiple reentry. Replacing this estimation function with a reinforcement learning model can not only avoid pure greedy search, but also maximize the exploration return brought by comprehensive consideration of global information.

[0137] In this embodiment, after the next target boundary point is determined in the above manner, the robot can move from the current position to the determined target boundary point, which is mainly achieved by the robot's autonomous navigation module.

[0138] The navigation of the robot in this embodiment involves local navigation and global navigation. Among them, the improved A* algorithm is used in this embodiment for global navigation. The traditional A* algorithm needs to use breadth-first search to comprehensively consider the impact of the starting point and the end point on the current search path and construct a heuristic function each time it searches for a path. However, when the robot explores an unknown space, the starting point of each navigation plan changes dynamically, but the end point remains unchanged. Therefore, the reverse search (searching for a reachable path from the end point to the starting point) strategy is more reasonable. Secondly, considering the use of historical search information, the speed of re-planning can be accelerated.

[0139] Regarding the local navigation algorithm, this embodiment uses an offline sampling path group algorithm as a local navigation planning algorithm. This module will complete tasks such as path optimization, obstacle avoidance, and trajectory tracking. The algorithm generates a series of alternative path groups based on the vehicle's dynamic constraints and predetermined parameters (sampling distance, sampling angle, etc.). When in use, it is only necessary to block the corresponding path according to the point cloud data of the radar device, and then select the path that is most likely to reach the destination.

[0140] It should be noted that, in this embodiment, the global and local navigation may also be replaced by other navigation algorithms.

[0141] See also Fig. 9 In this embodiment, when controlling the robot to move to the next target boundary point, it can be achieved by the following methods:

[0142] S1041, planning a travel path based on the current position of the robot and the next target boundary point.

[0143] S1042: Divide the travel path into a plurality of landmarks.

[0144] S1043, controlling the robot to move to each landmark point in sequence according to the order of the multiple landmark points, so as to move to the next target boundary point.

[0145] In this embodiment, after obtaining the planned next target boundary point, the robot calls the global navigation algorithm to roughly plan a path from its current position to the target boundary point, and divides this path into n landmarks (w1, w2, w3, ..., x n ). At this point, the local navigation algorithm is called again, and the target point of the local navigation is the landmark point planned globally. The local navigation is called continuously to follow each landmark point in turn, and finally the target boundary point can be reached. In the process of navigation, the mapping algorithm needs to be adjusted continuously to continuously update the map information.

[0146] In this embodiment, the detection of global boundary points is different from that of local ones, and a more efficient and precise detection algorithm is required. First, the global map has a larger range, and each breadth-first traversal will consume a lot of computing power. Secondly, since the error of the sensor is unpredictable, the global map is not necessarily accurate. The points obtained by simply searching the map with breadth-first search may not meet expectations. Some points are not boundaries but are classified as boundaries, and some points that do not meet the boundary definition are also defined as boundaries. Treating points that are not boundaries as boundaries will lead to global boundary redundancy, and the robot will go to areas that have already been explored, wasting exploration time. And some points that do not meet the boundary definition, such as being too close to obstacles or the point itself is unreachable, will also cause the robot to stop in one place, affecting the entire exploration process.

[0147] In order to solve the above problems, in this embodiment, the boundary detection algorithm is optimized to speed up the boundary detection. Each time the boundary is updated, only the area updated by the radar device is updated, and the boundary not in the updated area is not updated, and then all boundaries are clustered.

[0148] like Fig.10 and Fig.11 As shown, assuming that the area updated by the radar equipment this time is ((x min ,x max ),(y min ,y max )), when searching the map with breadth-first search, only the boundary points of the updated area will be detected, and the boundaries of the unupdated area will be directly added to the boundary set F.

[0149] In this embodiment, in order to solve the unreachable boundary problem that may occur in the actual exploration process, a detection / filtering synchronization idea is proposed, and multiple threads are used to simultaneously detect and filter out unreasonable boundaries.

[0150] First, when the robot speed is faster than the map update, it is easy for the robot's surroundings to appear unexplored, such as Fig.12 As shown. Some small hollow areas will appear near the robot's motion trajectory, but these hollow areas are not substantive boundaries, but are caused by errors in the mapping algorithm, and these areas need to be filtered out. Secondly, since 2D maps have unpredictable errors, this embodiment adopts a framework that mainly uses 2D detection and 3D point cloud auxiliary filtering.

[0151] See also Fig.13 Specifically, in this embodiment, the filtering of boundary points can be achieved by:

[0152] S201, determining global boundary points covered by obstacles among global boundary points based on the 3D point cloud data.

[0153] S202: Filter the determined global boundary points covered by obstacles.

[0154] In this embodiment, Fig.14 As shown in the figure, for the boundaries that may cover obstacles, 3D point cloud data is used for auxiliary filtering. The 3D point cloud data can be highly filtered first to remove the ground point cloud. Then, according to the location of the boundary, if there is a point cloud of an obstacle nearby, it will be filtered.

[0155] In this embodiment, when the local exploration has run into trouble, that is, when the local map has no local boundary points available for exploration, the global rescue module is triggered. According to the above local exploration, all reasonable existing boundaries have been detected. If there are still global boundary points that have not been explored, the global exploration can be achieved in the following way. Please refer to Fig.15 :

[0156] S1051, construct a distance matrix between all unexplored global boundary points and the current position of the robot.

[0157] S1052, determining the order of exploring each global boundary point according to the distance matrix corresponding to all global boundary points.

[0158] S1053, start exploring each global boundary point in sequence, and update the global map when the robot explores each global boundary point.

[0159] In this embodiment, it is assumed that the currently unexplored global boundary point set is F(f1,f2,f3,…,f n ), first, the distance matrix d between all global boundary points and the current position of the robot can be constructed i,j Then, the order of visiting each global boundary point is obtained according to the single peddler algorithm, and the first global boundary point is used as the target point according to the order, and then the robot is controlled to move to the target point according to the above-mentioned travel control method, and the global map is continuously updated. Of course, in this embodiment, in addition to using the single peddler algorithm, other algorithms that can ensure coverage of all boundary points can also be used instead.

[0160] The exploration method provided in this embodiment is used to solve the problems that autonomous exploration in large-scale indoor environments is prone to fall into local optimality, there are a large number of turns in the exploration route, and the multi-objective optimization function has high computational complexity. The local exploration planning and global rescue based on reinforcement learning in this embodiment can plan the global optimal exploration trajectory, with small calculation amount, which can effectively improve the exploration quality.

[0161] The exploration method provided in this embodiment is compared and verified with the traditional algorithm in the Gazebo simulation environment from multiple experimental indicators such as exploration distance / time and exploration area / time. Fig.16 , Fig.17 As shown in the figure, HI-RL is a hierarchical exploration framework based on reinforcement learning proposed in this embodiment, NBVT is a heuristic search method based on perspective sampling, and Explore_lite is a traditional method based on greedy boundary exploration. From the experimental values, it can be seen that the exploration efficiency and exploration area completeness of the method proposed in this embodiment are greatly improved compared with the traditional method. Fig.18 It shows the 3D and 2D maps of the robot’s autonomous exploration environment, and also proves the feasibility and completeness of the autonomous exploration solution, which can fully and autonomously explore unknown environments.

[0162] The autonomous exploration solution provided in this embodiment integrates the multi-objective optimization function of the traditional method into the reward function of reinforcement learning, combines machine learning and exploration tasks, and effectively improves the efficiency of the robot's autonomous exploration in unknown environments.

[0163] See also Fig.19 , is an exemplary component diagram of an intelligent device provided in an embodiment of the present application, and the intelligent device may be the robot mentioned above. The intelligent device may include a storage medium 110, a processor 120, an exploration device 130 based on reinforcement learning, and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the intelligent device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0164] The exploration device 130 based on reinforcement learning can be understood as the above-mentioned smart device, or the processor 120 of the smart device, or can be understood as a software function module independent of the above-mentioned smart device or processor 120 that implements the above-mentioned exploration method based on reinforcement learning under the control of the smart device.

[0165] It should be noted that in addition to the above-mentioned components, smart devices may also include other components, such as sensors such as radar equipment.

[0166] like Fig. 20 As shown, the exploration device 130 based on reinforcement learning may include a first construction module 131, a second construction module 132, an optimization module 133, a control module 134 and an exploration module 135. The functions of each functional module of the exploration device 130 based on reinforcement learning are described in detail below.

[0167] A first construction module 131 is used to construct map information of the space to be explored based on the acquired radar data;

[0168] It can be understood that the first building module 131 can be used to execute the above step S101. The detailed implementation of the first building module 131 can refer to the above content related to step S101.

[0169] A second construction module 132 is used to construct a local map within a preset range based on the map information and with the current position of the robot as the center, and obtain local boundary points of the local map;

[0170] It can be understood that the second building module 132 can be used to execute the above step S102. The detailed implementation of the second building module 132 can refer to the above content related to step S102.

[0171] An optimization module 133, configured to import the local map and the local boundary point into a reinforcement learning model to optimize the reinforcement learning model, and output a next target boundary point based on the optimized reinforcement learning model;

[0172] It can be understood that the optimization module 133 can be used to execute the above step S103. The detailed implementation of the optimization module 133 can refer to the above content related to step S103.

[0173] The control module 134 is used to control the robot to move to the next target boundary point and update the map information and the local map until the local map has no local boundary points available for exploration, and then determine the unexplored global boundary points;

[0174] It can be understood that the control module 134 can be used to execute the above step S104. The detailed implementation of the control module 134 can refer to the above content related to step S104.

[0175] The exploration module 135 is used to control the robot to explore the unexplored global boundary points until the exploration is completed when all global boundary points are traversed.

[0176] It can be understood that the exploration module 135 can be used to execute the above step S105. The detailed implementation of the exploration module 135 can refer to the above content related to step S105.

[0177] In a possible implementation, the map information includes a 2D grid map, and the 2D grid map includes a plurality of grid points. The second construction module 132 may be used to:

[0178] For each grid point in the local map, multiple neighborhood points of the grid point are obtained; for each neighborhood point, if the neighborhood point is in a known area and the neighborhood point has a neighborhood point in an unknown area, the neighborhood point is marked as a boundary point, wherein the known area is the area commonly occupied by all current local maps, and the unknown area is the area outside all current local maps.

[0179] In a possible implementation, the reinforcement learning model is a two-layer network architecture, including network 1 and network 2, and the optimization module 133 can be used to:

[0180] The current observation state is obtained based on the local map, local boundary points and the current posture information of the robot; multiple execution actions are obtained according to the current observation state and the reinforcement learning model fitting, and the next target boundary point of the robot under each execution action is obtained, and the new observation state at each next target boundary point is obtained; the current score of the target execution action obtained by the network one under the current observation state is obtained, and the target execution action is the execution action with the largest corresponding score; the next score of the execution action under the new observation state obtained by the network two under the target execution action is obtained; a loss function is constructed based on the current score and the next score, and the reinforcement learning model is optimized based on the loss function.

[0181] In a possible implementation, the loss function also includes a reward function, and the reward function includes information gain. The optimization module 133 may obtain the information gain in the following manner:

[0182] Based on the probability information of each grid point being occupied in all the current local maps, the map entropy of all the current local maps is calculated; and the difference information of the map entropy of all the current local maps at adjacent moments is calculated as the information gain.

[0183] In a possible implementation, the loss function also includes a reward function, and the reward function includes information gain. The optimization module 133 may obtain the information gain in the following manner:

[0184] According to the number of free space grid points and the number of occupied grid points in all current local maps, the area of ​​the known area formed by all current local maps is calculated; the difference information between the areas of the known areas at adjacent moments is calculated as the information gain.

[0185] In a possible implementation, the control module 134 may be used to:

[0186] A travel path is planned based on the current position of the robot and the next target boundary point; the travel path is divided into a plurality of landmark points; and the robot is controlled to travel to each landmark point in sequence according to the sequence of the plurality of landmark points so as to travel to the next target boundary point.

[0187] In a possible implementation, the map information further includes 3D point cloud data, and the exploration device 130 based on reinforcement learning may further include a filtering module, which may be used to:

[0188] The global boundary points covered by obstacles are determined based on the 3D point cloud data; and the determined global boundary points covered by obstacles are filtered.

[0189] In a possible implementation, the exploration module 135 may be used to:

[0190] Construct a distance matrix between all unexplored global boundary points and the current position of the robot; determine the order of exploring each global boundary point based on the distance matrix corresponding to all global boundary points; start exploring each global boundary point in sequence, and update the global map when the robot explores each global boundary point.

[0191] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0192] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the exploration method based on reinforcement learning provided in the above embodiment is implemented.

[0193] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above-mentioned exploration method based on reinforcement learning can be executed. Regarding the processes involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant descriptions in the above-mentioned method embodiments, which will not be described in detail here.

[0194] In summary, the exploration method, apparatus and intelligent device based on reinforcement learning provided in the embodiments of the present application, after constructing the map information of the space to be explored based on the obtained radar data, constructs a local map within a preset range based on the map information and with the current position of the robot as the center, and obtains the local boundary points of the local map. The next target boundary point is obtained based on the optimized reinforcement learning model, the robot is controlled to move to the next target boundary point, and the map information and the local map are updated until the local map has no local boundary points available for exploration, and the unexplored global boundary points are determined. The robot is controlled to explore the unexplored global boundary points until the exploration is completed when all global boundary points are traversed. In this solution, the exploration task is divided into two parts, local and global. The local part uses reinforcement learning to plan the target point, which has a small amount of calculation, accurate decision-making and is easy to converge, while the global part maintains a map of an unvisited area. When the local embedding converges, global planning is enabled to avoid the problem of incomplete exploration.

[0195] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A reinforcement learning-based exploration method, characterized in that: The method comprises: Construct map information of the space to be explored based on the acquired radar data; According to the map information, a local map within a preset range is constructed with the current position of the robot as the center, and local boundary points of the local map are obtained; Importing the local map and the local boundary point into the reinforcement learning model to optimize the reinforcement learning model, and outputting the next target boundary point based on the optimized reinforcement learning model; Controlling the robot to move to the next target boundary point, and updating the map information and the local map, until the local map has no local boundary points available for exploration, and determining the unexplored global boundary points; Controlling the robot to explore each unexplored global boundary point until the exploration is completed when all global boundary points are traversed; The reinforcement learning model is a two-layer network architecture, including a network 1 and a network 2. The step of importing the local map and the local boundary points into the reinforcement learning model to optimize the reinforcement learning model includes: The current observation state is obtained based on the local map, local boundary points and the current posture information of the robot; multiple execution actions are obtained according to the current observation state and the reinforcement learning model fitting, and the next target boundary point of the robot under each execution action is obtained, and the new observation state at each next target boundary point is obtained; the current score of the target execution action obtained by the network one under the current observation state is obtained, wherein the target execution action is the execution action with the largest corresponding score; the next score of the execution action under the new observation state obtained by the network two under the target execution action is obtained; a loss function is constructed based on the current score and the next score, and the reinforcement learning model is optimized based on the loss function.

2. The exploration method based on reinforcement learning according to claim 1, characterized in that: The map information includes a 2D grid map, and the 2D grid map includes a plurality of grid points; The step of obtaining local boundary points of the local map includes: For each grid point in the local map, obtaining a plurality of neighborhood points of the grid point; For each of the neighborhood points, if the neighborhood point is in a known area and the neighborhood point has a neighborhood point in an unknown area, the neighborhood point is marked as a boundary point, wherein the known area is the area commonly occupied by all current local maps, and the unknown area is the area outside all current local maps.

3. The exploration method based on reinforcement learning according to claim 1, characterized in that: The loss function also includes a reward function, and the reward function includes information gain, which is obtained in the following way: Based on the probability information of each grid point being occupied in all the current local maps, the map entropy of all the current local maps is calculated; The difference information of the map entropy of all the current local maps at adjacent moments is calculated as the information gain.

4. The exploration method based on reinforcement learning according to claim 1, characterized in that: The loss function also includes a reward function, and the reward function includes information gain, which is obtained in the following way: According to the number of grid points in free space and the number of grid points in occupied state in all the current local maps, the area of ​​the known area formed by all the current local maps is calculated; The difference information between the areas of the known regions at adjacent moments is calculated as the information gain.

5. The exploration method based on reinforcement learning according to claim 1, characterized in that: The step of controlling the robot to move to the next target boundary point comprises: A travel path is obtained based on the current position of the robot and the next target boundary point planning; Dividing the travel path into a plurality of landmarks; The robot is controlled to move to each landmark point in sequence according to the order of the multiple landmark points, so as to move to the next target boundary point.

6. The exploration method based on reinforcement learning according to claim 1, characterized in that: The map information also includes 3D point cloud data. After the step of determining the unexplored global boundary points, the method further includes: Determine, based on the 3D point cloud data, global boundary points covered by obstacles among the global boundary points; The determined global boundary points covered by obstacles are filtered.

7. The exploration method based on reinforcement learning according to claim 1, characterized in that: The step of controlling the robot to explore each unexplored global boundary point includes: Construct a distance matrix between all unexplored global boundary points and the robot's current position; According to the distance matrix corresponding to all global boundary points, determine the order of exploring each global boundary point; Start exploring each global boundary point in sequence, and update the global map when the robot explores each global boundary point.

8. An exploration device based on reinforcement learning, characterized in that: For implementing the exploration method based on reinforcement learning according to any one of claims 1 to 7, the device comprises: A first construction module is used to construct map information of the space to be explored based on the acquired radar data; A second construction module is used to construct a local map within a preset range based on the map information and with the current position of the robot as the center, and obtain local boundary points of the local map; An optimization module, used for importing the local map and the local boundary point into the reinforcement learning model to optimize the reinforcement learning model, and outputting the next target boundary point based on the optimized reinforcement learning model; A control module, used for controlling the robot to move to the next target boundary point, and updating the map information and the local map, until the local map has no local boundary points available for exploration, and determining the unexplored global boundary points; The exploration module is used to control the robot to explore the unexplored global boundary points until the exploration is completed when all global boundary points are traversed.

9. A smart device, characterized in that: The smart device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the smart device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1-7.

Images