A method for simultaneous localization and mapping based on hierarchical reinforcement learning

Abstract

The application provides a method for simultaneous localization and mapping based on hierarchical reinforcement learning, comprising: obtaining a hierarchical reinforcement learning model, which comprises a localization module, a map reconstruction module, a global strategy module, a planner and a local strategy module; continuously obtaining a pair of observation maps, and determining a control action for moving the robot in the environment, a localization result and an updated map corresponding to the environment based on reinforcement learning by the hierarchical reinforcement learning model, wherein training data is collected after the current control action is performed for the reinforcement learning of the global strategy module, each piece of training data comprising a global reward corresponding to a control action, the global reward being positively correlated with exploration efficiency and map accuracy and additionally rewarding exploration efficiency when the control action leads to a loop signal.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to the field of mobile robot localization and navigation, and more specifically, to a method for simultaneous localization and mapping based on hierarchical reinforcement learning. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM), a core technology for intelligent mobile robots, enables robots to localize themselves and build environmental maps, guiding them in tasks such as navigation. However, traditional SLAM technology often passively perceives the environment, requiring experts to operate the robot or sensor devices to acquire environmental observations and passively create an environmental map. This not only consumes manpower but also limits its application scenarios. Active SLAM, on the other hand, allows robots to autonomously explore their environment and build environmental maps.

[0003] Traditional active SLAM methods primarily focus on map coverage and reducing map uncertainty.

[0004] Early methods primarily focused on a specific problem, including boundary-based greedy search strategies, criterion-based optimization algorithms, and particle filter-based posterior approximation evaluations.

[0005] Some hand-designed strategies consider both map uncertainty and coverage, and these methods have achieved some success in addressing map exploration and positioning stability. However, constructing hand-designed strategies is complex and it is difficult to propose a complete strategy.

[0006] Deep reinforcement learning (DRL) methods offer a way to automatically learn exploration strategies by addressing the exploration-exploitation dilemma. Early DRL methods, such as DQN, were applied to robotic exploration problems. However, these methods often require large amounts of training data and suffer from poor generalization performance during model transfer.

[0007] Existing traditional proactive SLAM methods primarily rely on manually designed strategies, which are time-consuming, labor-intensive, and difficult to formulate comprehensively. They are typically suitable for relatively simple and predictable environments, but have limited exploration capabilities in unknown and complex environments. Therefore, more intelligent exploration strategies are needed to address the challenges of unknown environments.

[0008] End-to-end methods based on deep reinforcement learning (DRL) can learn the inherent patterns of different scenarios by learning from a large amount of exploration task data. Compared with traditional methods, DRL-based methods avoid the tedious manual rule formulation and instead learn complete policies directly from interactions with the environment. However, these methods often use extrinsic reward functions to guide the model, focusing more on obstacle avoidance or mapping problems rather than solving proactive SLAM problems. Although the motivation to use learning methods for exploration is good, transforming the exploration problem into an end-to-end learning problem has drawbacks. These methods are less efficient because they ignore the information provided by mature mapping, localization, and path planning methods. The model needs to map raw observations to exploration actions, and exploration tasks require complete sequences of actions, which increases the learning difficulty of the model. Therefore, these methods require a large amount of training data to converge, and the performance of the converged model is not outstanding. In addition, since the model learns the mapping from raw observations to exploration actions, differences in different scenes, lighting, and even observation positions pose a significant challenge to the model's generalization performance.

[0009] Active SLAM involves large-scale state spaces and continuous decision variables, thus DRL faces many challenges. Hierarchical reinforcement learning is a structured reinforcement learning method that improves learning efficiency and generalization ability by decomposing complex decision problems into multiple interconnected sub-problems. For active SLAM, it can be decomposed into sub-problems such as localization, mapping, exploration decision-making, path planning, and action execution. However, most existing hierarchical methods use neural networks for pose estimation, which makes it impossible to correct the accumulated errors in localization and mapping, causing the map to deteriorate continuously during exploration. Simultaneously, the map serves as input to the policy network; a deteriorated map affects policy formulation, leading to a vicious cycle or even exploration failure.

[0010] In summary, traditional methods have achieved certain results in terms of map coverage and positioning stability, but their design is complex and lacks completeness. End-to-end learning-based methods, through techniques such as DRL and hierarchical reinforcement learning, offer the possibility of automatically learning exploration strategies, but they suffer from problems such as low learning efficiency and accumulated errors. Summary of the Invention

[0011] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method for simultaneous localization and mapping based on hierarchical reinforcement learning.

[0012] The objective of this invention is achieved through the following technical solution:

[0013] According to a first aspect of the present invention, a method for simultaneous localization and mapping (SLAM) based on hierarchical reinforcement learning is provided. The method includes: acquiring a hierarchical reinforcement learning model, comprising a localization module, a map reconstruction module, a global policy module, a planner, and a local policy module; continuously acquiring paired observation maps of the environment observed by a robot through a binocular camera; and having the hierarchical reinforcement learning model determine, based on reinforcement learning, control actions for controlling the robot's movement in the environment, localization results, and updating the map corresponding to the environment. The hierarchical reinforcement learning model is configured to operate in the following manner: using the localization module to determine the corresponding localization result based on the SLAM algorithm and the paired observation maps; and using the mapping module to construct a global map based on the paired observation maps and their corresponding localization results. The system constructs a global map based on multiple depth observation maps and corresponding localization results collected during the loopback signal generation period, upon receiving a loopback signal. A global target point is determined using the global policy module based on the robot's heading angle, the global map, and a local map determined using the localization results and the global map. A collision-free path containing short-term target points is determined using the planner based on the local map and the global target point. Control actions are determined using the local policy module based on the short-term target points and the local map. After executing the control actions, training data is collected for reinforcement learning of the global policy module. Each training data point includes a global reward corresponding to a control action. This global reward is positively correlated with exploration efficiency and map accuracy, and provides an additional reward for exploration efficiency after the control action results in a loopback signal.

[0014] Optionally, each training data point includes a quintuple. Among them, s t Represents the heading angle, global map, and local map at time t, a t This represents the control action at time t. This represents the global reward corresponding to the control action at time t. s represents the local reward corresponding to the control action at time t. t+1 Indicates the execution of control action a t Then, the heading angle, global map, and local map at time t+1.

[0015] Optionally, the global reward is configured to be calculated as follows: a first reward value positively correlated with exploration efficiency is determined based on the difference between the area of ​​the explored region in the global map at time t and the area of ​​the explored region in the global map at time t-1; after dividing the depth observation map at time t into multiple grid maps of different granularities, a second reward value positively correlated with the uniformity of distribution of far and near feature points is determined based on the multiple grid maps of different granularities; when a loopback signal is generated at time t and the first reward value is not 0, a third reward value positively correlated with the first reward value is determined, otherwise the third reward value is 0; the first reward value, the second reward value, and the third reward value are weighted and summed to obtain the global reward at time t.

[0016] Optionally, the second reward value is calculated as follows:

[0017]

[0018] Where l corresponds to the mesh diagram of the l-th granularity, L represents the total number of mesh diagrams divided into L granularities, and C l n represents the weight hyperparameters corresponding to the mesh diagram of the l-th granularity. l,fa k represents the number of grid points containing only distant feature points in the grid diagram of the l-th granularity. l,1 n l,fa The corresponding weight hyperparameter, n l,ca k represents the number of grid points containing only near-feature points in the grid diagram of the l-th granularity. l,2 n l,ca The corresponding weight hyperparameter, n l,fca k represents the number of grid cells in the grid diagram of the l-th granularity that simultaneously contain both distant and near feature points. l,3 Represents n l,fca The corresponding weight hyperparameters are defined as follows: spatial points in the mesh graph with a depth exceeding a preset depth threshold are considered distant feature points, while the rest are considered near feature points.

[0019] Optionally, the third reward value is calculated as follows:

[0020]

[0021] Here, max(·,·) represents taking the maximum value. This represents the area of ​​the explored region in the global map at time t. S represents the area of ​​the explored region in the global map at time t-1. t,loop This is the value indicating whether a loopback signal is generated at time t, where S is the value indicating whether a loopback signal is generated at time t. t,loop It is 1 if it is true, otherwise it is 0.

[0022] Optionally, both the global map and the local map are raster maps.

[0023] Optionally, the local reward is configured to be related to the change in distance between the robot's position and the short-term target point after the robot performs a control action, wherein a local reward greater than zero is given if the distance is closer, and a local reward less than or equal to zero is given otherwise.

[0024] Optionally, the positioning module may be an ORB-SLAM2 module or an ORB-SLAM3 module.

[0025] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions. Attached Figure Description

[0026] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0027] Figure 1 This is a schematic diagram of the modules of a hierarchical reinforcement learning model according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram illustrating the construction of a global map according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of a grid map according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of the global policy module according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of meshes of different granularities according to an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of a local strategy module according to an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0034] As mentioned in the background section, end-to-end learning methods, through techniques such as DRL and hierarchical reinforcement learning, offer the possibility of automatically learning exploration strategies, but suffer from problems such as low learning efficiency and accumulated errors. To address this, the inventors abandon pure end-to-end learning methods and establish a hierarchical reinforcement learning model. In this model, the localization module is implemented based on existing SLAM algorithms and can correct accumulated errors based on loop closure signals. Furthermore, by utilizing the hierarchical approach, the exploration problem of active SLAM is decomposed into a strategy-specified global target point and a navigation problem (i.e., determining a collision-free path and control actions) that extracts short-term target points from the global target point. This significantly reduces the model's task complexity and improves performance. Moreover, the global reward of the global policy module is positively correlated with exploration efficiency and accuracy, and provides additional rewards for exploration efficiency after the control action generates a loop closure signal. This allows the model to explore efficiently while considering mapping accuracy, thereby improving both mapping efficiency and accuracy.

[0035] Before describing the embodiments of the present invention in detail, some of the terms used therein are explained as follows:

[0036] Paired observation images, including RGB observation images and depth observation images taken by binocular cameras at the same location.

[0037] A loopback signal is a signal generated when a robot moves from a point in its environment and returns to that point.

[0038] The heading angle is the angle between the direction of the robot's movement and the preset reference coordinates.

[0039] To understand the overall architecture of the model in this invention, we will first explain the hierarchical reinforcement learning model.

[0040] See Figure 1 According to one embodiment of the present invention, the hierarchical reinforcement learning model includes a localization module, a mapping module, a global policy module, a planner, and a local policy module; in addition, an experience pool is required to collect training data. Each component will be described separately below.

[0041] I. Positioning Module

[0042] The localization module is used to determine the corresponding localization result based on the paired observation maps. This localization result is the result obtained when the robot captured the paired observation maps. However, it is also the localization result corresponding to the paired observation maps, because the paired observation maps and the localization result are associated during map reconstruction in order to determine whether a certain location is a free area or an obstacle area based on the paired observation maps.

[0043] According to one embodiment of the present invention, the localization module employs an ORB-SLAM2 (Oriented FAST and Rotated BRIEF-Simultaneous Localization and Mapping 2) module or an ORB-SLAM3 module, or any other SLAM algorithm capable of simultaneous localization and mapping based on paired images acquired by a binocular camera.

[0044] II. Mapping Module

[0045] The mapping module is used to build a global map based on paired observation maps and their corresponding positioning results, and reconstructs the global map each time a loopback signal is received, in order to reduce the cumulative error of positioning and mapping. At the same time, based on the global map and the current location, a local map of the surrounding area can be extracted.

[0046] According to one embodiment of the present invention, the mapping module constructs a global map based on paired observation maps and their corresponding positioning results. This global map is a raster map, which indicates obstacle areas and free areas (i.e., unobstructed areas). Correspondingly, the local map extracted from this global map is also a raster map. See also Figure 3 Preferably, the mapping module generates a global map in the form of a raster map in the following manner: 3D coordinates of all pixels are calculated based on the depth observation map and camera focal length to generate a dense point cloud; a top-down projection map is generated based on the dense point cloud; and a raster map (global map) is generated based on the keyframe poses extracted from the localization results and the projection map. The projection map includes an obstacle map and an exploration map. The obstacle map indicates obstacle and non-obstacle areas, while the exploration map indicates explored and unexplored areas. This embodiment's technical solution can achieve at least the following beneficial technical effects: For the active SLAM problem, when an agent explores in diverse environments, the collected observation data also exhibits significant data differences; illustratively, some raster maps corresponding to environments shown in the observation maps are as follows: Figure 3 As shown, where, Figure 3 a) and Figure 3 b represents the observations and maps in two different scenarios using the Habitat simulator. Figure 3 c represents the observations and map in Isaac sim, while Figure 3d represents the observations and maps in the real-world environment. Clearly, there are significant data differences between different experimental environments and between simulation and real-world environments in terms of observation. Directly inputting observation data into the network model could lead to significant differences in model performance across different environments, and even cause active SLAM exploration to fail. Therefore, to address this issue, the model input design avoids directly using observation data and instead employs information with similar representations across various environments—raster maps. (See attached image) Figure 3 As shown, the maps in the four environments have similar representations; regardless of whether it's a simulation environment or a real-world environment, the map representation is identical. This input method avoids visual errors caused by inconsistent scene styles, enabling the model to have good generalization performance across various environments. Furthermore, models trained in simulation environments can be directly used in real-world environments, thus solving the problem of reinforcement learning being difficult to train in real-world settings.

[0047] Because positioning and mapping suffer from cumulative errors, and the generation of a loopback signal provides all depth observation maps and corresponding positioning results along a closed loop, these cumulative errors can be corrected. According to one embodiment of the present invention, upon receiving a loopback signal, a global map is reconstructed based on multiple depth observation maps and corresponding positioning results collected during the loopback signal generation period. This embodiment achieves at least the following beneficial technical effects: the solution reduces the cumulative error in positioning and mapping through map reconstruction after loopback, thus minimizing the impact of cumulative errors on subsequent global and short-term target points.

[0048] III. Global Strategy Module

[0049] The global strategy module is used to determine the global target point based on the robot's heading angle, the global map, and the local map.

[0050] See Figure 4 It provides an optional implementation structure for the global strategy module. According to one embodiment of the present invention, the global strategy module includes a first embedding layer for extracting heading angle features based on heading angle, a first deep network for extracting map features based on global and local maps, a first linear layer for extracting global fusion features based on heading angle features and map features, a first strategy layer for determining a multidimensional target vector based on the global fusion features, a Gaussian sampling layer (DiagGaussian) for performing Gaussian sampling on the multidimensional target vector to obtain global target points, and a first scoring layer for determining a global score based on the global fusion features.

[0051] According to one embodiment of the present invention, a first embedding layer is used to convert the heading angle into a heading angle feature in the form of a multi-dimensional vector. For example, the first embedding layer is, for example, a 1D CNN.

[0052] According to one embodiment of the present invention, a first deep network is used to extract map features based on a multi-channel map (e.g., 8 channels, 320 pixels long, and 320 pixels wide), wherein the multi-channel map is a map obtained by stitching together a global map and a local map of the same size according to channels. The first deep network is, for example, a 4-layer, 5-layer, or 6-layer CNN; or, it can be a feature extraction module of some existing image-based deep neural networks.

[0053] According to one embodiment of the present invention, a first linear layer is used to obtain a concatenated vector of heading angle features and map features, and to extract global fusion features based on the concatenated vector. The first linear layer is, for example, a 1-layer or 2-layer fully connected network.

[0054] According to one embodiment of the present invention, the first strategy layer employs the Actor, i.e., the strategy function, in the Actor-Critic algorithm. The Gaussian sampling layer performs Gaussian sampling on the multi-dimensional target vector (e.g., a 128-dimensional or 256-dimensional vector) output by the first strategy layer, outputting a global target point. The global target point is a global coordinate, such as (x, y) on a global map. The first scoring layer employs the Critic, i.e., the scoring function, in the Actor-Critic algorithm. The relevant principles of the Actor-Critic algorithm can be found in the Actor-Critic algorithm itself, and will not be elaborated upon here. The difference in this application lies in the design of the global reward. The global reward will be described in detail below.

[0055] According to one embodiment of the present invention, the global reward is configured to be calculated in the following manner:

[0056] The first reward value, which is positively correlated with exploration efficiency, is determined based on the difference between the area of ​​the explored region in the global map at time t and the area of ​​the explored region in the global map at time t-1.

[0057] After dividing the depth observation map at time t into multiple grid maps of different granularities, a second reward value that is positively correlated with the distribution uniformity of far and near feature points is determined based on the multiple grid maps of different granularities.

[0058] When a loopback signal is generated at time t and the first reward value is not 0, a third reward value that is positively correlated with the first reward value is determined; otherwise, the third reward value is 0.

[0059] The weighted sum of the first, second, and third reward values ​​yields the global reward at time t.

[0060] Preferably, the global reward is calculated as follows:

[0061]

[0062] in, Let represent the global reward at time t. This represents the first reward value at time t. This represents the second reward value at time t. Let α represent the third reward value at time t. The corresponding weight hyperparameter, β represents The corresponding weight hyperparameter, γ, represents The corresponding weight hyperparameters. Among them, These also correspond to rewards for exploration efficiency, map accuracy, and loopback signal, respectively. A schematic explanation follows.

[0063] According to one embodiment of the present invention, the first reward value is calculated in the following manner:

[0064]

[0065] in, Let represent the area of ​​the explored region in the global map at time t. This represents the area of ​​the explored region in the global map at time t-1.

[0066] According to one embodiment of the present invention, the second reward value is calculated in the following manner:

[0067]

[0068] Where, l corresponds to the mesh diagram of the l-th granularity, L represents the mesh diagram divided into a total of L granularities (the number of granularity types L is set to, for example, 2, 3 or 4, to obtain different implementation methods), C l n represents the weight hyperparameters corresponding to the mesh diagram of the l-th granularity. l,fa k represents the number of grid points containing only distant feature points in the grid diagram of the l-th granularity. l,1 Represents n l,fa The corresponding weight hyperparameter, n l,ca k represents the number of grid points containing only near-feature points in the grid diagram of the l-th granularity. l,2 Represents n l,ca The corresponding weight hyperparameter, n l,fca k represents the number of grid cells in the grid diagram of the l-th granularity that simultaneously contain both distant and near feature points. l,3 Represents n l,fcaThe corresponding weight hyperparameters are defined as follows: ORB feature points in the grid map with a depth exceeding a preset depth threshold are considered far feature points, while the rest are considered near feature points. ORB feature points refer to the feature points extracted by the ORB-SLAM algorithm that reflect the accuracy of localization. For example, ORB-SLAM2 extracts ORB features for each keyframe during localization, extracting some ORB feature points in each keyframe. These ORB feature points reflect the accuracy of localization; therefore, considering the uniformity of the ORB feature point distribution reflects map accuracy. For ease of understanding, the following is used... Figure 5 The diagram shows the second reward values ​​for four scenarios when the grid is divided into two granularities. It can be seen that the more uniform the distribution of distant and near feature points, the larger the corresponding second reward value.

[0069] According to one embodiment of the present invention, the third reward value is calculated in the following manner:

[0070]

[0071] Here, max(·,·) represents taking the maximum value. This represents the area of ​​the explored region in the global map at time t. S represents the area of ​​the explored region in the global map at time t-1. t,loop This is the value indicating whether a loopback signal is generated at time t, where S is the value indicating whether a loopback signal is generated at time t. t,loop The value is 1 if it is not 1, and 0 otherwise. Those skilled in the art can also make adjustments to obtain other similar implementations, for example, by... By changing the value of 0 in the code to a smaller value (e.g., 0.001, 0.1, etc.), other similar implementation schemes can be obtained, such as:

[0072]

[0073] This invention designs a reliable reward function for the global strategy module and proposes for the first time a reward corresponding to the uniformity of feature point distribution. It explicitly considers the map area, the uniformity of feature point distribution at different distances, and the loop closure signal. The gain of the map area directly reflects the exploration efficiency, and the feature point distribution and loop closure signal can effectively characterize the mapping accuracy, enabling the model to explore efficiently while taking into account the mapping accuracy.

[0074] IV. Planner

[0075] The planner uses the local map and global target points to determine a collision-free path that includes short-term target points.

[0076] According to one embodiment of the present invention, the planner may employ the Fast Marching Method (FMM), the A-Star Algorithm, or the D-Star Algorithm. After determining a collision-free path using the planner, the preset path can be divided into multiple sub-paths (e.g., 12 or 24 segments), where the target point of the most recent sub-path is the short-term target point.

[0077] V. Local Strategy Module

[0078] The local strategy module is used to determine control actions based on short-term target points and the local map. Short-term target points can be represented by their angle (Dis_Angle) and distance (Dis_Location).

[0079] See Figure 6 It provides an optional implementation structure for the local strategy module. According to one embodiment of the present invention, the local strategy module includes a second embedding layer for extracting angle features based on the angle of the short-term target point, a third embedding layer for extracting distance features based on the distance of the short-term target, a second deep network for extracting local map features based on the local map, a second linear layer for extracting local fusion features based on angle features, distance features and local map features, a second strategy layer for determining control actions based on the local fusion features, and a second scoring layer for determining local scores based on the local fusion features.

[0080] According to one embodiment of the present invention, the second embedding layer is used to convert the angle of the short-term target point into angular features in the form of a multi-dimensional vector. For example, the second embedding layer is a 1D CNN.

[0081] According to one embodiment of the present invention, the third embedding layer is used to convert the distance of short-term target points into distance features in the form of multi-dimensional vectors. For example, the third embedding layer is, for example, a 1D CNN.

[0082] According to one embodiment of the present invention, a second deep network is used to extract local map features based on a local map. The second deep network is, for example, a 4-layer, 5-layer, or 6-layer CNN; or, it is a feature extraction module of some existing image-based deep neural network.

[0083] According to one embodiment of the present invention, the second linear layer is used to obtain a stitched vector of angle features, distance features, and local map features, and to extract local fusion features based on the stitched vector. The second linear layer is, for example, a 1-layer or 2-layer fully connected network.

[0084] According to one embodiment of the present invention, the second strategy layer adopts the Actor, i.e., the strategy function, in the Actor-Critic algorithm. The output of the second strategy layer is the control action (the set of control actions (or the selection space of control actions), such as forward, left turn, and right turn; or, forward, backward, left turn, and right turn. To reduce the difficulty of control and scoring, the second strategy layer can retain the maximum action value from the multiple actions obtained each time and set the rest to 0 as the output control action; for example, assuming that the initial forward value obtained by the second strategy layer is A, the left turn value is B, and the right turn value is C, and B > A > C, then the output control action is (0, B, 0), which means that the action magnitude of this left turn is B. The second scoring layer adopts the Critic, i.e., the scoring function, in the Actor-Critic algorithm. The relevant principles of the Actor-Critic algorithm can be referred to, and the present invention will not elaborate on them. The difference in this application lies in the design of the local reward.

[0085] According to one embodiment of the present invention, the local reward is configured to be related to the change in distance between the robot's position and a short-term target point after the robot performs a control action, wherein a local reward greater than zero is given if the distance is closer, and a local reward less than or equal to zero is given otherwise.

[0086] VI. Experience Pool of Hierarchical Reinforcement Learning Model

[0087] The experience pool is used to collect training data. After executing a control action, the collected training data is used for reinforcement learning of the global policy module and the local policy module. Each piece of training data includes a global reward corresponding to a control action. The global reward is positively correlated with the exploration efficiency and exploration accuracy, and provides an additional reward for the exploration efficiency after the control action causes a loop signal to be generated.

[0088] According to one embodiment of the present invention, each training data item includes a quintuple. Among them, s t Represents the heading angle, global map, and local map at time t, a t This represents the control action at time t. This represents the global reward corresponding to the control action at time t. s represents the local reward corresponding to the control action at time t. t+1 Indicates the execution of control action a t Then, the heading angle, global map, and local map at time t+1. Among them, the quintuple... It can be used for reinforcement learning in the global policy module. It can be used for reinforcement learning in local policy modules. The loss function for reinforcement learning can refer to the loss function defined in the actor-critic algorithm, which will not be elaborated upon in this invention.

[0089] Furthermore, in the real world, collecting training data is a lengthy process and it's difficult to evaluate the mapping results. Therefore, simulators (such as the Habitat simulator) can be used to simulate robot movement and the paired observation maps it observes, thus quickly obtaining a large amount of training data. Of course, if timeliness is not a concern, training data can also be collected directly in the real world while exploring. Additionally, the training data obtained from the simulator can be used to train the global and local policy modules, and the trained modules can be directly used for simultaneous localization and mapping in the real world. Alternatively, the training data from the simulator can be used to train the global and local policy modules, and after training, simultaneous localization and mapping can be performed in the real world, while continuously collecting real-world training data to reinforce the global and local policy modules.

[0090] According to one embodiment of the present invention, a method for simultaneous localization and mapping based on hierarchical reinforcement learning is provided. The method includes: acquiring a hierarchical reinforcement learning model, which includes a localization module, a map reconstruction module, a global policy module, a planner, and a local policy module; continuously acquiring paired observation maps observed by a robot in its environment through a binocular camera, and having the hierarchical reinforcement learning model determine control actions, localization results, and update the map corresponding to the environment based on reinforcement learning; wherein the hierarchical reinforcement learning model is configured to operate in the following manner: using the localization module to determine the corresponding localization results based on the SLAM algorithm and the paired observation maps; using the mapping module to build a global map based on the paired observation maps and their corresponding localization results, and reconstructing the global map based on multiple depth observation maps collected during the loopback signal generation period and the corresponding localization results each time a loopback signal is received; using the global policy module to determine a global target point based on the robot's heading angle, the global map, and a local map determined using the local positioning results and the global map; using the planner to determine a collision-free path containing short-term target points based on the local map and the global target point; and using the local policy module to determine control actions based on the short-term target points and the local map.

[0091] According to another embodiment of the present invention, a robot (such as a robotic vacuum cleaner) capable of simultaneous localization and mapping based on hierarchical reinforcement learning is provided. The robot includes one or more processors; and a memory for storing executable instructions; the one or more processors are configured to perform the methods of the foregoing embodiments by executing the executable instructions.

[0092] In summary, this invention proposes a method for simultaneous localization and mapping (SLAM) based on hierarchical reinforcement learning (or a proactive visual SLAM framework based on hierarchical reinforcement learning). It employs a hierarchical reinforcement learning architecture to address the "exploration-exploitation" dilemma in efficient exploration and accurate mapping within proactive SLAM. The framework decomposes the exploration problem of proactive SLAM into a policy-specified global target point and a short-term target point navigation problem, significantly reducing the model's task complexity and improving performance. It can be widely applied to localization, navigation, and exploration tasks for intelligent mobile robots in indoor and outdoor environments. The framework incorporates a map reconstruction step, considering global map reconstruction after loop closure. This solves the problem of incremental mapping error accumulation in existing hierarchical proactive SLAM methods, outputting more accurate maps while better guiding the exploration strategy formulation of the policy model. Furthermore, it addresses the issues of insufficient reward function design and inadequate generalization ability. This invention offers significant advantages in mapping accuracy and exploration efficiency, and possesses strong generalization ability, making it widely applicable in various environments.

[0093] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0094] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0095] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0096] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for simultaneous localization and mapping based on hierarchical reinforcement learning, the method comprising: Obtain a hierarchical reinforcement learning model, which includes a localization module, a map reconstruction module, a global policy module, a planner, and a local policy module; The system continuously acquires paired observation images of the robot's environment through binocular cameras. The hierarchical reinforcement learning model then determines control actions, localization results, and updates the map corresponding to the environment based on reinforcement learning. The hierarchical reinforcement learning model is configured to operate as follows: The localization module uses the SLAM algorithm and paired observation maps to determine the corresponding localization result. The mapping module is used to build a global map based on paired observation maps and their corresponding positioning results. Each time a loopback signal is received, the global map is reconstructed based on multiple depth observation maps and corresponding positioning results collected during the loopback signal generation period. The global target point is determined using the global strategy module based on the robot's heading angle, the global map, the local map determined by the localization results, and the local map determined by the global map. The planner uses the local map and global target points to determine a collision-free path that includes short-term target points; The local strategy module is used to determine control actions based on short-term target points and the local map; After executing a control action, training data is collected for reinforcement learning of the global policy module. Each training data point includes a quintuple { },in, This represents the heading angle, global map, and local map at time t. This represents the control action at time t. This represents the global reward corresponding to the control action at time t. This represents the local reward corresponding to the control action at time t. Indicates the execution of control actions Then, at time t+1, the heading angle, global map, and local map are used. The global reward is positively correlated with exploration efficiency and map accuracy, and an additional reward is given to exploration efficiency after the control action causes a loopback signal. The global reward is configured to be calculated as follows: A first reward value positively correlated with exploration efficiency is determined based on the difference between the area of ​​the explored region in the global map at time t and the area of ​​the explored region in the global map at time t-1; after dividing the depth observation map at time t into multiple grid maps of different granularities, a second reward value positively correlated with the evenness of the distribution of far and near feature points is determined based on the multiple grid maps of different granularities; when a loopback signal is generated at time t and the first reward value is not 0, a third reward value positively correlated with the first reward value is determined; otherwise, the third reward value is 0; the first reward value, the second reward value, and the third reward value are weighted and summed to obtain the global reward at time t, where: The second reward value is calculated as follows: in, Corresponding to the Mesh diagram of seed size, Indicates the total number of divisions. Mesh diagram of seed size, Indicates the first The weight hyperparameters corresponding to the granularity of the mesh diagram. Indicates the first The number of grids containing only distant feature points in a grid map of a certain granularity. express The corresponding weight hyperparameters, Indicates the first The number of grids containing only near-feature points in a granular grid map. express The corresponding weight hyperparameters, Indicates the first The number of grid cells in a granularity grid that simultaneously contain both far and near feature points. express The corresponding weight hyperparameters are defined as follows: ORB feature points in the mesh graph with a depth exceeding a preset depth threshold are considered distant feature points, while the rest are considered near feature points. The third reward value is calculated in the following manner: in, This indicates taking the maximum value. This represents the area of ​​the explored region in the global map at time t. This represents the area of ​​the explored region in the global map at time t-1. This is a value indicating whether a loopback signal is generated at time t, where if a loopback signal is generated at time t, then... It is 1 if it is true, otherwise it is 0.

2. The method according to claim 1, characterized in that, Both the global map and the local map are raster maps.

3. The method according to claim 1, characterized in that, The local reward is configured to be related to the change in distance between the robot's position and the short-term target point after the robot performs a control action, wherein a local reward greater than zero is given if the distance becomes closer, and a local reward less than or equal to zero is given otherwise.

4. The method according to any one of claims 1-3, characterized in that, The positioning module uses either the ORB-SLAM2 module or the ORB-SLAM3 module.

5. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 4.

6. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 4 by executing the executable instructions.

Images