A Method for Constructing a Large-Scale Semantic Map for Mobile Robots

Abstract

A method for constructing a large-scale semantic map for a mobile robot provided by the present invention belongs to the technical field of visual SLAM, and specifically includes: using the mobile robot to obtain the current image, constructing a visual odometer based on the feature point method, and outputting the pose transformation of adjacent frame images; generating a semantic point cloud map through semantic segmentation, constructing a semantic octree map using the pose transformation, and updating the semantic octree map by adopting an occupancy probability update algorithm and a semantic fusion algorithm; constructing a three-layer memory structure model of the mobile robot, and based on the node movement algorithm between WM and LTM, realizing the reuse of the semantic octree map, and further expanding the current semantic octree map into a global semantic map. The present invention uses a loop detection algorithm to reuse the previous semantic map, fuses the local semantic octree map to generate a global semantic map, and realizes the construction of a high-precision and reusable semantic map.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual SLAM, and particularly relates to a method for constructing a large-scale semantic map for a mobile robot. Background Art

[0002] In recent years, with the rapid development of robot technology and artificial intelligence, the demand for robots in industrial and other fields has gradually increased, such as in aspects like UAV search and rescue, delivery robots, virtual reality, etc. For complex behavior decision-making tasks, only constructing a traditional geometric metric map is not sufficient to complete the tasks, which puts higher requirements on the semantic understanding ability of robots. Therefore, semantic map construction has gradually become an important research topic in the field of visual SLAM (Simultaneous Localization and Mapping). This topic studies: while a mobile robot is positioning, incrementally constructing a three-dimensional map containing semantic information based on the semantic prediction value of a single frame.

[0003] The existing semantic map construction methods generally have the following problems: The method widely uses the Bayesian probability fusion strategy. The main defect of this strategy is that its memory consumption is proportional to the number of semantic categories, and it is not universal for semantic segmentation networks with different numbers of categories. In addition, for large-scale and long-term running SLAM systems, when an unexpected situation interrupts the semantic mapping process, the previous map information may be lost. These problems not only limit the universality of the semantic map construction method, but also reduce the efficiency and robustness of large-scale and long-term semantic mapping. Summary of the Invention

[0004] Aiming at the problems existing in the above-mentioned existing semantic map construction technology, the present invention provides a method for constructing a large-scale semantic map for a mobile robot, which uses visual odometry for state estimation in an indoor unknown environment, and simultaneously constructs a large-scale, high-precision, and reusable semantic map, with strong universality and high precision and computational efficiency.

[0005] A method for constructing a large-scale semantic map for a mobile robot, characterized by comprising the following steps:

[0006] Step 1: Use the mobile robot to obtain the current image;

[0007] Step 2: Construct a visual odometer based on the feature point method, and output the pose transformation of adjacent frame images;

[0008] Step 3: Construct and update the semantic octree map, specifically including:

[0009] Step 3.1: Based on the semantic segmentation algorithm of deep learning, output the per-pixel semantic color, semantic label, and confidence corresponding to the current image, and form semantic information;

[0010] Step 3.2: Calculate the real-world coordinates of each pixel in the current image to generate a real point cloud map, and then incorporate semantic information into the real points according to the subsequent semantic fusion algorithm to form a semantic point cloud map;

[0011] Step 3.3: Use pose transformation to convert the semantic point cloud in the semantic point cloud map to the global coordinate system, construct a voxel-based semantic octree map, and use the occupancy probability update algorithm and semantic fusion algorithm to update the semantic octree map in real time;

[0012] Step 4: Construct a three-layer memory structure model for the mobile robot, consisting of short-term memory (STM), working memory (WM), and long-term memory (LTM);

[0013] Step 5: Implement the reuse of the semantic octree map based on the node movement algorithm between WM and LTM, specifically including:

[0014] For the node movement from LTM to WM, perform loop closure detection on WM to obtain the node L with the highest loop closure probability in WM h , if the adjacent node L h of the node L with the highest loop closure probability h is in LTM, then move the adjacent node L r to WM; if there are multiple nodes L h with the highest loop closure probability in a single loop closure detection, then at most retrieve two adjacent nodes L h ;

[0015] For the node movement from WM to LTM, when the time to process one frame of image in WM exceeds the data processing time limit T WM of WM, the node with the lowest weight and the longest storage time will be transferred from WM to LTM, but it is not allowed to transfer the adjacent node L h of the node L with the highest loop closure probability r ; thereby expanding the current semantic octree map into a global semantic map.

[0016] Furthermore, the semantic fusion algorithm described in Step 3.3 is the maximum value semantic fusion algorithm or the improved Bayesian semantic fusion algorithm; the maximum value semantic fusion algorithm is applicable to a computing platform with low computing power, and the improved Bayesian semantic fusion algorithm is applicable to a computing platform with high computing power and has a higher semantic mapping accuracy.

[0017] Furthermore, the improved Bayesian semantic fusion algorithm is specifically:

[0018] The N semantic information with the highest confidence stored in each voxel constitutes the semantic set S; and the remaining semantic information is grouped into one category to form the set other_classes; the confidence other_conf of other_classes and the sum of the N confidences in S is 1;

[0019] The semantic set corresponding to the voxel formed before the current image is updated is S1, and the semantic set corresponding to the voxel formed after the current image is updated is S2. The fusion situation between the two is as follows:

[0020] If S1 and S2 have the same semantic color, then perform the Bayesian fusion algorithm on the N semantic labels of S1 and the corresponding other_classes1, and the N semantic labels of S2 and the corresponding other_classes2; otherwise, add the unique semantic information of S2 to S1, and at the same time add the unique semantic information of S1 to S2 to make S1 and S2 have the same semantic color. Then, set the confidences of the semantic information added to S1 and S2 to be β×other_conf1 and β×other_conf2 respectively, where β < 1, and reduce the confidence other_conf1 of other_classes1 to (1 - β)other_conf1, and reduce the confidence other_conf2 of other_classes2 to (1 - β)other_conf2; based on the confidences of the semantic information added to S1 and S2, perform the Bayesian fusion algorithm on the multiple semantic labels corresponding to S1 after addition and other_classes1, and the multiple semantic labels corresponding to S2 after addition and other_classes2, and the fusion result is S fusion Only keep the N semantic colors with the highest confidence.

[0021] Furthermore, generally, the value of N is not less than 2.

[0022] Furthermore, the STM, as the entry for obtaining images and pose transformations, and adding nodes, is used to calculate the similarity of consecutive frame images in time to update the weights of the nodes. When the number of nodes in the STM reaches the preset threshold S STM the node added earliest to the STM will be moved to the WM.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] The present invention proposes a method for constructing a large-scale semantic map for a mobile robot, which realizes the construction of a high-precision and reusable semantic map. Among them, a loop detection algorithm is used to reuse the previous semantic map, and local semantic octree maps are fused to generate a global semantic map, verifying the feasibility and stability of the semantic map reuse strategy; preferably, the maximum value semantic fusion algorithm or the improved Bayesian semantic fusion algorithm can be used as the semantic fusion algorithm, so that the update of the semantic map is not affected by the number of semantic categories, balancing the semantic mapping accuracy and the consumption of computing resources, and having a better semantic reconstruction effect in most scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the method for constructing a large-scale semantic map for a mobile robot proposed in Embodiment 1 of the present invention;

[0026] Figure 2 It is a schematic diagram of constructing and updating a semantic map in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solution of the present invention will be described in detail below with reference to the drawings and embodiments.

[0028] Embodiment 1

[0029] This embodiment proposes a method for constructing a large-scale semantic map for a mobile robot, including the following steps:

[0030] Step 1: Use the mobile robot to obtain the current single-frame RGB image and depth image, and a set of single-frame RGB images and depth images constitute an RGB-D image;

[0031] Step 2: Construct a visual odometer based on the feature point method, and output the pose transformation of adjacent frame images, specifically:

[0032] Step 2.1: Extract GFTT (Good Feature To Track) key points from the single-frame RGB image, use the depth image as the mask for GFTT detection, and do not extract features at invalid depth values. When extracting, limit the maximum number of features MaxFeatures, which is set to 500 in this embodiment;

[0033] Step 2.2: For short-term data association, there is no need to extract descriptors. The optical flow method directly acts on the GFTT features, thus achieving faster feature data association for key frames. For medium-term data association, calculate BRIEF (Binary Robust Independent Elementary Features) descriptors, quickly calculate the Hamming distance between descriptors through XOR operation, and then judge whether two feature points match according to the ratio of the Hamming distances of the nearest neighbor and the second nearest neighbor feature points. If they match, calculate the pose transformation according to the PnP (Perspective-n-Point) algorithm; otherwise, discard this pair of feature points.

[0034] Step 2.3: Since there will be a large amount of noise in the feature point matching process of Step 2.2, the pose transformation is locally optimized by bundle adjustment. Specifically, the Levenberg-Marquardt method is used for iterative solution, and at the same time, the camera pose and 3D road points are adjusted. Finally, the optimized pose transformation is output in the pose update module, and the visual odometer and TF tree are updated simultaneously.

[0035] Step 3: Construct and update the semantic octree map, specifically including:

[0036] Step 3.1: Train the Deeplab v3+ semantic segmentation network model based on the SUN RGBD indoor dataset. This model can recognize 37 semantic labels. Input a single-frame RGB image into the DeepLab v3+ semantic segmentation network model in real time, and output the per-pixel semantic color, semantic label, and confidence corresponding to the current single-frame RGB image, forming semantic information. Among them, the pixels with semantic colors form a semantic image.

[0037] Step 3.2: Calculate the real-world coordinates of each pixel according to the position coordinates of each pixel in the current single-frame RGB image on the image, the depth image, and the camera internal parameters to generate a real point cloud map, and then integrate the semantic information into the real points according to the adopted semantic fusion algorithm to form a semantic point cloud map.

[0038] Step 3.3: Use the pose transformation to convert the semantic point cloud to the global coordinate system, construct a voxel-based semantic octree map, and use the occupancy probability update algorithm and semantic fusion algorithm to update the semantic octree map in real time.

[0039] For the occupancy probability update algorithm, when inserting a node far from the optical center during the occupancy probability update process of the voxel, updating this node is very time-consuming. On the one hand, considering the ranging range of the depth camera and the computing power limitation of the computing platform, the algorithm only inserts points within a certain distance from the optical center of the camera. Taking the depth camera used in this embodiment as an example, the maximum range M of the inserted points is 5 meters. On the other hand, the voxel where the line connecting the optical center and the target point is located must be in an unoccupied state, otherwise the target point cannot be observed. In this embodiment, ray casting is performed on the image pixels within a limited range U (U < M), rays are projected from the optical center to the object surface, and at the same time, voxels are cleared along the rays. Generally, the ray casting range U is set to 2 meters;

[0040] The semantic fusion algorithm is the maximum value semantic fusion algorithm or the improved Bayesian semantic fusion algorithm; the maximum value semantic fusion algorithm is applicable to a computing platform with low computing power, and the improved Bayesian semantic fusion algorithm is applicable to a computing platform with high computing power and has a high semantic mapping accuracy;

[0041] The specific improved Bayesian semantic fusion algorithm is as follows:

[0042] The N semantic information with the highest confidence stored in each voxel constitutes the semantic set S; and the remaining semantic information is grouped into one category to form the set other_classes; the confidence other_conf of other_classes and the sum of the N confidences in S is 1; in this embodiment, N = 2;

[0043] The semantic set corresponding to the voxel formed before the update of the current image is S1, and the semantic set corresponding to the voxel formed after the update of the current image is S2. The fusion situation between the two is as follows:

[0044] If S1 and S2 have the same semantic color, then perform the Bayesian fusion algorithm on the N semantic tags of S1 and the corresponding other_classes1, as well as the N semantic tags of S2 and the corresponding other_classes2; otherwise, add the semantic information unique to S2 to S1, and at the same time add the semantic information unique to S1 to S2, so that S1 and S2 have the same semantic color. Then, set the confidence levels of the semantic information added to S1 and S2 to be β×other_conf1 and β×other_conf2 respectively, where β < 1, and reduce the confidence level other_conf1 of other_classes1 to (1 - β)other_conf1, and reduce the confidence level other_conf2 of other_classes2 to (1 - β)other_conf2; based on the confidence levels of the semantic information added to S1 and S2, perform the Bayesian fusion algorithm on the multiple semantic tags corresponding to S1 after addition and other_classes1, as well as the multiple semantic tags corresponding to S2 after addition and other_classes2, and the fusion result is S fusion Only keep the N semantic colors with the highest confidence levels;

[0045] Step 4: Construct a three-layer memory structure model for the mobile robot, which consists of STM, WM, and LTM; where, the STM is the entry for acquiring images and pose transformations, as well as adding nodes, and is used to calculate the similarity of consecutive frame images in time, so as to update the weights of the nodes. When the number of nodes in the STM reaches the preset threshold S STM the node that was added to the STM earliest will be moved to the WM; the nodes in the WM are mainly used for loop detection, and the time limit for data processing indirectly affects the size of the WM. When the time for the WM to process new data reaches the data processing time limit T WM the nodes with lower memory management weights in the graph optimization will be transferred from the WM to the LTM, so as to keep the size of the WM almost unchanged; the LTM stores the nodes that have not been used for a long time, and these nodes are not used for loop detection and graph optimization, but some nodes in the LTM may be transferred to the WM when reusing the semantic map;

[0046] The nodes in the three-layer memory structure model save the pose transformation of the mobile robot, visualization information (RGB image, depth image, and semantic image), the bag-of-words vectors for loop detection, and the weights for memory management; the edges in the three-layer memory structure model store the rigid geometric transformations between the nodes, and the edges can be divided into adjacent edges and loop edges. The adjacent edge is the geometric transformation between the current node and the previous node, and a loop edge is added between two nodes when a new node is detected to have a loop with a certain node in the graph optimization;

[0047] To ensure the running efficiency, the maximum number of nodes S of the STM needs to be set according to the speed of the mobile robot and the frequency of receiving pose transformations STM = 10; Secondly, the time limit T for data processing WM indirectly affects the size of the WM. Set the time limit T for processing the working memory WM = 700 milliseconds. A higher value of T WM means that more nodes will be stored in the WM. The WM will store more loop closure hypotheses and can represent the scene more completely;

[0048] Step 5: Based on the node movement algorithm between the WM and the LTM, realize the reuse of the semantic octree map, specifically including:

[0049] For the node movement from the LTM to the WM, perform loop closure detection on the WM to obtain the node L with the highest loop closure probability in the WM h , if the adjacent node L h of the node L with the highest loop closure probability r is in the LTM, then move the adjacent node L r to the WM; if there are multiple nodes L h with the highest loop closure probability in a loop closure detection, then at most retrieve two adjacent nodes L r ;

[0050] For the node movement from the WM to the LTM, when the time for processing one frame of image in the WM exceeds the data processing time limit T WM = 700 milliseconds, the node with the lowest weight and the longest storage time will be transferred from the WM to the LTM, but it is not allowed to transfer the adjacent node L h of the highest loop closure hypothesis node L r after loop closure detection;

[0051] Furthermore, expand the current semantic octree map into a global semantic map. If there is local semantic information inconsistency between two local semantic octree maps, this method takes the latest observed semantic information as the standard to avoid double images in the global map;

[0052] Among them, to efficiently construct a complete visual dictionary, the number of feature points for loop closure detection is limited not to exceed the set threshold T fmax , when the number of extracted features is too small, that is, the ratio of extracted features is less than T bad , the generated bag-of-words vector is not used for loop closure detection. In this embodiment, T fmax is set to 500, and T bad is set to 0.25.

[0053] Although the above description of the illustrative embodiments of the present invention is provided for those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

Claims

1. A method for constructing a large-scale semantic map for a mobile robot, characterized in that It includes the following steps: Step 1: Use a mobile robot to obtain the current image; Step 2: Construct a visual odometer based on the feature point method and output the pose transformation of adjacent frame images; Step 3: Construct and update the semantic octree map, specifically including: Step 3.1: Based on the semantic segmentation algorithm of deep learning, output the per-pixel semantic color, semantic label, and confidence corresponding to the current image, forming semantic information; Step 3.2: Calculate the real-world coordinates of each pixel in the current image, generate a real point cloud map, and then integrate the semantic information into the real points according to the subsequent semantic fusion algorithm to form a semantic point cloud map; Step 3.3: Use the pose transformation to convert the semantic point cloud to the global coordinate system, construct a voxel-based semantic octree map, and use the occupancy probability update algorithm and semantic fusion algorithm to update the semantic octree map in real time; The semantic fusion algorithm is an improved Bayesian semantic fusion algorithm, specifically: The N semantic information with the highest confidence stored in each voxel constitutes the semantic set S, and the remaining semantic information is grouped into one category to form the set other_classes; the confidence other_conf of other_classes and the sum of the N confidences in S are 1; The semantic set corresponding to the voxel formed before the update of the current image is S1, and the semantic set corresponding to the voxel formed after the update of the current image is S2. The fusion situation of the two is as follows: If S1 and S2 have the same semantic color, then perform the Bayesian fusion algorithm on the N semantic labels of S1 and the corresponding other_classes1, and the N semantic labels of S2 and the corresponding other_classes2; otherwise, add the unique semantic information of S2 to S1, and at the same time add the unique semantic information of S1 to S2 to make S1 and S2 have the same semantic color. Then, set the confidence of the semantic information added to S1 and S2 to be β×other_conf1 and β×other_conf2 respectively, where β < 1, and reduce the confidence other_conf1 of other_classes1 to (1 - β)other_conf1, and reduce the confidence other_conf2 of other_classes2 to (1 - β)other_conf2; based on the confidence of the semantic information added to S1 and S2, perform the Bayesian fusion algorithm on the multiple semantic labels corresponding to S1 after addition and other_classes1, and the multiple semantic labels corresponding to S2 after addition and other_classes2. The fusion result only retains the N semantic colors with the highest confidence; Step 4: Construct a three-layer memory structure model of the mobile robot, which consists of STM, WM, and LTM; Step 5: Based on the node movement algorithm between WM and LTM, realize the reuse of the semantic octree map, specifically including: For the node movement from LTM to WM, loop detection is performed on WM to obtain the node L with the highest loop probability in WM h , if the adjacent node L h of node L h is in LTM, then move the adjacent node L r to WM; if there are multiple nodes L h in one loop detection, then at most take back two adjacent nodes L h ; For the node movement from WM to LTM, when the time for processing one frame of image in WM exceeds the data processing time limit T of WM WM the node with the lowest weight and the longest storage time will be transferred from WM to LTM, but the adjacent node L h of node L r is not allowed to be transferred; and then the current semantic octree map is extended to a global semantic map.

2. The method for constructing a large-scale semantic map for a mobile robot according to claim 1, wherein The STM serves as the entry point for acquiring images, pose transformation, and adding nodes. It calculates the similarity of consecutive frame images over time to update the weights of the nodes. When the number of nodes in the STM reaches the preset threshold S STM the nodes that were added to the STM earliest will be moved to the WM.

Images