A method for constructing three-dimensional semantic maps for indoor mobile robots

Through the combination of RGB-D depth camera and ORB-SLAM2 and YOLO V5 networks, the problem of difficult to integrate semantic information in the existing technology is solved, and a rich three-dimensional semantic map is built, which improves the accuracy of robot map construction.

CN115035260BActive Publication Date: 2024-11-05HARBIN ENG UNIV

Patent Information

Application Number
CN202210594240.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-27
Publication Date
2024-11-05
Estimated Expiration
2042-05-27

Smart Images

  • Figure CN115035260B_ABST
    Figure CN115035260B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for constructing a three-dimensional semantic map of an indoor mobile robot. The robot key frame information and posture are obtained by running the ORB-SLAM2 algorithm through the RGB-D depth camera carried by the robot, and the RGB image and the depth image of the key frame are formed into a spatial point cloud by back-projection; the spatial point cloud of the key frame is filtered and spliced ​​according to the posture to form a three-dimensional dense point cloud map; the RGB image of the key frame is used for target recognition using the YOLO V5 network to obtain 2D semantic information, and 3D semantic labels are obtained according to back-projection; the dense point cloud map is segmented by a point cloud segmentation algorithm; the obtained semantic information is merged with the segmentation result of the dense point cloud map to construct a 3D semantic map. The present invention forms a three-dimensional map with richer information by processing the RGB-D information collected by the RGB-D depth camera, and merging the semantic information with the SLAM result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of robot semantic map construction, and relates to a robot map construction method, in particular to a method for constructing a three-dimensional semantic map of an indoor mobile robot. Background Art

[0002] With the rapid development of intelligent technology, the mobile robot industry has also ushered in a booming development. Advanced mobile robots are equipped with many sensors with specific functions, and can complete designated tasks in various scenarios without direct human operation. Among them, for intelligent robots used in indoor scenarios, they generally use the robot's autonomous exploration method to complete the map construction task.

[0003] At present, typical semantic construction methods for mobile robots mainly include methods based on visual images and methods based on point cloud processing. For example, Zhang Cheng et al., in the paper "Semantic SLAM Algorithm in Dynamic Environment", used monocular image sequences to carry out research on semantic segmentation, visual odometry, etc., and finally constructed a global semantic map; for another example, Zhang Zheng et al., in the paper "Robust Semantic SLAM Algorithm with Variable Structure", studied the use of point cloud images to carry out research on the robustness of semantic SLAM.

[0004] The above existing technologies have the following problems in constructing semantic maps for mobile robots: first, visual images can only provide RGB color information, and it is difficult to extract three-dimensional spatial information such as point clouds; second, it is not easy to obtain the semantic information of the represented objects from point cloud information, and it is difficult to integrate semantics with SLAM mapping and positioning results. Summary of the invention

[0005] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a method for constructing a three-dimensional semantic map for an indoor mobile robot, by processing the RGB-D information collected by an RGB-D depth camera, fusing the semantic information with the SLAM results, and forming a three-dimensional map with richer information.

[0006] In order to solve the above technical problems, a method for constructing a three-dimensional semantic map of an indoor mobile robot according to the present invention comprises the following steps:

[0007] Step 1: Run the ORB-SLAM2 algorithm through the RGB-D depth camera carried by the robot to obtain the robot's key frame information and corresponding posture, and form a single-frame spatial point cloud by back-projecting the RGB image and depth image corresponding to each key frame;

[0008] Step 2: Filter the spatial point cloud corresponding to the key frame, and splice it according to the posture corresponding to the key frame to form a three-dimensional dense point cloud map;

[0009] Step 3: Use the YOLO V5 network to identify the RGB image corresponding to the key frame, obtain the 2D semantic information in the key frame, and obtain the 3D semantic label based on back projection;

[0010] Step 4: Segment the dense point cloud map through the point cloud segmentation algorithm;

[0011] Step 5: Combine the obtained semantic information with the segmentation result of the dense point cloud map to construct a 3D semantic map.

[0012] Furthermore, step 1 includes:

[0013] In the Tracking thread of the ORB-SLAM2 system, the processing of key frames is added to obtain the spatial point cloud data corresponding to the key frames. Specifically, the following operations are performed on all pixels in the key frames to obtain the point cloud data of the key frames in the camera coordinate system: Assume that the coordinates of a pixel point P′ in the image are [u, v] T , the camera's intrinsic parameter matrix is ​​K, and the depth data corresponding to the pixel is d, then the pixel is converted into a three-dimensional point P in the camera coordinate system c (X′,Y′,Z′) is:

[0014]

[0015] Get the three-dimensional point P c After obtaining the spatial coordinates of the pixel point P′, the RGB information is obtained as P c RGB information, perform the above operation on all pixels in the key frame, and obtain the point cloud data of the key frame in the camera coordinate system through the point cloud constructor of the PCL library;

[0016] Then convert the point cloud data of the keyframe in the camera coordinate system into the point cloud data in the world coordinate system:

[0017] Let P c The point converted to the world coordinate system is P w (X,Y,Z),P w (X,Y,Z) satisfies:

[0018]

[0019] Among them, T cw is the camera pose.

[0020] Furthermore, the filtering process of the spatial point cloud corresponding to the key frame in step 2 includes:

[0021] Use the outlier removal filter in the PCL point cloud library to remove outliers in the point cloud data, and calculate the average distance from each point in the spatial point cloud to all adjacent points. The average distance is within the outlier distance threshold d pt Points outside the range are identified as outliers and removed from the data. Then, the point cloud data is downsampled by voxel grid filtering.

[0022] Furthermore, the outlier distance threshold d pt Specifically:

[0023] Assume that the distances of all points in the spatial point cloud follow a normal distribution (μ,σ 2 ), the threshold parameter for judging whether it is an outlier is H out , then the outlier distance threshold d pt for:

[0024] d pt =μ+H out ·σ.

[0025] Furthermore, in step 2, stitching the key frames corresponding to the poses to form a three-dimensional dense point cloud map includes:

[0026] In the ORB-SLAM2 system, the LoopClosing thread optimizes the key frame poses in the detected closed loop, extracts the key frames where the camera poses change after the closed-loop optimization, deletes the point cloud data corresponding to the key frames in the world coordinate system, regenerates the corresponding point cloud data based on the camera poses after the closed-loop optimization, and then projects it into the world coordinate system to construct a three-dimensional dense point cloud map.

[0027] Furthermore, step 3 includes:

[0028] Step 3.1: Create a dataset based on the YOLO V5 network model

[0029] Take pictures of the target object to get the data set, use the labeling program Labelimg to label the collected data set, select the training set and test set from the captured pictures respectively; select pictures containing the target object from the COCO data set as the training set and test set respectively; configure the data and set the model training parameters for the YOLO V5 network, the data configuration includes the number of object categories and the name of the object category, use the training set to train the YOLO V5 network model parameters, and use the test set to verify the network model;

[0030] Step 3.2: Extract 2D object semantics and construct 3D semantic labels

[0031] The YOLO V5 network uses the RGB-D depth camera to collect RGB image information and obtains the 2D recognition box in the RGB image as 2D target semantic information in real time;

[0032] In the Tracking thread of ORB-SLAM2, an interface is added to read the key frames generated in real time. The trained YOLO V5 network model is used to perform target detection on each newly generated key frame to obtain the pixel coordinates of the center point of the target object and the length and width of the bounding box. Then, when generating spatial point cloud data based on the RGB image and depth image corresponding to the key frame, the RGB information of all pixels in the 2D bounding box of the target object is changed to the set color information. At this time, the single-frame point cloud data generated according to the key frame has a 3D semantic label. The key frame point cloud data is then combined with the result of pose estimation to back-project into the world coordinate system to complete point cloud stitching, and a global dense point cloud map with 3D semantic labels is obtained, that is, color information is added to the point cloud of the target object in the point cloud map.

[0033] Furthermore, in step 4, segmenting the dense point cloud map by the point cloud segmentation algorithm includes:

[0034] Step 4.1: Use the supervoxel clustering segmentation method to segment the dense point cloud map and convert the point cloud into a surface structure. The segmentation result is represented by an adjacency graph G = {v, ε}, where v i ∈v is the surface block obtained by segmentation, and ε represents the surface blocks connected to each other (v i ,v j ), each surface block contains a centroid c i and a normal vector n i ;

[0035] Step 4.2: After obtaining the surface block, use RANSAC to process the surface block to generate candidate planes PC = {pc1, pc2, ..., pc m}, then calculate the surface block centroid c i To flat pc m The distance d(c i ,pc m ), set the threshold δ, and get all the plane pc m The surface blocks within the distance δ are named Π={v i ∈V|d(c i ,pc m )<δ}, definition:

[0036]

[0037] Among them, η represents the constraint condition for distinguishing foreground objects from background, D(pc m ) represents the weight of the point cloud plane, and the minimum energy P of the graph segmentation problem* for:

[0038]

[0039] Where E(P) is the energy of the fit, satisfying:

[0040]

[0041] Where V and E are the sets of vertices and edges in the graph, respectively, corresponding to the adjacency graph G = {v, ε} of cluster segmentation.

[0042] Furthermore, step 5 includes:

[0043] The bounding box of the target object point cloud is obtained according to the 2D recognition box, where the width value b of the bounding box is the maximum depth difference corresponding to all the pixels in the 2D recognition box. Assume that all the pixels in the 2D recognition box are a set P = {P1, P2, P3..., P n},but

[0044] b=max(Depth(P i )-Depth(P j ))

[0045] Among them, P i ,P j ∈P,Depth(P i ) is the pixel point P i The corresponding depth value is back-projected into the world coordinate system using 2D pixel points, and the length and width of the 2D recognition box are mapped into the world coordinate system to obtain the length and height of the bounding box;

[0046] Then the results of point cloud segmentation are tested. Only when all point clouds in a cluster are located in the bounding box, the random color attribute label of the cluster is replaced with the 3D semantic label obtained in step 3 to obtain a semantic map.

[0047] Furthermore, the RGB-D depth camera adopts Kinect V2 camera.

[0048] Furthermore, the YOLO V5 network adopts the YOLOv5s network.

[0049] Beneficial effects of the present invention: The present invention obtains RGB-D information of the indoor environment by carrying an RGB-D depth camera (such as a Kinect camera) on a robot, and completes target recognition based on the key frames generated by the ORB-SLAM2 system through the trained YOLO v5s network model to obtain semantic information and then convert it into 3D semantic labels. A three-dimensional dense point cloud map is generated based on the RGB image and depth image corresponding to the key frame combined with the pose information of the key frame. On this basis, the point cloud map is segmented, and the semantic labels are merged with the dense point cloud map to construct an information-rich three-dimensional semantic map. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of the present invention;

[0051] Figure 2 Constructing flow charts for dense point clouds;

[0052] Figure 3 It is a dense point cloud map;

[0053] Figure 4 This is a sample of some of the collected data sets;

[0054] Figure 5 Calibration of data samples;

[0055] Figure 6 Real-time detection results for YOLO v5s;

[0056] Figure 7 To generate a point cloud label flow chart;

[0057] Figure 8 This is a point cloud label map in a real scene;

[0058] Fig. 9 It is the supervoxel clustering segmentation result;

[0059] Fig.10 This is the point cloud segmentation result map;

[0060] Fig.11 is the fused semantic map. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0062] The flowchart of the present invention is as follows Figure 1As shown, the present invention uses Kinect V2 camera as a visual sensor, obtains key frames in the process of camera movement based on ORB-SLAM 2 algorithm, obtains the camera posture by matching feature points in key frames, optimizing the backend, etc., and then forms a spatial point cloud by back-projecting the corresponding RGB image and depth image of each key frame. After filtering, the image is spliced ​​with the camera posture information to form a three-dimensional dense point cloud map. At the same time, the YOLO V5 network model is used to perform target recognition on the RGB image corresponding to the key frame to obtain the semantic information of the target, and project it to the spatial point cloud to realize the construction of 3D semantic labels. Then, the point cloud segmentation algorithm is used to segment the dense point cloud map. Finally, the obtained semantic information is integrated with the segmentation result of the dense point cloud map to construct a 3D semantic map.

[0063] 1. Improve ORB-SLAM2 to construct 3D dense point cloud

[0064] The present invention achieves the goal of obtaining dense point cloud by improving some threads in ORB-SLAM2. The specific process is as follows: Figure 2 The main steps are as follows:

[0065] Step 1: Back-projection of 2D pixels to 3D point cloud data

[0066] The Tracking thread in the ORB-SLAM2 system is modified to add keyframe processing to obtain the 3D point cloud data corresponding to the keyframe. The principle of this process is to back-project the two-dimensional pixel coordinates and depth data of the environment into the world coordinate system through the camera's internal and external parameters. Suppose the coordinates of a pixel point P′ in the image are [u, v] T , the camera's intrinsic parameter matrix is ​​K, and the depth data corresponding to the pixel is d; then the pixel is converted into a three-dimensional point P in the camera coordinate system c (X′, Y′, Z′) can be expressed as follows:

[0067]

[0068] Get the three-dimensional point P c After obtaining the spatial coordinates of the pixel point P′, the RGB information is obtained as P c RGB information, perform the above operation on all pixels in the key frame, and obtain the point cloud data of the key frame in the camera coordinate system through the point cloud constructor of the PCL library. After obtaining the point cloud data corresponding to the key frame, convert it into a point cloud in the world coordinate system. Let P c The point converted to the world coordinate system is P w (X, Y, Z), the process can be expressed as follows, where T cw is the camera pose:

[0069]

[0070] Step 2: Dense point cloud data filtering

[0071] The present invention uses the outlier removal filter in the PCL point cloud library to remove abnormal noise points in the point cloud data. The point cloud obtained after the outlier removal filtering will be smoother and suitable for subsequent point cloud clustering. Then, the point cloud data is downsampled by the voxel grid filtering method. When the outlier removal filtering is performed, for each point, the average distance from it to all adjacent points is calculated. The result obtained should obey the Gaussian distribution, and its shape is determined by the Gaussian distribution parameters (μ, σ 2 ) determines that points with average distances outside the standard range are considered outliers and removed from the data.

[0072] Assume that the distances of all points in the point cloud follow a normal distribution (μ,σ 2 ), the threshold parameter for judging whether it is an outlier is H out , then the outlier distance threshold d pt It can be expressed as:

[0073] d pt =μ+H out ·σ (3)

[0074] That is, the average distance is in the interval (0,d pt ) will be considered as outliers and filtered out.

[0075] The present invention conducts experiments based on a hardware platform (Intel Core i5-10400F CPU, NVIDIA GTX1650), and adjusts the resolution parameter of the voxel grid filter to 0.01, that is, when the voxel grid size is 1cm*1cm*1cm, the best operation effect can be obtained.

[0076] Step 3: Point cloud stitching

[0077] In the present invention, the global dense point cloud data obtained in step 1 is directly based on the camera pose T of each key frame. cw However, the LoopClosing thread in the ORB-SLAM2 system will optimize the key frame poses in the entire closed loop according to the detected closed loop, and the pose estimation results of some key frames will change.

[0078] In order to avoid ghosting in the constructed dense point cloud map, the present invention extracts the key frame where the camera posture changes after closed-loop optimization, then deletes the point cloud data corresponding to the key frame in the world coordinate system and regenerates the corresponding point cloud data according to the posture after closed-loop optimization, projects it into the world coordinate system, completes the point cloud splicing work after loop detection, and finally constructs a 3D dense point cloud map based on the ORB-SLAM2 system. Figure 3 shown.

[0079] 2. 3D semantic label construction based on YOLO v5s network model

[0080] The present invention needs to perform target recognition on the key frames generated by the Tracking thread in the ORB SLAM2 system, and then map the obtained 2D semantic information to the spatial point cloud data to complete the construction of 3D semantic labels. The specific steps are as follows:

[0081] Step 1: Create a dataset based on the YOLO v5s network model

[0082] This patent uses a camera to capture common objects in the laboratory, mainly including computer equipment, books and other common items. Some of the captured images are as follows: Figure 4 As shown, the collected data sets are annotated and labeled using the labeling program Labelimg, such as Figure 5 As shown, for example, the process of labeling "display" is marked with the label "tv". 500 images are selected from the captured images as training sets and 200 images are selected as test sets. In addition, 1500 images containing objects in indoor scenes are selected from COCO as training sets and 300 images are selected as test sets. In order to be used with the COCO dataset, the labeling files are stored in json file format. In the present invention, there are 35 common objects in indoor environments.

[0083] Before training, you need to modify the configuration files of the YOLO v5s data and model, change the number of object categories nc in the two files to 35, and change the names in the category list to the names of the object categories. Here, the training set has 2,000 images and the test set has 500 images. After preparing the data set, the YOLO v5s network model parameters are trained. The hyperparameter configuration of the training model is shown in Table 1.

[0084] Table 1 Parameter settings of YOLO v5s network model

[0085]

[0086] Step 2: Extract 2D target semantics and construct 3D semantic labels

[0087] After YOLO v5s network training is completed, the Kinect V2 camera is used to collect the corresponding RGB image information in real time to realize the real-time detection function. The results are as follows Figure 6 As shown, the 2D target semantic information in the RGB image can be obtained, which can meet the requirement of the present invention for real-time construction of 3D semantic labels.

[0088] The process of mapping 2D semantic information to 3D semantic labels is as follows Figure 7 As shown in the figure, first, add an interface in the Tracking thread of ORB-SLAM2. Based on the trained YOLO v5s network, perform target detection on each newly generated key frame to obtain the pixel coordinates of the center point of the target object and the length and width of the bounding box. Then, when generating spatial point cloud data based on the RGB image and depth image corresponding to the key frame, the RGB information of all pixels in the 2D bounding box of the target object needs to be changed to the set color information. At this time, the single-frame point cloud data generated according to the key frame has a semantic label. The obtained key frame point cloud data is then combined with the result of pose estimation to back-project into the world coordinate system to complete point cloud splicing, and a global dense point cloud map with 3D semantic labels is obtained, that is, color information is added to the point cloud of the target object in the point cloud map.

[0089] The keyframe generated in ORB-SLAM2 is used as input to obtain the keyframe image with 2D bounding box and confidence, and then the corresponding 3D semantic label in the keyframe is obtained by back-projection. The result is as follows: Figure 8 As shown, the point cloud of the display is set to red, the point cloud of the keyboard is set to green, and the point cloud of the mouse is set to blue.

[0090] 3. Dense point cloud segmentation

[0091] The present invention performs point cloud segmentation on the constructed dense point cloud map in order to obtain a scene that can be understood by the robot. At the same time, according to the segmentation results of the dense point cloud map, an accurate target point cloud is obtained, which is convenient for subsequent combination with 3D semantic labels. First, the original dense point cloud map is segmented using a segmentation method based on supervoxel clustering. According to the similarity of the points, the irregular point cloud is converted into a surface structure. The patches formed after supervoxel segmentation have a centroid and a normal vector. Point cloud segmentation can be defined as a graph segmentation problem, and then the graph cut method is used for further segmentation. The detailed steps are as follows:

[0092] Step 1: Supervoxel clustering segmentation

[0093] Supervoxel segmentation uses the spatial octree structure of the point cloud and clusters it through the SupervoxelClustering class in the PCL library. According to the segmentation results, the centroid of the octree structure (voxel_centroid_cloud) is labeled and given a random color stored in the colored_voxel_cloud parameter.

[0094] The segmentation principle uses the regional growth of k-means clustering to segment the point cloud. The process of supervoxel clustering is similar to the crystallization process, just like the crystallization of polycrystalline nuclei caused by supersaturation of the solution. The supervoxel clustering segmentation method will evenly distribute the crystal nuclei used for growth in the entire space according to certain rules, and then set the crystal nucleus distance and the minimum grain size, and the smaller grains will be absorbed by the larger grains. In the supervoxel segmentation process, the expansion of the seed point is determined by the feature distance, which takes into account the feature space calculation of space, color and normal vector. The distance formula in supervoxel clustering segmentation is as follows:

[0095]

[0096] In the above formula: D c Represents the degree of color difference, D n Represents the degree of difference in normal direction, D s Indicates the degree of difference in distance between point clouds. c 、w s 、w n Represent the weight of each variable. By searching around the crystal nucleus, the voxel with the smallest D value is found to be the most similar, which represents the next crystal nucleus to grow.

[0097] The present invention selects Kinect V2 camera as the sensor and sets appropriate parameters: voxel_resolution=0.008 (voxel size, resolution of spatial octree), seed_resolution=0.5 (seed resolution). The weights of color, spatial distance and normal vector are adjusted according to the laboratory environment, and the supervoxel clustering segmentation results are obtained as shown in Fig. Fig. 9 shown.

[0098] Step 2: Point cloud segmentation based on geometric information

[0099] The supervoxel segmentation result obtained in the previous step is represented by the adjacency graph G = {v, ε}, where v i ∈v is the surface block obtained by segmentation, and ε represents the surface blocks connected to each other (v i ,v j ). After supervoxel segmentation, each surface block has a centroid c i , and a normal vector n i, at this time, scene segmentation can be regarded as a graph segmentation problem.

[0100] Assume that there are K actual planes {s1,s2,…,s k}, and these planes have been divided into surface blocks. Here we define a variable {b i}1 N , b i ∈[0,K],b i =K means the surface block belongs to plane s k , N represents the actual plane s k is divided into N surface blocks, such as Fig. 9 The desktop is divided into two clusters. If we can extract the planes of all objects in the point cloud map and assign surface blocks to them, we can get a more accurate point cloud segmentation result.

[0101] After obtaining the surface block, RANSAC is used to process the surface block to generate a candidate plane PC = {pc1, pc2, ..., pc m}, calculate d(c i ,pc m ), which is the centroid of the surface block c i To flat pc m Adding a threshold δ, we can get all the distances to the plane pc m The surface blocks within the distance δ are named ∏={v i ∈V|d(c i ,pc m )<δ}. Define the following formula:

[0102]

[0103] In the above formula: η represents the constraint condition for distinguishing foreground objects from background, D(pc m ) represents the weight of the point cloud plane. In the experiment, η=40 and δ=4(cm). The minimum energy P of the graph segmentation problem * It can be expressed as follows:

[0104]

[0105] In the above formula, E(P) is the energy of the fitting, which is shown in the following formula:

[0106]

[0107] Where V and E are the sets of vertices and edges in the graph, respectively, corresponding to the adjacency graph G = {v, ε} of cluster segmentation.

[0108] Based on the point cloud minimum cut theory, the edge that satisfies the energy minimization is cut off, that is, the adjacent surface blocks can be merged according to the constraint conditions to obtain the point cloud segmentation result. The purpose of segmentation at this time is to merge the clusters belonging to the same object in the clustering results. The segmentation result is as follows Fig.10 As shown, the segmentation effect of target objects such as displays, books, cups, bottles, etc. is very good, and the merging effect of background objects such as tables, lockers, floors, etc. is also very good, which meets the requirements of the present invention.

[0109] 4. Integrate semantic information and build semantic maps

[0110] After completing the dense point cloud segmentation, the present invention combines the segmentation results with the 3D semantic labels, optimizes the segmentation results through semantic information, and can segment out the targets that cannot be extracted in the point cloud segmentation; at the same time, semantic information can be added to each segmented target to build a more accurate semantic map. After the point cloud is segmented, each cluster has an attribute label. After replacing the attribute label of the cluster with the semantic label of the specific target, the specific object in the point cloud has semantic information, thereby constructing a semantic map. The specific implementation steps are as follows:

[0111] Step 1: Storage of semantic tags

[0112] After the dense point cloud map is segmented, the point clouds of different colors represent the point cloud set and spatial sampling area in each supervoxel, and this color is the attribute label of the cluster. However, the attribute label at this time is only used to distinguish the cluster results, which is a random label without any meaning. This label is also the attribute label of each cluster in the final point cloud segmentation result.

[0113] In the second part, the present invention uses YOLO v5s to perform target recognition on the key frames generated in ORB-SLAM2, and adds color labels with semantic information to the results of point cloud segmentation based on the 3D semantic labels obtained from the recognition results, thereby completing the storage of object semantic labels.

[0114] However, at this time, all pixels in the 2D target recognition bounding box are directly mapped to the 3D semantic labels in combination with the depth data of the Kinect camera. In the complex environment of the laboratory, the target recognition bounding box is not accurate enough, which can easily lead to the addition of wrong semantic information to the surrounding point cloud of the target object, such as Figure 8 As shown in Figure 3, there is a situation where semantic labels are incorrectly added to the surrounding point cloud of the target object.

[0115] Step 2: Fusion of semantic labels and point cloud bounding boxes

[0116] In order to obtain a more accurate semantic map, the present invention obtains the bounding box of the target object point cloud based on the 2D recognition box, where the width value b of the bounding box is the maximum depth difference corresponding to all the pixels in the 2D recognition box. Assume that all the pixels in the 2D recognition box are a set P = {P1, P2, P3..., P n},but

[0117] b=max(Depth(P i )-Depth(P j )) (8)

[0118] Where P i ,P j ∈P,Depth(P i ) is the pixel point P i The length and height of the bounding box are obtained by mapping the length and width of the 2D recognition box. The principle is to back-project the 2D pixel points into the world coordinate system, which will not be described here.

[0119] Next, the present invention will test the results of point cloud segmentation. Only when all point clouds in a cluster are located in the bounding box, specific semantic information will be added, that is, specific color attributes will be added to it, thereby obtaining a more accurate semantic map. Fig.11 and Fig.10 As shown in Figure 2, by fusing the bounding box obtained by semantic labels with the point cloud segmentation results, the object TV can be segmented more completely, and the clustering of the actual object edge point cloud also successfully adds the correct semantic information; Figure 8 Compared with adding semantic information to the point cloud directly through the bounding box obtained by 2D object recognition, this method can also effectively avoid adding semantic information to the point cloud around the target. At the same time, combining semantic information can also improve the point cloud segmentation effect, such as Fig.11 In the above example, the added semantic information changes the color attribute and the re-clustering can segment small objects such as keyboard and mouse on the desktop. According to the above two steps, the following Fig.11 The three-dimensional semantic map shown.

Claims

1. A method for constructing a three-dimensional semantic map for an indoor mobile robot. It is characterized in that The following steps are involved: Step 1: Run the ORB-SLAM2 algorithm through the RGB-D depth camera carried by the robot to obtain the robot's key frame information and corresponding posture, and form a single-frame spatial point cloud by back-projecting the RGB image and depth image corresponding to each key frame; Step 2: Filter the spatial point cloud corresponding to the key frame, and splice it according to the posture corresponding to the key frame to form a three-dimensional dense point cloud map; In the Tracking thread of the ORB-SLAM2 system, the processing of key frames is added to obtain the spatial point cloud data corresponding to the key frames. Specifically, the following operations are performed on all pixels in the key frames to obtain the point cloud data of the key frames in the camera coordinate system: Assume that the coordinates of a pixel point P′ in the image are [u, v] T , the camera's intrinsic parameter matrix is ​​K, and the depth data corresponding to the pixel is d, then the pixel is converted into a three-dimensional point P in the camera coordinate system c (X′,Y′,Z′) is: Get the three-dimensional point P c After obtaining the spatial coordinates of the pixel point P′, the RGB information is obtained as P c RGB information, perform the above operation on all pixels in the key frame, and obtain the point cloud data of the key frame in the camera coordinate system through the point cloud constructor of the PCL library; Then convert the point cloud data of the keyframe in the camera coordinate system into the point cloud data in the world coordinate system: Let P c The point converted to the world coordinate system is P w (X,Y,Z),P w (X,Y,Z) satisfies: Among them, T cw is the camera pose; Step 3: Use the YOLO V5 network to identify the RGB image corresponding to the key frame, obtain the 2D semantic information in the key frame, and obtain the 3D semantic label based on back projection; Step 3.1: Create a dataset based on the YOLO V5 network model Take pictures of the target object to get the data set, use the labeling program Labelimg to label the collected data set, select the training set and test set from the captured pictures respectively; select pictures containing the target object from the COCO data set as the training set and test set respectively; configure the data and set the model training parameters for the YOLO V5 network, the data configuration includes the number of object categories and the name of the object category, use the training set to train the YOLO V5 network model parameters, and use the test set to verify the network model; Step 3.2: Extract 2D object semantics and construct 3D semantic labels The YOLO V5 network uses the RGB-D depth camera to collect RGB image information and obtains the 2D recognition box in the RGB image as 2D target semantic information in real time; In the Tracking thread of ORB-SLAM2, an interface is added to read the key frames generated in real time. The trained YOLOV5 network model is used to perform target detection on each newly generated key frame to obtain the pixel coordinates of the center point of the target object and the length and width of the bounding box. Then, when generating spatial point cloud data based on the RGB image and depth image corresponding to the key frame, the RGB information of all pixels in the 2D bounding box of the target object is changed to the set color information. At this time, the single-frame point cloud data generated according to the key frame has a 3D semantic label. The key frame point cloud data is then combined with the result of pose estimation to back-project into the world coordinate system to complete point cloud stitching, and a global dense point cloud map with 3D semantic labels is obtained, that is, color information is added to the point cloud of the target object in the point cloud map. Step 4: Segment the dense point cloud map through the point cloud segmentation algorithm; Step 5: Combine the obtained semantic information with the segmentation result of the dense point cloud map to construct a 3D semantic map.

2. A method for constructing a three-dimensional semantic map of an indoor mobile robot according to claim 1, Features: The filtering process of the spatial point cloud corresponding to the key frame in step 2 includes: Use the outlier removal filter in the PCL point cloud library to remove outliers in the point cloud data, and calculate the average distance from each point in the spatial point cloud to all adjacent points. The average distance is within the outlier distance threshold d pt Points outside the bounds are identified as outliers and removed from the data, and then the point cloud data is downsampled using the voxel grid filtering method.

3. A method for constructing a three-dimensional semantic map of an indoor mobile robot according to claim 2, Features: The outlier distance threshold d pt Specifically: Assume that the distances of all points in the spatial point cloud follow a normal distribution (μ,σ 2 ), the threshold parameter for judging whether it is an outlier is H out , then the outlier distance threshold d pt for: d pt =μ+H out ·s.

4. A method for constructing a three-dimensional semantic map of an indoor mobile robot according to claim 1, Features: Step 2, stitching together the key frames to form a three-dimensional dense point cloud map, includes: In the ORB-SLAM2 system, the LoopClosing thread optimizes the key frame poses in the detected closed loop, extracts the key frames where the camera poses change after the closed-loop optimization, deletes the point cloud data corresponding to the key frames in the world coordinate system, regenerates the corresponding point cloud data based on the camera poses after the closed-loop optimization, and then projects it into the world coordinate system to construct a three-dimensional dense point cloud map.

5. A method for constructing a three-dimensional semantic map of an indoor mobile robot according to claim 1, Features: Step 4, segmenting the dense point cloud map by using a point cloud segmentation algorithm, includes: Step 4.1: Use the supervoxel clustering segmentation method to segment the dense point cloud map and convert the point cloud into a surface structure. The segmentation result is represented by an adjacency graph G = {v, ε}, where v i ∈v is the surface block obtained by segmentation, and ε represents the surface blocks connected to each other (v i ,v j ), each surface block contains a centroid c i and a normal vector n i ; Step 4.2: After obtaining the surface block, use RANSAC to process the surface block to generate a candidate plane PC = {pc 1 ,pc 2 ,…,pc m }, then calculate the surface block centroid c i To flat pc m The distance d(c i ,pc m ), set the threshold δ, and get all the plane pc m The surface blocks within the distance δ are named Π={v i ∈V|d(c i ,pc m )<δ}, definition: Among them, η represents the constraint condition for distinguishing foreground objects from background, D(pc m ) represents the weight of the point cloud plane, and the minimum energy P of the graph segmentation problem * for: P * =argminE(P), Where E(P) is the energy of the fit, satisfying: Where V and E are the sets of vertices and edges in the graph, respectively, corresponding to the adjacency graph G = {v, ε} of cluster segmentation.

6. A method for constructing a three-dimensional semantic map of an indoor mobile robot according to claim 1, Features: Step 5 includes: The bounding box of the target object point cloud is obtained according to the 2D recognition box, where the width value b of the bounding box is the maximum depth difference corresponding to all the pixels in the 2D recognition box. Assume that all the pixels in the 2D recognition box are a set P = {P 1 , P 2 , P 3 ......, P n },but b=max(Depth(P i )-Depth(P j )) Among them, P i ,P j ∈P,Depth(P i ) is the pixel point P i The corresponding depth value is back-projected into the world coordinate system using 2D pixel points, and the length and width of the 2D recognition box are mapped into the world coordinate system to obtain the length and height of the bounding box; Then the results of point cloud segmentation are tested. Only when all point clouds in a cluster are located in the bounding box, the random color attribute label of the cluster is replaced with the 3D semantic label obtained in step 3 to obtain a semantic map.

7. A method for constructing a three-dimensional semantic map for an indoor mobile robot according to any one of claims 1 to 6, Features: The RGB-D depth camera uses a KinectV2 camera.

8. A method for constructing a three-dimensional semantic map for an indoor mobile robot according to any one of claims 1 to 6, Features: The YOLOV5 network adopts the YOLOv5s network.

Citation Information

Patent Citations

  • Deep learning-based method for constructing three-dimensional semantic map of indoor environment

    CN110243370A

  • Robot semantic map object recognition method based on deep learning

    CN112288857A

  • Area semantic learning and map point identification method for power transformation operation scene

    WO2021249575A1

  • Humanoid inspection operation method and system for semantic intelligent substation robot

    WO2022021739A1

Cited By

  • Improved dynamic environment SLAM method based on YOLOv6 algorithm

    CN116678401A

  • Method for improving instance segmentation precision

    CN121259310A