An indoor slam method based on atlases and semantic information

Abstract

An indoor SLAM method based on atlases and semantic information belongs to the technical field of simultaneous localization and mapping. The method combines an improved semantic segmentation network to generate a plane instance mask with semantic information, and extracts plane parameters through PCA principal component analysis. A uniform tracking model and plane parameters complete plane matching, and planes that do not conform to the Atlanta structure are removed using the gravity direction. Finally, the Atlanta coordinate system is detected and matched through the results of plane matching, the matching results are used for pose estimation, and the Atlanta coordinate system is corrected. The method can be applied to actual physical environments, reduce the influence of accumulated errors in weak texture environments on visual odometry pose estimation, and provide robust and high-precision positioning information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of simultaneous localization and mapping. Specifically relates to a visual SLAM method applied to indoor environment. BACKGROUND

[0002] With the rapid progress of artificial intelligence and computer technology, intelligent autonomous mobile robots have gradually become the key direction in the field of robot research. Intelligent autonomous mobile robots combine a variety of intelligent technologies, among which, the simultaneous localization and mapping (SLAM) technology is the core of its implementation value. SLAM technology provides robots with the ability to self-localize and map the environment in an unknown environment, and is the basis for the smooth operation of intelligent robots.

[0003] Simultaneous localization and mapping (SLAM) refers to the technology of estimating the position of the main body in motion in real time and simultaneously constructing a map of the surrounding environment under the condition of lacking prior information about the environment. According to the type of sensor used, SLAM algorithms are mainly divided into two categories: visual SLAM algorithms based on monocular cameras, RGB-D cameras and binocular cameras, and laser SLAM algorithms based on laser radars. In recent years, visual SLAM algorithms have been widely used in intelligent autonomous mobile robots and autonomous driving due to their low cost, rich environmental perception information and high indoor positioning accuracy.

[0004] Most traditional visual SLAM algorithms are based on traditional point and line features. However, in indoor environments, the number of feature points in weak texture environments is greatly reduced, which leads to an unavoidable cumulative error in positioning and affects the accuracy of visual odometry pose estimation. In recent years, to solve the problem of poor positioning accuracy of visual SLAM algorithms in indoor weak texture environments, domestic and foreign scholars have proposed methods to detect indoor Manhattan structures and extract planes in images that conform to Manhattan structures, converting inter-frame tracking to tracking in the Manhattan coordinate system to avoid the influence of cumulative errors in indoor weak texture environments. However, this method has some limitations: (1) traditional plane detection uses a hierarchical clustering method, which is significantly affected by sensor noise, which may lead to false plane detection, and this method only uses pixel information, which is prone to misjudging curved surfaces as planes; (2) the increase in the number of Manhattan coordinate systems may lead to errors in Manhattan coordinate system matching, resulting in reduced recognition accuracy; (3) the use of point features for mixed positioning may lead to cumulative error.

[0005] Therefore, it is a technical problem that needs to be solved by those skilled in the art to provide a robust visual SLAM method that can be applied to most indoor weak texture scenes. SUMMARY

[0006] The main purpose of the present application is to provide a visual SLAM method based on the Atlanta coordinate system applied to an indoor weak texture environment, which can be applied to the actual physical environment, reduce the influence of accumulated error on the visual odometry pose estimation in the weak texture environment, and provide robust and high-precision positioning information. The present application aims to solve the following problems:

[0007] 1. The traditional visual SLAM algorithm has an indelible accumulated error in indoor weak texture environment positioning.

[0008] 2. When the number of Manhattan coordinate systems in the environment is too large, the SLAM algorithm using Manhattan structure positioning will have the problem of reduced recognition accuracy.

[0009] 3. Using point feature hybrid positioning will bring accumulated error.

[0010] The present application provides an RGB-D visual SLAM method based on the Atlanta structure applied to an indoor weak texture environment, which combines an improved semantic segmentation network to generate a plane instance mask with semantic information, and extracts plane parameters through PCA principal component analysis method; the uniform speed tracking model and the plane parameters complete plane matching, and the gravity direction is used to eliminate the Manhattan planes that do not conform to the Atlanta structure; finally, the Atlanta coordinate system is detected and matched through the plane matching result, the matching result is used for pose estimation, and the Atlanta coordinate system is corrected.

[0011] The specific working process of the present application is as follows:

[0012] Step 1, using an RGB-D depth camera to collect standard indoor images, obtaining a sequence of continuous RGB images and depth images;

[0013] Step 2, for the RGB images collected in step 1, using an improved Yolov8 plane instance segmentation network to segment the plane semantic image with Manhattan structure;

[0014] Step 3, combining the plane image obtained in step 2 with the depth image obtained in step 1, using PCA principal component analysis method to obtain plane parameters;

[0015] Step 4, completing plane matching based on the uniform speed tracking model and the plane parameters;

[0016] Step 5, according to the semantic information and the parameters of the matched plane, obtaining the gravity direction, using the gravity direction to eliminate the Manhattan planes that do not conform to the Atlanta structure;

[0017] Step 6, the plane is orthogonally detected, all atlantic coordinate systems are detected, the atlantic coordinate system of the current frame is matched with the atlantic coordinate system in the map, matched to the key frame, and the atlantic coordinate system state is marked.

[0018] Step 7, based on the key frame matched by the current frame, pose estimation and mapping are carried out, and the pose of the current frame is used to correct the atlantic coordinate system with an inaccurate state.

[0019] The semantic segmentation network in step 2 is an improved Yolov8-seg plane instance segmentation model, which can perform pixel-level instance segmentation on indoor planes. The specific steps are as follows:

[0020] The RGB image is preprocessed, first the image is adjusted to 640*640*3 size, then the OpenCV function is used for pixel normalization and R, B color channel conversion to ensure the uniformity and standardization of image data. The preprocessed picture is input into the backbone network of the Yolov8-seg instance segmentation network for feature extraction, the extracted features are input into the multi-scale feature fusion module, the fused features are enhanced by attention, and the network pays more attention to the edge features of the plane through the channel attention module and the spatial attention module. Finally, it is input into the multi-scale instance segmentation module to produce two outputs output0 and output1. The first output output0 is 1*116*8400. The first 84 items of 116 include cx, cy, w, h and the recognition accuracy scores of 80 plane semantic categories. Cx and cy represent the center point coordinates of the target detection box, w and h represent the width and height of the target detection box; the last 32 columns are used to calculate the plane mask data. The second output output1 is 1*32*160*160. The mask data corresponding to the target is obtained by using the matrix multiplication of the last 32 fields of output0 and the data of output1, so as to obtain the plane instance segmentation mask and the plane semantic information. After filtering the plane using the plane semantic information, the plane with Manhattan structure is extracted, and the semantic information of the plane is saved as the plane of objects such as ceiling, floor, wall and table that meet the Manhattan structure. The indoor plane such as road plane is deleted.

[0021] Step 3 is as follows:

[0022] The three-dimensional coordinates of the point cloud are obtained by the depth image and the camera intrinsic parameter. For each pixel point (u, v), the depth value d is obtained, and the three-dimensional coordinates (x, y, z) of the pixel point are calculated according to the camera intrinsic parameter (fx, fy, u0, v0), where fx is the vertical focal length, fy is the horizontal focal length, u0 is the horizontal coordinate of the image principal point, and v0 is the vertical coordinate of the image principal point, as shown in formula (1):

[0023]

[0024] z = d (1)

[0025] According to the filtered plane obtained in step two, the three-dimensional point cloud coordinates of the plane in the depth image are obtained through the coordinates of the plane in the RGB image, and the center point coordinates of the plane in the current frame and the plane normal vector are extracted using the PCA principal component analysis method. The three-dimensional coordinates of each point cloud are denoted as (x i ,y i ,z i ), and each point is subtracted by the mean value of the dimension to obtain a P matrix, as shown in equation (2):

[0026]

[0027] The covariance matrix C is calculated using the centralized coordinate matrix P, as shown in equation (3):

[0028]

[0029] The eigenvalues and eigenvectors of the point cloud are obtained by eigenvalue decomposition of the covariance matrix C, and the eigenvector corresponding to the smallest eigenvalue is the normal vector n c of the plane.

[0030] Step 4 is as follows:

[0031] The pose of the current frame is roughly estimated according to the uniform speed tracking model, and the pose of the current frame estimated by the uniform speed tracking model is denoted as The displacement of the current frame is D, the center point coordinates of the plane in the current frame obtained in step two are M c , the normal vector of the plane in the current frame is n c , and the parameter representation of the plane in the initial coordinate system is calculated, as shown in equation (4):

[0032]

[0033] M w = M c + D (4)

[0034] The angle difference and plane distance difference between the current plane and the map plane are calculated, where n map is the normal vector of the map plane, and d map is the vertical distance from the world origin to the map plane, as shown in equation (5):

[0035] △angle = n w · n map

[0036] △d = d map + n w · Mw (5)

[0037] When the angle difference is lower than the threshold 0.05 and the plane distance difference is lower than 0.03 meters, it is considered that the plane matching is successful.

[0038] Step 5 is specifically as follows:

[0039] The plane matching to the plane marked as perpendicular to the gravity direction or the plane with semantics of ground in the current image is used to extract the gravity direction, and the gravity direction is multiplied by all the planes of the current frame, if the result is less than 0.05, it is considered that the plane is perpendicular to the gravity direction, or if the result is greater than 0.95, it is considered that the plane is parallel to the gravity direction, thus the plane is the plane conforming to the Atlanta structure, and the remaining planes are removed.

[0040] Step 6 is specifically as follows:

[0041] All the plane normal vectors are detected by normal vector multiplication, if three mutually perpendicular normal vectors are detected, it is considered that the three normal vectors constitute an Atlanta coordinate system, whether the map planes matched with the three planes belong to a certain key frame is inquired, if they belong to a certain key frame, it is considered that the matching is successful, if there is an Atlanta coordinate system matched successfully, the Atlanta pose estimation is used, otherwise the point feature is used for pose estimation.

[0042] There are three rules for the state of the Atlanta coordinate system: 1. during the period when the SLAM positioning has not used the point feature tracking, the state of the Atlanta coordinate system detected by all the frames is set to accurate; 2. if the point feature pose estimation is experienced, the Atlanta coordinate system encountered again is temporarily set to inaccurate. 3. if the accurate Atlanta coordinate system is matched, the state is also set to accurate.

[0043] Step 7 is specifically as follows:

[0044] The current frame is used to perform rotation estimation with the accurate Atlanta coordinate system in the map. The plane normal vectors are arranged in the order of n x , n y , n z , to obtain the relative rotation matrix R cy of the current frame relative to the reference Atlanta coordinate system frame, and singular value decomposition (SVD) is used to ensure the orthogonality. Let R be the rotation matrix between the Atlanta coordinate system and the reference frame where it appears, R be the rotation matrix between the reference frame and the world coordinate system, and R cw be the rotation matrix of the current frame, as shown in formula (6).

[0045] R cy = [n x , n y,n z ]

[0046]

[0047] After completing the rotation estimation, the ORB feature points in the current frame are detected, and the feature points are matched using the ORB feature descriptors, and through the matched ORB feature points, the re-projection error of the camera measurement is constructed, after fixing the rotation matrix, the translation estimation is performed by minimizing the re-projection error, as shown in formula (7),

[0048]

[0049] Wherein M is the number of point features, t cw is the displacement of the current frame, K is the camera intrinsic matrix, P i is the three-dimensional coordinates of the feature points, Z is the z-axis value of the three-dimensional coordinates of the feature points, u i is the pixel coordinates of the feature points in the current image frame.

[0050] After completing the displacement and rotation estimation, the state of the Atlanta coordinate system in the current frame is corrected, the plane normal vectors are arranged in the order of xyz direction as n' x , n' y , n' z , the relative rotation matrix R cy' of the current frame relative to the non-accurate Atlanta coordinate system frame is obtained, y'w as shown in formula (8),

[0051] R cy' =[n' x ,n' y ,n' z ]

[0052]

[0053] And the state of this Atlanta coordinate system is set to accurate. The world coordinates of all the feature points are obtained through the current frame rotation matrix, and the mapping task is completed. The current rotation matrix R cw and the translation matrix t cw are used as the current pose, and the positioning task is completed.

[0054] The application has the following advantages:

[0055] (1) The Yolov8-seg semantic segmentation network is improved in the application, which significantly improves the detection accuracy of the plane.(2) The application can be applied to most actual indoor scenes, and the plane is screened based on semantic information, which overcomes the recognition error caused by the increase of Manhattan structure, and can effectively improve the positioning accuracy of the SLAM algorithm in the indoor weak texture environment. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The application is applied to the RGB-D visual SLAM method flowchart of indoor weak texture environment;

[0057] Figure 2 The plane instance mask image generated by the inference result of the semantic segmentation network

[0058] a. RGB image

[0059] b. The plane instance mask image generated by the original yolov5 semantic segmentation network;

[0060] c. The plane instance mask image generated by the improved semantic segmentation network;

[0061] Figure 3 The average plane segmentation accuracy mAP50-95 (95%> IoU> 50%) before and after network improvement

[0062] Figure 4 The positioning trajectory of the RGB-D visual SLAM method of the application applied to the indoor weak texture environment

[0063] a. Real trajectory

[0064] b. The trajectory of the method of the application DETAILED DESCRIPTION

[0065] In order to more fully express the technical scheme of the application, the application will be further described in combination with the drawings and examples.

[0066] As Figure 1 shown, the specific steps of the application are as follows:

[0067] 1. Collecting RGB image and depth image

[0068] The image acquisition device adopts an RGB-D camera, the camera is calibrated and the parameters are written into the SLAM framework sensor parameter yaml file, the collected RGB image frame stream and depth image frame stream are transmitted to the computer in real time, and the tracking thread of the SLAM framework receives the image data published by the sensor.

[0069] 2. Indoor plane instance segmentation and rejection

[0070] The RGB image is preprocessed, first the image is adjusted to 640*640*3 size, then the pixel normalization and R, B color channel conversion are performed using OpenCV function, which ensures the uniformity and standardization of image data. The preprocessed picture is input into the backbone network of the Yolov8-seg instance segmentation network for feature extraction, the extracted features are input into the multi-scale feature fusion module, the fused features are enhanced by attention, and the channel attention module and spatial attention module are used to make the network pay more attention to the edge features of the plane, and finally the multi-scale instance segmentation module is input, two outputs output0 and output1 are generated, the first output output0 is 1*116*8400. The first 84 items of 116 include cx, cy, w, h and the recognition accuracy score of 80 plane semantic categories, cx and cy represent the center point coordinates of the target detection box, w and h represent the width and height of the target detection box; the last 32 columns are used to calculate the plane mask data. The second output output1 is 1*32*160*160. The mask data corresponding to the target is obtained by using the matrix multiplication of the last 32 fields of output0 and the data of output1, so as to obtain the plane instance segmentation mask and plane semantic information. The planes with Manhattan structure are extracted after filtering the planes using the plane semantic information, and the semantic information of the planes of objects such as ceiling, floor, wall and table that meet the Manhattan structure is saved, and the planes such as road plane that are not common in indoor environment are deleted.

[0071] 3. Calculate plane parameters

[0072] The three-dimensional coordinates of the point cloud are obtained from the depth image and the camera intrinsic parameters. For each pixel point (i, j), the depth value d i,j is obtained x , the three-dimensional coordinates (x, y, z) of the pixel point are calculated according to the camera intrinsic parameters (f y , u0, v0), as shown in formula (1):

[0073]

[0074] z=d (1)

[0075] According to the filtered plane obtained in step two, the three-dimensional point cloud coordinates of the plane in the depth image are obtained through the coordinates of the plane in the RGB image, and the center point coordinates and the plane normal vector of the plane in the current frame are extracted using the PCA principal component analysis method, and the three-dimensional coordinates of each point cloud are denoted as (x i , y i , z i ), and each point is subtracted from the mean value of the dimension to obtain the P matrix, as shown in formula (2):

[0076]

[0077] Compute covariance matrix: Compute the covariance matrix C using the centered coordinate matrix P as in equation (3):

[0078]

[0079] Eigen decomposition of the covariance matrix C gives the eigenvalues and eigenvectors of the point cloud, where the eigenvector corresponding to the smallest eigenvalue is the normal vector n of the plane c .

[0080] 4. Uniform velocity tracking model matching plane

[0081] Estimate the pose of the current frame according to the uniform velocity tracking model, and denote the pose of the current frame estimated by the uniform velocity tracking model as The displacement of the current frame is D, and the center point coordinates of the plane in the current frame obtained in step two are M c , the normal vector of the plane in the current frame is n c , and the parameter representation of the plane in the initial coordinate system is n w , as shown in equation (4):

[0082]

[0083] M w = M c + D (4)

[0084] Calculate the angle difference and plane distance difference between the current plane and the map plane, where n map is the normal vector of the map plane, and d map is the vertical distance from the world origin to the map plane, as shown in equation (5):

[0085] △angle = n w · n map △d = d map + n w · M w (5) When the angle difference is less than the threshold value 0.05 and the plane distance difference is less than 0.03 meters, it is considered that the plane matching is successful.

[0086] 5. Gravity direction rejection plane

[0087] Match to the plane marked as perpendicular to the gravity direction, or there is a plane with semantic ground in the current image, use this plane to extract the gravity direction, and multiply all the planes in the current frame by this gravity direction, if the size is close to 0, or close to 1, it means that the plane is perpendicular to the gravity direction, or the plane is parallel to the gravity direction, so this plane is the plane that meets the Atlanta structure, and the remaining planes are rejected.

[0088] 6. Atlanta coordinate system tracking

[0089] All the plane normal vectors are detected by normal vector multiplication. If three mutually perpendicular normal vectors are detected, it means that the three normal vectors form an Atlanta coordinate system. Query whether the three planes matched with the three planes belong to a certain key frame. If they belong to a certain key frame, it means that the matching is successful. If there is an Atlanta coordinate system with successful matching, the Atlanta pose estimation is used. Otherwise, the point feature is used for pose estimation.

[0090] There are three rules for the state of the Atlanta coordinate system: 1. During the period when the SLAM positioning has not used the point feature tracking, the state of the Atlanta coordinate system detected by all frames is set to accurate; 2. If the point feature pose estimation is experienced, the Atlanta coordinate system encountered is temporarily set to inaccurate. 3. If the accurate Atlanta coordinate system is matched, the state is also set to accurate.

[0091] 7. Rotation and displacement estimation

[0092] The rotation estimation is performed using the current frame and the accurate Atlanta coordinate system in the map. The plane normal vectors are arranged in the order of xyz direction as n x , n y , and n z , to obtain the relative rotation matrix R cy of the current frame relative to the reference Atlanta coordinate system frame, and singular value decomposition (SVD) is used to ensure its orthogonality. Let R be the rotation matrix between the Atlanta coordinate system and the reference frame where it appears, be the rotation matrix between the reference frame and the world coordinate system, and the current frame rotation R cw is as shown in equation (6).

[0093] R cy = [n x , n y , n z ]

[0094]

[0095] After the rotation estimation is completed, the ORB feature points in the current frame are detected, and the ORB feature descriptors are used to match the feature points. Through the matched ORB feature points, the reprojection error of the camera measurement is constructed. After the rotation matrix is fixed, the translation estimation is performed by minimizing the reprojection error, as shown in equation (7),

[0096]

[0097] where M is the number of point features, t cw is the displacement of the current frame, K is the camera intrinsic matrix, and P iZ is the z-axis value of the three-dimensional coordinates of the feature point.

[0098] After displacement and rotation estimation is completed, the Atlanta coordinate system with the state of non-accuracy in the current frame is corrected, R cy' , to obtain the rotation matrix R cw of the non-accuracy Atlanta coordinate system relative to the world coordinate system, as shown in formula (8),

[0099] R cy' = [n' x , n' y , n' z ]

[0100]

[0101] and the state of this Atlanta coordinate system is set to be accurate. The world coordinates of all feature points are obtained through the current rotation matrix, and the mapping task is completed. The current rotation matrix R cw and the translation matrix t cw are used as the current pose to complete the positioning task.

[0102] The above description of the specific implementation method of the present application is only used for the technical personnel in the technical field to understand the present application, and is not used to limit the scope of the present application.

Claims

1. A method for indoor SLAM based on atlases and semantic information, characterized in that, Comprise the following steps: Step 1, using RGB-D depth camera to collect standard indoor image, get continuous RGB image and depth image sequence; Step 2, for the RGB image collected in step 1, the improved Yolov8 plane instance segmentation network is used to segment the plane semantic image with Manhattan structure; Step 3, combine the plane image obtained in step 2 with the depth image obtained in step 1, and use the PCA principal component analysis method to obtain the plane parameters; Step 4, complete plane matching based on uniform speed tracking model and plane parameters; Step 5, according to the semantic information and the parameters of the matched plane, the gravity direction is obtained, and the Manhattan plane which does not conform to the Atlanta structure is removed using the gravity direction; Step 6, orthogonal detection is performed on the plane, all Atlanta coordinate systems are detected, and the Atlanta coordinate system of the current frame is matched with the Atlanta coordinate system in the map, and matched to the key frame; Step 7, based on the key frame matched by the current frame, pose estimation and local mapping are performed; Step 2 is as follows: The RGB image is input into the backbone network of the Yolov8-seg instance segmentation network for feature extraction, the extracted features are input into the multi-scale feature fusion module, the fused features are enhanced through channel attention module and spatial attention module to obtain the target detection box of the plane, and finally input into the multi-scale instance segmentation module to generate plane instance segmentation mask and plane semantic information. The plane with Manhattan structure is extracted after screening the plane using the plane semantic information; Step 4 is as follows: According to the uniform velocity tracking model, the pose of the current frame is roughly estimated, and the pose of the current frame estimated by the uniform velocity tracking model is denoted as The displacement of the current frame is D, and the step 2 obtains the center point coordinate of the plane in the current frame as M c The normal vector of the plane in the current frame is n c The parameter representation of the plane in the initial coordinate system is calculated, as shown in formula (4): M w = M c + D (4) Compute the angle difference and the plane distance difference between the current plane and the map plane, where n map is the normal vector of the map plane, d map is the vertical distance from the world origin to the map plane, as shown in equation (5): Δangle = n w • n map Δd = d map + n w • M w (5) When the angle difference is less than the threshold 0.05 and the plane distance difference is less than 0.03 meters, it is considered that the plane matching is successful.

2. The indoor SLAM method based on Atlanta world and semantic information of claim 1, wherein, The center point coordinates and plane normal vector of the plane in the current frame are extracted using the PCA principal component analysis method, Let the three-dimensional coordinates of all point clouds be (x i ,y i ,z i ), and each point is subtracted from the mean to obtain the P matrix, as shown in equation (1): Calculate the covariance matrix: use the centralized coordinate matrix P to construct the covariance matrix C, as formula (2): The covariance matrix C is decomposed to obtain eigenvalues and eigenvectors of the point cloud, wherein the eigenvector corresponding to the smallest eigenvalue is the plane normal vector n c .

3. The indoor SLAM method based on Atlanta world and semantic information of claim 1, wherein, Step 5 is as follows: Match the plane marked as perpendicular to the gravity direction, or there is a plane with semantic information of ground in the current image, use this plane to extract the gravity direction, and multiply all the planes in the current frame through this gravity direction, if the result is less than 0.05, it means that the plane is perpendicular to the gravity direction, or the result is greater than 0.95, it means that the plane is parallel to the gravity direction, so this plane is the plane conforming to the Atlanta structure, and the remaining planes are removed.

4. The indoor SLAM method based on Atlanta world and semantic information of claim 1, wherein, Step 6 is as follows: All plane normal vectors are detected by normal vector multiplication, if three mutually perpendicular normal vectors are detected, it means that the three normal vectors constitute an Atlanta coordinate system, query whether the map planes matched with the three planes belong to a certain key frame at the same time, if there are multiple key frames, take the earliest one in the time sequence.

5. The indoor SLAM method based on Atlanta world and semantic information of claim 1, wherein, Step 7 is as follows: Through the rotation estimation of the Atlanta coordinate system in the key frame and the Atlanta coordinate system of the current frame, the relative rotation relationship of the current frame relative to the key frame is obtained, and then the pose of the current frame relative to the world coordinate system is obtained through the pose of the key frame relative to the world coordinate system.

Images