Instant positioning and map construction system and method with semantic perception

Abstract

The invention provides an instant positioning and map construction system and method with semantic perception, and the method comprises the steps that an image collection and preprocessing module obtains an RGB image collected by a camera, and carries out the preprocessing of the RGB image, and obtains a preprocessed image; a semantic segmentation module performs two-dimensional semantic segmentation on the preprocessed image to obtain a two-dimensional semantic segmentation image; the depth estimation module acquires depth information of the preprocessed image to obtain a depth prediction image of the preprocessed image; a camera pose estimation module performs camera pose estimation according to the input preprocessing image and the depth prediction image thereof to obtain a camera posematrix; and the three-dimensional dense point cloud reconstruction module carries out dense point cloud reconstruction of a three-dimensional scene on the current frame preprocessing image, the depthprediction image of the current frame preprocessing image and the current frame camera pose matrix, and maps the two-dimensional semantic segmentation image to dense point cloud of the three-dimensional scene to complete reconstruction of the three-dimensional scene with semantic information. The method has accuracy and robustness, and understanding and reconstruction of the scene are realized only by means of the monocular camera.

Description

technical field [0001] The present invention relates to the technical field of location and map construction (SLAM), in particular to a real-time location and map construction system and method with semantic awareness. Background technique [0002] Simultaneous localization and mapping (SLAM) is a high-profile research direction in the field of computer vision and automatic driving, and it is often applied to the reconstruction of 3D scenes and the estimation of camera pose. In recent years, real-time SLAM methods aimed at fusing depth maps obtained from depth cameras or laser devices have become more and more popular. They can be applied to navigation and mapping of mobile robots and drones, and are suitable for many virtual reality, augmented Realistic applications. In addition to navigation and mapping, SLAM can also be used for accurate reconstruction of 3D scenes. However, one of the main shortcomings of this method is some limitations of the depth camera itself. Sinc...

AI Technical Summary

Problems solved by technology

[0007] After searching, I found: R.Mur-Artal, J.M.M.Montiel and J.D.Tardós, "ORB-SLAM: AVersatile and Accurate Monocular SLAM System," in IEEE Transactions on Robotics, vol.31, no.5, pp.1147-1163, In Oct.2015, an ORB-SLAM technology was recorded. The monocular SLAM system in this technology has the limitation that the absolute scale cannot be estimated.

Therefore, the absolute scale of its reconstructed scene is vague and uncertain, which greatly limits the use of monocular SLAM systems

Another major limitation is the difficulty in estimating the camera pose when purely rotational camera motion occurs

In this case, stereo estimation cannot be applied due to the lack of a stereo baseline, so the pose information of the camera cannot be obtained, which often leads to tracking failures

At the same time, because the ORBSLAM system can only calculate the coordinates of the reference points, the 3D point cloud of the reconstructed scene is very sparse and does not have semantic information

[0008] At present, there is no description or report of similar technology to the present invention, and no similar information has been collected at home and abroad

Images