A system and method for semantic scene completion

Abstract

The invention relates to a method and system for completing a semantic scene. The method comprises obtaining a labeled color image and a depth image as training data, wherein the color image and the depth image are labeled with a semantic segmentation label and a semantic scene complement label; training convolution neural network with training data to get semantic segmentation model and semanticscene complement model, input the image to be complemented to semantic segmentation model to get semantic segmentation result; according to the camera parameters and depth images of color images, obtaining the mapping relationship between pixels in color images and voxels in depth images; according to the mapping relationship, projecting the result of semantic segmentation to three-dimensional space, and obtaining the semantic scene surface of the image to be completed; discretizing the surface of the semantic scene and inputting to the semantic scene completion model to obtain the three-dimensional structure of the image to be completed and the category of the object in the image to be completed, and outputting the three-dimensional structure and the category as the semantic scene completion results of the image to be completed.

Description

technical field [0001] The invention relates to the technical field of three-dimensional semantic scene completion, in particular to a multi-sensor three-dimensional semantic scene completion system and method based on a convolutional neural network. Background technique [0002] Semantic scene completion, that is, the restoration and semantic understanding of 3D scenes, has always been an important issue in computer vision technology. For example, robots can interact more with the surrounding environment only if they have the ability to perceive the three-dimensional structure of the scene and understand three-dimensional objects like humans. Therefore, if the three-dimensional structure of the scene can be well restored and the object categories in the scene can be obtained, it will greatly promote the development of artificial intelligence. In addition, semantic scene completion can also be widely used in augmented reality, path planning and navigation, construction of e...

AI Technical Summary

Problems solved by technology

[0007] To sum up, the current semantic scene completion has the following problems: it is more difficult to directly extract the features of color images and depth images in 3D space, because the limitation of storage and calculation time makes the voxel representation resolution of the scene not high, thus It will become more difficult to extract high-quality features; for different inputs such as RGB or Depth, using a customized network structure will lead to poor scalability, which is not conducive to the use of more sensors in the future; tightly coupled network design The mode makes it very difficult to improve the performance in the future. If you want to improve the overall performance, you need to change the structure of the entire framework

Images