Panorama-based self-supervised learning scene point cloud completion data set generation method

[0042] The specific implementation manners of the present invention will be further described below in conjunction with the drawings and technical solutions.

[0043] The present invention is based on the 2D-3D-Semantics dataset released by Stanford University. This dataset involves 6 large indoor areas originating from 3 different buildings mainly education and office. The data set contains a total of 1413 equirectangular panoramic RGB images, as well as corresponding depth maps, surface normal maps, semantic annotation maps, and camera metadata, which are sufficient to support the self-supervised learning scene point cloud compensation based on panoramic images proposed by the present invention. A complete dataset generation method. In addition, other equirectangular panoramas taken or collected are also applicable to the present invention.

[0044] The present invention includes four main modules, which are respectively 2D-3D rectangular projection module, viewpoint selection module, 3D-2D rectangular projection and point-to-point occlusion processing module, and 2D-3D rectangular projection and angle mask filtering module, such as figure 1 shown. First, the 2D-3D equirectangular projection module starts with v 1 Panoramic RGB image under viewpoint C 1 , panoramic depth map D 1 and the panoramic normal map N 1 As input, generate an initial point cloud P with RGB information 1 and the initial point cloud P with normal information 2. Second, the viewpoint selection module uses the initial point cloud P 1 As input, generate a new occlusion prediction viewpoint v 2. Third, the 3D-2D equirectangular projection and point-to-point occlusion processing module uses the initial point cloud P 1 , the initial point cloud P 2 and the viewpoint v 2 As input, generate v 2 Panoramic RGB image under viewpoint C 2 , panoramic depth map D 2 and the panoramic normal map N 2. Fourth, the 2D-3D equirectangular projection and angle mask filtering module uses v 2 Panoramic RGB image under viewpoint C 2 , panoramic depth map D 2 and the panoramic normal map N 2 As input, generate a defect cloud P with RGB information 3 , residual defect cloud P with normal information 4 , and an angle mask to filter the striped regions. Finally, the resulting residual defect cloud P 3 , residual defect cloud P 4 And the angle mask can be sent to the scene point cloud completion network, and finally the completed point cloud P is generated 5. Among them, residual defect cloud P 3 (input) and initial point cloud P 1 (intended target) or incomplete cloud P 4 (input) and initial point cloud P 2 (expected target) can be used as a self-supervised learning data pair.

[0045] An example visualization of an intermediate result would look like figure 2 shown, showing v 1 Panoramic RGB image under viewpoint C 1 , panoramic depth map D 1 and the panoramic normal map N 1 , the initial point cloud P 1 and the initial point cloud P 2; and v 2 Panoramic RGB image under viewpoint C 2 , panoramic depth map D 2 and the panoramic normal map N 2 , residual defect cloud P 3 and residual defect cloud P 4.

[0046] An example visualization of the angle mask looks like image 3 As shown in , it shows the angle mask generated by looking at the plane on the table from an approximately horizontal angle. It can be seen that the filtered area is striped, which can effectively solve the problem of point cloud incompleteness caused by the viewing angle.