A three-dimensional reconstruction method based on depth prediction and fusion

Abstract

The application provides a three-dimensional reconstruction method based on depth prediction and fusion, comprising the following steps: step one, obtaining color pictures of the same position and multiple angles of a scene, and sequentially splicing a plurality of color pictures to obtain an initial color panorama; step two, rotating the initial color panorama in step one N times to obtain N rotated color panoramas of different angles; step three, inputting the initial color panorama and the N rotated color panoramas into a pre-trained neural network to obtain an initial depth panorama and N rotated depth panoramas; and step four, removing edge data on both sides of the initial depth panorama and the N rotated depth panoramas; the application solves the problems of high cost of the existing three-dimensional model reconstruction method, cracks at the splicing position, and certain differences from the real scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, and particularly relates to a three-dimensional reconstruction method based on depth prediction and fusion. BACKGROUND

[0002] Three-dimensional models play an important role in various fields, such as industrial measurement, home decoration, indoor navigation, asset management, etc. Generally, three-dimensional models need depth information of a scene, and devices for obtaining depth information include a structured light-based depth sensor and a light flight time-based laser radar, which are relatively expensive. In order to save costs, color lenses are usually used to obtain scene information, and depth information corresponding to the scene is predicted by means of deep learning technology, and then point clouds obtained at different positions are fused to obtain a three-dimensional model of the entire scene. This method is used for predicting a single panoramic image, and the advantage is that the predicted depth information of the single panoramic image is more accurate.

[0003] However, the depth information predicted based on a single panoramic image has errors at the edges of the panoramic image, so that when the model is spliced, the models at the connection position are not aligned, and cracks are generated at the splicing position.

[0004] Especially for three-dimensional reconstruction of a color panoramic image obtained by a color camera, the depth information predicted by deep learning technology has a large error at the edges of the image, and the three-dimensional model generated finally has certain cracks at the splicing position, which causes a certain difference between the three-dimensional model and the real scene. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application provides a three-dimensional reconstruction method based on depth prediction and fusion, which solves the problems of high cost of the prior three-dimensional model reconstruction method, cracks at the splicing position, and certain difference from the real scene.

[0006] The technical scheme adopted by the present application to solve the technical problem is: a three-dimensional reconstruction method based on depth prediction and fusion, comprising the following steps:

[0007] Step 1: Obtain color pictures of the same position and multiple angles of a scene, and sequentially splice a plurality of color pictures to obtain an initial color panoramic image;

[0008] Step 2: Rotate the initial color panoramic image in step 1 for N times to obtain N rotated color panoramic images of different angles;

[0009] Step 3: input the initial color panoramic image and the N rotated color panoramic images into a pre-trained neural network to obtain an initial depth panoramic image and N rotated depth panoramic images;

[0010] Step four, removing the initial depth panorama and N rotating depth panorama edge data on both sides;

[0011] Step five, reversing the rotation of the N rotating depth panorama with removed edge data by corresponding angles, so that it is one-to-one corresponding with the initial depth panorama with removed edge data Figure One corresponding;

[0012] Step six, selecting the depth data mean of the depth values on both sides of the initial depth panorama and the N rotating depth panorama with removed edge data in step five, and adding it to both sides of the initial depth panorama with removed edge data, to complete the fusion of the edge depth information on both sides of the initial depth panorama with removed edge data;

[0013] Step seven, converting the initial depth panorama with fused depth information into a three-dimensional point cloud, and fusing the three-dimensional point clouds at different positions to complete the three-dimensional model reconstruction.

[0014] Preferably, the step four removes 1 / 8 degree data on the edges of the initial depth panorama and the N rotating depth panorama.

[0015] Preferably, the step seven converts the initial depth panorama with fused depth information into a three-dimensional point cloud in the following manner:

[0016]

[0017] Wherein, w, h are the resolution of the image, and d is the depth value stored at the pixel (u0, v0).

[0018] The present application has the following advantages:

[0019] The color panorama is rotated at a set angle, the depth panorama edge data is removed, the depth panorama is reversed at a corresponding angle after the removal of the depth panorama edge data, the depth values on both sides of the depth panorama are averaged, and then added to both sides of the initial depth panorama, which effectively reduces the depth information error on both sides of the depth panorama, improves the accuracy of the depth panorama, and further realizes the reduction of cracks in the reconstructed three-dimensional model at the splicing position, and realizes the consistency with the real scene. Compared with the existing three-dimensional reconstruction method, the present method has lower cost. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the present application, but do not constitute a limitation on the present application. In the drawings:

[0021] Figure 1 is a flow chart of the three-dimensional reconstruction method in the present embodiment. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present application will be described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are merely illustrative and explanatory in nature and are not intended to limit the application.

[0023] Embodiments

[0024] Referring to Figure 1 The three-dimensional reconstruction method based on depth prediction and fusion shown in the figure comprises the following steps:

[0025] Step one, acquire color pictures of the same position and multiple angles of the scene, and sequentially splice the color pictures to obtain an initial color panorama; the color camera is used to take pictures in different directions at a fixed position, and multiple color pictures are spliced to obtain a color panorama; it should be noted that in order to obtain a three-dimensional model of the entire scene, the adjacent shooting positions should not be too far apart, for example, a point can be shot every 3-5 meters;

[0026] Step two, rotate the initial color panorama in step one N times to obtain N rotated color panoramas of different angles; the initial color panorama is rotated N times to obtain N rotated color panoramas, in this embodiment, the color panorama is rotated 90 degrees in the horizontal direction, a total of 3 times, to obtain 4 color panoramas of the same size (the 4 color panoramas include the initial color panorama), but different angles; of course, the rotation angle and the number of rotations can also be set according to actual needs; the more the number of rotations, the fewer the cracks in the reconstructed three-dimensional model, which is consistent with the real scene;

[0027] Step three, input the initial color panorama and the N rotated color panoramas into a pre-trained neural network to obtain an initial depth panorama and N rotated depth panoramas;

[0028] Step four, remove the edge data of the initial depth panorama and the N rotated depth panoramas;

[0029] Step five, reversely rotate the N rotated depth panoramas with the edge data removed by a corresponding angle, so that they correspond to the initial depth panorama with the edge data removed; the rotated depth panoramas in step five satisfy the translational and rotational relationship between the panorama images used for prediction; for example, if the rotated color panorama is rotated N times, and each rotation is θ=360 / N degrees, then the rotated depth panorama is rotated by a corresponding angle in the opposite direction to align the depth data, that is, the depth value at the same pixel position represents the distance of the same physical position in the scene; Figure One

[0030] ​Step Six: Select the average depth data from both sides of the initial depth panoramic image (with edge data removed) and the N rotated depth panoramic images from Step Five, and add them to both sides of the initial depth panoramic image (with edge data removed) to complete the fusion of edge depth information on both sides of the initial depth panoramic image. Since the accuracy of depth information predicted by neural networks often has a certain error, multiple depth values ​​predicted at the same location are not equal, and the error is even greater at the leftmost and rightmost edges of the depth panoramic image. To overcome this problem, this embodiment discards the data from the leftmost and rightmost edges of each depth panoramic image. For example, for a 9000×4500 panoramic image, data at 1 / 8 degree of the edge can be discarded, i.e., the leftmost and rightmost edges of the image can be discarded. The data in the column is processed, and then all the rotated depth panoramas are rotated in the opposite direction at the corresponding angle to obtain multiple depth panoramas after the depth data is aligned (the initial depth panorama and N rotated depth panoramas). Then, the average value of all depth data is taken pixel by pixel. For example, on the N+1 aligned depth panorama, after determining a pixel position, the N depth values ​​at that pixel position are taken out and added together, and then divided by N+1. If there is depth data that has been deleted at that pixel position, the average value of the remaining depth data is taken.

[0031] Step 7: Convert the initial depth panorama with fused depth information into a 3D point cloud, fuse the 3D point clouds from different locations, and complete the 3D model reconstruction.

[0032] Furthermore, the method for converting the initial depth panoramic image with fused depth information into a 3D point cloud in step seven is as follows: The x-axis of the coordinate system containing the 3D point cloud is set to point to the center of the initial depth panoramic image, the z-axis to point above the initial depth panoramic image, and the y-axis is given according to the right-handed coordinate system. Then, in the initial depth panoramic image coordinate system uv, the 3D coordinates corresponding to the initial depth panoramic image pixel (u0, v0) can be given by the following formula:

[0033]

[0034] Where w and h are the resolution of the image, and d is the depth value stored at pixel (u0, v0).

[0035] Furthermore, in step four, N+1 depth panoramic edge 1 / 8 degree data are removed.

[0036] This embodiment obtains N+1 color panoramic images from different angles by rotating a color image at a certain angle and number of times. After training a pre-trained neural network, edge data with large errors are removed, and the image is rotated in the opposite direction at the corresponding angle N times to obtain more accurate depth information, avoiding depth information errors at the edges of the depth panoramic image. This makes the 3D model more accurate, eliminates cracks in the 3D reconstruction process, and further optimizes the 3D model.

[0037] The above merely describes the preferred embodiments of the present application, and it should be understood that the above description of the embodiments is only used to help understand the method of the present application and its core idea, and is not used to limit the protection scope of the present application, and any modification, equivalent replacement, etc. within the idea and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for 3D reconstruction based on depth prediction and fusion, characterized in that: Comprise the following steps: Step one, obtain the scene same position and multiple angle color picture, a plurality of color picture is spliced in turn to obtain the initial color panorama; Step two, rotate the initial color panorama in step one N times to obtain N different angle rotating color panorama; Step three, input the initial color panorama and N rotating color panorama into the neural network trained in advance, obtain the initial depth panorama and N rotating depth panorama; Step four, remove the edge data of the initial depth panorama and N rotating depth panorama; Step five, reverse rotate the N rotating depth panorama which removes the edge data by corresponding angle, so that it corresponds to the initial depth panorama which removes the edge data; That is, the depth value at the same pixel position represents the distance of the same physical position in the scene; Step six, select the depth data mean value of the two side depth values of the initial depth panorama and N rotating depth panorama which removes the edge data and corresponds to each other, that is, the depth value at the same pixel position represents the distance of the same physical position in the scene, and add it to the two sides of the initial depth panorama which removes the edge data, complete the fusion of the edge depth information of the initial depth panorama; Step seven, convert the initial depth panorama which fuses the depth information into three-dimensional point cloud, fuse the three-dimensional point cloud at different positions, and complete the three-dimensional model reconstruction. 2.The method of claim 1, wherein: In step four, remove 1 / 8 degree data of the edge of the initial depth panorama and N rotating depth panorama. 3.The method of claim 1, wherein: In step seven, the method for converting the initial depth panorama which fuses the depth information into three-dimensional point cloud is as follows: Wherein, w, h is the resolution of the image, d is the depth value stored at pixel (u0, v0).

Images