A 3D reconstruction method based on a single image

Abstract

The invention discloses a three-dimensional reconstruction method based on a single image, including a training phase: the training samples include sampling points on a three-dimensional model and actual implicit function values ​​corresponding to the sampling points, and a single image corresponding to the three-dimensional model; a feature space is generated through a single input image, the sampling points of the three-dimensional model are used as query points, the implicit function values ​​of the query points are predicted, and the loss is calculated according to the predicted implicit function values ​​and the actual implicit function values ​​to complete the fitting of the implicit function field; a reconstruction phase: after the training phase is completed, a feature space is generated for the single input image, a number of points are sampled in the observation space, and these points constitute the query points; the implicit function values ​​of the query points are predicted; the query points and their predicted implicit function values ​​are input into a marching cubes algorithm to reconstruct the three-dimensional model. The method improves the problem of serious lack of prior conditions in single image reconstruction and can obtain fine details in the visible part.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and relates to a method for three-dimensional reconstruction of a two-dimensional image, and in particular to a deep learning reconstruction method based on a single picture. Background Art

[0002] Due to ambiguity (such as the presence of parallax), single-view Figure 3 3D reconstruction is an ill-posed problem. Different from 3D reconstruction based on multi-view images, single-view Figure 3 Reconstruction requires strong prior knowledge. Figure 3 The 3D reconstruction methods are mainly divided into template-based explicit methods and non-template-based implicit methods. Implicit methods can be further divided into voxel-based methods, point cloud-based methods and implicit function-based methods, each of which has its own advantages and disadvantages. The current research hotspot is the implicit function method, which has several common problems: (1) the generated reconstruction results lack details; (2) the lack of priors leads to the loss of important parts. Summary of the invention

[0003] In response to the above technical problems, the present invention proposes a three-dimensional reconstruction method based on a single image. This method makes full use of the success of deep learning methods in the image field and the three-dimensional prior information of 2.5D sketches to obtain high-quality reconstruction results.

[0004] To achieve the above object, the technical solution adopted by the present invention is: a three-dimensional reconstruction method based on a single image, comprising the following steps:

[0005] Training phase: The training samples include sampling points on the 3D model, the actual implicit function values ​​corresponding to the sampling points, and a single picture corresponding to the 3D model. The feature space is generated through a single input picture. The sampling points of the 3D model are used as query points to predict the implicit function values ​​of the query points. The loss is calculated based on the predicted implicit function values ​​and the actual implicit function values ​​to complete the fitting of the implicit function field.

[0006] Reconstruction phase: After the training phase is completed, a feature space is generated for a single input image, and several points are sampled in the observation space, which constitute query points; the implicit function values ​​of the query points are predicted; the query points and their predicted implicit function values ​​are input into the marching cubes algorithm to reconstruct the 3D model.

[0007] Furthermore, the training phase specifically includes the following steps:

[0008] Step 11, predicting a 2.5D sketch of the input image, including predicting a depth map and a normal map;

[0009] Step 12, using a generative adversarial network to predict the invisible part of the object in the input image and generate an invisible part image;

[0010] Step 13, using a stacked hourglass network to extract features from the 2.5D sketch and the invisible part image to obtain a feature cube;

[0011] Step 14, querying the feature space formed by the feature cube using the query point, combining the features obtained by the query to form the front features of the object, and using the features obtained by the query in the feature cube of the invisible part image as the back features of the object;

[0012] Step 15, using a feature selector to select front features and back features;

[0013] Step 16, put the selected features into the multi-layer perceptron MLP to obtain the implicit function field.

[0014] Furthermore, the reconstruction phase specifically includes the following steps:

[0015] Step 21, predicting a 2.5D sketch of the input image, including predicting a depth map and a normal map;

[0016] Step 22, using a generative adversarial network to predict the invisible part of the object in the input image and generate an invisible part image;

[0017] Step 23, using a stacked hourglass network to extract features from the 2.5D sketch and the invisible part image to obtain a feature cube;

[0018] Step 24, querying the feature space formed by the feature cube using the query point, combining the features obtained by the query to form the front features of the object, and using the features obtained by the query in the feature cube of the invisible part image as the back features of the object;

[0019] Step 25, using a feature selector to select the front face feature and the back face feature;

[0020] Step 26, putting the selected features into the multi-layer perceptron MLP to obtain the implicit function field;

[0021] Step 27, running the marching cubes algorithm according to the implicit function field to reconstruct a three-dimensional model.

[0022] Furthermore, the prediction of 2.5D sketches is completed using two neural networks (the network architecture is based on ResNet). One neural network converts the input image into its corresponding depth map. The image is a single-channel image, and each pixel corresponds to the depth value of the predicted object in the observation space (the observation space is the three-dimensional space after three-dimensional reconstruction); the other neural network converts the input image into its corresponding normal map. Each pixel is the normal value of the predicted object in the observation space. The normal value is a three-dimensional vector stored in a three-channel image.

[0023] Furthermore, in the prediction of the 2.5D sketch in the training phase: using the input image and the 2.5D sketch training pair corresponding to the input image, the two neural networks have loss functions, and the loss functions include L1 loss and Gan loss;

[0024] The loss function is:

[0025]

[0026] Where G is the generator parameter of the neural network, and D is the discriminator parameter of the neural network;

[0027] Including an L1 loss:

[0028]

[0029] Where y is the true value of the 2.5D sketch, and G(x,z) is the 2.5D sketch predicted by the neural network;

[0030] With a GAN loss:

[0031]

[0032] Furthermore, the Pix2PixHD model in the generative adversarial network is used to predict the invisible part of the object in a single image, and the training samples use sample pairs of the input image and the invisible part image. This step aims to make use of the powerful representation ability of the generative adversarial network and the achievements in the field of image translation to predict the invisible part in advance, so as to generate a more reasonable inference about the invisible part and avoid missing or flawed invisible parts in the final reconstruction result. This task involves predicting the invisible part, which is more difficult than simple normal and depth prediction and requires a network with more powerful representation capabilities. The Pix2PixHD model can complete the image translation work with high quality. Specifically, it transfers a picture from domain A to domain B, which is to transfer the picture of the visible part to the invisible part. Through this network, we can get sufficiently detailed and reasonable inferences about the invisible part.

[0033] Furthermore, pixel-aligned features: The features of the depth map, normal map, and invisible part of the image are extracted by stacking hourglass networks, and a feature cube of 512*512*N dimensions is obtained; N is the feature vector dimension; the feature vector dimension of the depth map and normal map is 128 dimensions, and the feature vector of the invisible part of the image is 256 dimensions. The space defined by the feature cube is the feature space; the features in the feature cube are pixel-aligned features.

[0034] Furthermore, the query point is used to query the feature space formed by the feature cube, and the method for obtaining the feature is as follows: for a query point, the query point is projected from the observation space to the feature space to obtain the projection point, and the four features closest to the projection point in the feature space are bilinearly interpolated to obtain a feature, which is called the query-obtained feature; the position of the projection point is the first two components of the query point, that is, the x and y coordinates. In order to obtain the front and back features to assist in reconstruction, the two feature vectors obtained from the 2.5D sketch are combined to obtain a 256-dimensional front feature vector, and the features of the invisible part of the image are used as the back feature vector, which is also 256-dimensional.

[0035] Furthermore, the feature selector performs feature selection based on the front feature vector, back feature vector and z component of the query point, and finally obtains a 256-dimensional feature vector; the function of the feature selector is:

[0036] Select(F front ,F back ,z)=F front +(F back -F front )(z 2 (3-2z)),z∈[0,1];

[0037] where F front For positive features, F back is the back feature, and z is the third component of the query point, that is, the depth coordinate.

[0038] Furthermore, the feature vector obtained by the feature selector is used as the input of the multi-layer perceptron MLP to obtain an output value between 0 and 1, that is, the predicted implicit function value for the query point. For any query point, there is an output value, that is, the fitted implicit function field.

[0039] Furthermore, in the training phase, the hourglass network, feature selector, and multi-layer perceptron MLP are stacked and trained together, and the following backbone network loss function is used:

[0040]

[0041] Where g is the predicted implicit function, and the value it returns is the predicted implicit function field; f * (Xi ) represents the true implicit function value in the sample.

[0042] Furthermore, in the reconstruction stage, the observation space is uniformly sampled to obtain the predicted implicit function value corresponding to each point in the observation space, and the predicted implicit function value is input into the marching cubes algorithm to extract the 0.5 isosurface to obtain the reconstruction result.

[0043] Furthermore, before predicting the 2.5D sketch of the input image, it is first determined whether the pixel dimension of the input image is 512*512; if not, it is scaled to 512*512.

[0044] The present invention has the following beneficial effects: the method of the present invention can perform three-dimensional reconstruction of an object through a single picture, eliminating the tedious steps of multi-view image acquisition and camera calibration; it improves the serious lack of prior conditions in single-picture reconstruction and can obtain fine details in the visible part. The reconstructed object has rich details in the visible part of the single picture and can generate reasonable speculations in the invisible part; the number of neural network parameters used is small, and the reconstruction process can be run on an ordinary computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The present invention is a flowchart of a training process of a 3D reconstruction method based on a single image according to an embodiment of the present invention.

[0046] Figure 2 The figure is a flowchart of a reconstruction process of a three-dimensional reconstruction method based on a single image according to an embodiment of the present invention.

[0047] Figure 3 This is a partial flow chart of feature extraction of a 3D reconstruction method based on a single image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings.

[0049] Embodiment: The single-image-based 3D reconstruction method of this embodiment is a single-image reconstruction method assisted by a 2.5D sketch based on a pixel-aligned implicit function.

[0050] The method of this embodiment mainly includes a training process and a reconstruction process.

[0051] (I) Feature extraction stage

[0052] The feature extraction stage is common to both the training and reconstruction processes. Specifically, Figure 3As shown, the training stage takes the image of the training sample as input, and the reconstruction stage takes the single image of the object to be reconstructed as input; it is determined whether the pixel dimension of the input image is 512*512; if not, it is scaled to 512*512; after the 2.5D sketch prediction network and the invisible part inference network, a prediction of the 2.5D sketch including the depth map and the normal map and an inferred image of the invisible part of the object are generated.

[0053] The prediction of 2.5D sketch is completed by two neural networks (network architecture is based on ResNet). One neural network converts the input image into its corresponding depth map. The image is a single channel, and each pixel corresponds to the depth value of the predicted object in the observation space (the observation space is the three-dimensional space after three-dimensional reconstruction); the other neural network converts the input image into its corresponding normal map. Each pixel is the normal value of the predicted object in the observation space. The normal value is a three-dimensional vector stored in a three-channel image. Thanks to the great advantages of deep learning in images, we can obtain stronger and more accurate prior information based on depth information and normal information than without auxiliary information, thereby obtaining more accurate and detailed three-dimensional reconstruction.

[0054] The Pix2PixHD model in the generative adversarial network is used to predict the invisible part of the object in a single image. The training samples use sample pairs of the input image and the invisible part image. This step aims to use the powerful representation ability of the generative adversarial network and the achievements in the field of image translation to predict the invisible part in advance, so as to generate a more reasonable inference about the invisible part and avoid missing or flawed invisible parts in the final reconstruction result. The Pix2PixHD model can complete the image translation work with high quality. Specifically, it transfers a picture from domain A to domain B, which is to transfer the picture of the visible part to the invisible part. Through this network, we can get sufficiently detailed and reasonable inferences about the invisible part.

[0055] The three images are fed into three different stacked hourglass networks to extract the features of the depth map, normal map, and invisible part of the image, respectively, to obtain a feature cube of 512*512*N dimensions, where N is the feature vector dimension; the feature vector dimension of the depth map and normal map is 128, and the feature vector of the invisible part of the image is 256. The space defined by the feature cube is the feature space; the features in the feature cube are pixel-aligned features.

[0056] (II) Training Phase

[0057] like Figure 1As shown in the figure, the task completed in the training phase is mainly to fit the pixel alignment implicit function with the neural network. The training samples include the points sampled on the 3D model and their corresponding implicit function values, and the single picture corresponding to the 3D model. In this step, the sampling points of the 3D model are mainly used as query points, and the loss is calculated and the network parameters are updated based on the predicted implicit function values ​​and the actual implicit function values. Specifically:

[0058] 1. In the feature extraction stage, a single input image is used to generate a feature space;

[0059] In the prediction of 2.5D sketches in the training phase: the input image and the 2.5D sketch training pair corresponding to the input image are used, and the Adam optimizer is used for optimization during training. The two neural networks for the prediction of 2.5D sketches have loss functions, including L1 loss and Gan loss;

[0060] The loss function is:

[0061]

[0062] Where G is the generator parameter of the neural network, and D is the discriminator parameter of the neural network;

[0063] Including an L1 loss:

[0064]

[0065] Where y is the true value of the 2.5D sketch, and G(x,z) is the 2.5D sketch predicted by the neural network;

[0066] With a GAN loss:

[0067]

[0068] During the training phase of the invisible part images, the training samples use sample pairs of the input image and the invisible part image to complete the training of the generative adversarial network alone.

[0069] 2. Take the sampling points in the training samples as query points, project the query points from the observation space to the feature space, obtain the projection points, perform bilinear interpolation on the four features closest to the projection points in the feature space to obtain a feature, which is called the query feature; the position of the projection point is the first two components of the query point, that is, the x and y coordinates. In order to obtain the front and back features to assist reconstruction, the two feature vectors obtained from the 2.5D sketch are combined to obtain a 256-dimensional front feature vector, and the features of the invisible part of the image are used as the back feature vector, which is also 256-dimensional.

[0070] 3. After the feature selector, the front feature, the back feature, and the z depth coordinate of the query point are taken as input, and finally a 256-dimensional feature vector (i.e., pixel alignment feature) is obtained; the function of the feature selector is:

[0071] Select(F front ,F back ,z)=F front +(F back -F front )(z 2 (3-2z)),z∈[0,1];

[0072] where F front For positive features, F back is the back feature, and z is the third component of the query point, that is, the depth coordinate.

[0073] 3. The feature vector obtained by the feature selector is used as the input of the multi-layer perceptron MLP to obtain an output value between 0 and 1, that is, the predicted implicit function value for the query point. For any query point, there is an output value, that is, the fitted implicit function field.

[0074] 4. Stack the hourglass network, feature selector, and multi-layer perceptron MLP together for training. Use the RMSProp optimizer for training and use the following backbone network loss function:

[0075]

[0076] Where g is the predicted implicit function, and the value it returns is the predicted implicit function field; f * (X i ) represents the true implicit function value in the sample.

[0077] According to the predicted values ​​and the actual values, the loss is calculated through the backbone network loss function, and the parameters of the stacked hourglass network, feature selector, and multi-layer perceptron MLP are updated to complete the fitting of the implicit function field.

[0078] 3. Reconstruction Phase

[0079] like Figure 2 As shown, when performing this stage, it is assumed that all networks have been trained. In this stage, inputting a picture can get the corresponding 3D model reconstruction. The specific steps of this stage are as follows:

[0080] 1. The input image first goes through the feature extraction stage as described above to obtain the feature space;

[0081] 2. According to the structure required by marching cubes, sample several points in space, which constitute query points;

[0082] 3. Use the query point to project into the feature space to obtain the projection point. Perform bilinear interpolation on the four features closest to the projection point in the feature space to obtain a feature, which is called the query feature. The position of the projection point is the first two components of the query point, i.e., the x and y coordinates. In order to obtain the front and back features to assist in reconstruction, the two feature vectors obtained from the 2.5D sketch are combined to obtain a 256-dimensional front feature vector. The features of the invisible part of the image are used as the back feature vector, which is also 256-dimensional.

[0083] 4. After the feature selector, the front feature, the back feature, and the z depth coordinate of the query point are taken as input, and finally a 256-dimensional feature vector (i.e., pixel alignment feature) is obtained; the function of the feature selector is:

[0084] Select(F front ,F back ,z)=F front +(F back -F front )(z 2 (3-2z)),z∈[0,1];

[0085] where F front For positive features, F back is the back feature, and z is the third component of the query point, that is, the depth coordinate.

[0086] 5. Input the obtained 256-dimensional feature vector into the multi-layer perceptron MLP to obtain the predicted implicit function value of the query point; input the query point and its predicted implicit function value into the marching cubes algorithm to extract the 0.5 isosurface and reconstruct the 3D model. Because an implicit function field that can output implicit function values ​​for any query point has been obtained, it is only necessary to query the predicted implicit function value on the pattern required by the marching cubes algorithm (generally many vertices of regular hexahedrons) to obtain the reconstruction result according to the marching cubes algorithm.

[0087] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method based on a single image, characterized in that: The following steps are involved: Training phase: The training samples include sampling points on the 3D model, the actual implicit function values ​​corresponding to the sampling points, and a single picture corresponding to the 3D model. The feature space is generated through a single input picture. The sampling points of the 3D model are used as query points to predict the implicit function values ​​of the query points. The loss is calculated based on the predicted implicit function values ​​and the actual implicit function values ​​to complete the fitting of the implicit function field. Step 11, predicting the 2.5D sketch of the input image. The prediction of the 2.5D sketch is completed by two neural networks. One neural network converts the input image into its corresponding depth map. The depth map is a single channel, and each pixel corresponds to the depth value of the predicted object in the observation space; the other neural network converts the input image into its corresponding normal map. Each pixel is the normal value of the predicted object in the observation space. The normal value is a three-dimensional vector and is stored in a three-channel image. Step 12, using a generative adversarial network to predict the invisible part of the object in the input image and generate an invisible part image; Step 13, using a stacked hourglass network to extract features from the 2.5D sketch and the invisible part image, to obtain a feature cube of 512*512*N dimensions, where N is the feature vector dimension; the feature vector dimension of the depth map and the normal map is 128 dimensions, and the feature vector dimension of the invisible part image is 256 dimensions; The space defined by the feature cube is the feature space; the features in the feature cube are pixel-aligned features; Step 14, querying the feature space formed by the feature cube using the query point, combining the features obtained by the query to form the front features of the object, and using the features obtained by the query in the feature cube of the invisible part image as the back features of the object; Step 15, using a feature selector to select front features and back features; Step 16, putting the selected features into the multi-layer perceptron MLP to obtain the implicit function field; Reconstruction phase: After the training phase is completed, a feature space is generated for a single input image, and several points are sampled in the observation space, which constitute query points; the implicit function values ​​of the query points are predicted; the query points and their predicted implicit function values ​​are input into the marching cubes algorithm to reconstruct the 3D model; The feature space formed by the feature cube is queried with the query point to obtain the feature. For a query point, the query point is projected from the observation space to the feature space to obtain the projection point. The four features closest to the projection point in the feature space are bilinearly interpolated to obtain a feature, which is called the queried feature. The features extracted from the depth map and the normal map in the 2.5D sketch are spliced ​​to obtain a 256-dimensional front feature vector. The back feature vector of the invisible part of the image is also 256-dimensional.

2. The single-image-based three-dimensional reconstruction method according to claim 1, characterized in that: The reconstruction phase includes the following steps: Step 21, predicting a 2.5D sketch of the input image, including predicting a depth map and a normal map; Step 22, using a generative adversarial network to predict the invisible part of the object in the input image and generate an invisible part image; Step 23, using a stacked hourglass network to extract features from the 2.5D sketch and the invisible part image to obtain a feature cube; Step 24, querying the feature space formed by the feature cube using the query point, combining the features obtained by the query to form the front features of the object, and using the features obtained by the query in the feature cube of the invisible part image as the back features of the object; Step 25, using a feature selector to select the front face feature and the back face feature; Step 26, putting the selected features into the multi-layer perceptron MLP to obtain the implicit function field; Step 27, running the marching cubes algorithm according to the implicit function field to reconstruct a three-dimensional model.

3. The single-image-based 3D reconstruction method according to claim 1, characterized in that: In the prediction of the 2.5D sketch in the training phase: using the input image and the 2.5D sketch training pair corresponding to the input image, the two neural networks have loss functions, including L1 loss and Gan loss; the loss function is: ； in are the generator parameters of the neural network, is the discriminator parameter of the neural network; The L1 loss is: ； in is the true value of the 2.5D sketch, is the 2.5D sketch predicted by the neural network; The GAN loss is: 。 4. The single-image-based 3D reconstruction method according to claim 1 or 2, characterized in that: Use the Pix2PixHD model in the Generative Adversarial Network to predict the invisible parts of the object in a single image; The invisible part prediction in the training phase uses sample pairs of input image and invisible part image.

5. The single-image-based 3D reconstruction method according to claim 1 or 2, characterized in that: The feature selector performs feature selection based on the front feature vector, back feature vector and z component of the query point, and finally obtains a 256-dimensional feature vector; the function of the feature selector is: ； in For positive features, For the back feature, is the depth coordinate of the query point.

6. The single-image-based 3D reconstruction method according to claim 5, characterized in that: The feature vector obtained by the feature selector is used as the input of the multi-layer perceptron MLP to obtain an output value between 0 and 1, that is, the predicted implicit function value for the query point. For any query point, there is an output value, that is, the fitted implicit function field.

7. The single-image-based 3D reconstruction method according to claim 1, characterized in that: During the training phase, the stacked hourglass network, feature selector, and multi-layer perceptron MLP are trained together, and the following backbone network loss function is used: ； in is the predicted implicit function, and the value it returns is the predicted implicit function field; Represents the true implicit function value in the sample.

8. The single-image-based 3D reconstruction method according to claim 1 or 2, characterized in that: In the reconstruction stage, the observation space is uniformly sampled to obtain the predicted implicit function value corresponding to each point in the observation space. The predicted implicit function value is input into the marching cubes algorithm to extract the 0.5 isosurface to obtain the reconstruction result.

9. The single-image-based 3D reconstruction method according to claim 1 or 2, characterized in that: Before predicting the 2.5D sketch of the input image, first determine whether the pixel dimension of the input image is 512*512; if not, scale it to 512*512.

Images