A single-view 3D reconstruction method based on structural features and implicit features

Abstract

The present invention discloses a single-view 3D reconstruction method based on structural features and implicit features, and the method specifically includes the following steps: inputting an image to a hierarchical reconstruction network to obtain structural features; presetting sampling space points, extracting global features from the input image through a convolutional neural network and saving feature maps, and obtaining rough features from the global features through a tree structure graph convolutional network, 3D convolution and interpolation operations; obtaining local features through upsampling feature maps, standard viewpoint conversion and local feature splicing and dimensionality reduction operations of different channels; processing the preset spatial point coordinates to obtain point features, splicing the point features with structural features, rough features and local features respectively, putting them into a three-branch decoder for decoding and then adding them to obtain the SDF value of the reconstructed target; and using the Marching Cube algorithm to extract the zero-value surface to obtain the reconstruction result. The method of the present invention does not require knowing the camera parameters in advance. Compared with the existing methods, the present invention can better reconstruct objects with complex shapes.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision three-dimensional reconstruction, and in particular to a single-view three-dimensional reconstruction method based on structural features and implicit features. Figure 3 Dimensional reconstruction method. Background Art

[0002] 3D reconstruction is an important research direction in the field of computer vision, which aims to reconstruct the 3D geometry and surface details of objects from 2D images. This technology has broad potential and application value in multiple application scenarios such as autonomous driving, virtual reality, and augmented reality. However, traditional 3D reconstruction methods still face many challenges in single-view reconstruction tasks, such as insufficient topological constraints on object shapes.

[0003] Many shapes in the 3D world (especially man-made objects) have highly structured features, such as hierarchical features and nested symmetries. These structural features show rich diversity in objects of the same category, such as different styles of chairs, tables and other furniture models. Therefore, in single-view Figure 3 When reconstructing a complex topological shape, capturing the hierarchical structure of the object is a key factor in reconstructing the complex topological shape. The hierarchical structure of the object contains important information about the geometric distribution and topological structure of the shape. By analyzing the hierarchical structure, complex shapes can be reconstructed and geometric details can be optimized. In recent years, some methods rely on the extraction of local features and the inference of rough shapes to perform single-view Figure 3 Dimensional reconstruction lacks effective constraints on the global topological structure, so it is difficult to achieve ideal results when reconstructing complex shapes. Summary of the invention

[0004] In order to improve the shortcomings of existing methods in that they are difficult to accurately reconstruct complex three-dimensional objects in a single image due to the lack of effective topological constraints, the present invention combines the advantages of existing methods and designs a single-view reconstruction method based on structural features and implicit features. Figure 33D reconstruction method. The method uses a hierarchical reconstruction network to extract the hierarchical structure information of the object from the input two-dimensional image and encodes it into structural features; preset sampling space points, input the image through a convolutional neural network to extract global features and save the feature map, let the global features pass through the tree structure graph convolution network, 3D convolution and interpolation operations in sequence to obtain rough features; obtain local features by upsampling the feature map, standard viewpoint conversion and splicing and dimensionality reduction operations of local features of different channels; process the preset space point coordinates through a multi-layer perceptron to obtain point features, splice the point features with the structural features, rough features and local features respectively, put them into a three-branch decoder for decoding and then add them to obtain an implicit representation of the reconstructed target (signed distance field, SDF), extract the zero-value surface after the final SDF prediction value passes through the Marching Cube algorithm to convert the implicit representation into an explicit three-dimensional model, and the method of the present invention can directly reconstruct the three-dimensional surface of the object from the single-view image without knowing or estimating the camera parameters in advance.

[0005] Specifically, the present invention provides a single-view method based on structural features and implicit features. Figure 3 The reconstruction method mainly includes the following steps:

[0006] S1, input the two-dimensional image to the hierarchical structure reconstruction network, extract the hierarchical structure information in the image, and obtain the structural features used to represent the hierarchical structure of the object through encoding;

[0007] S2. Preset the sampled spatial points in the standard three-dimensional space, use the convolutional neural network (ResNet50) to extract the global features of the input image, save the feature map in the extraction process, output the global features as a rough point cloud through the tree-structured graph convolutional network (TreeGCN), use 3D convolution to convert the rough point cloud into a probabilistic occupancy grid, map the grid to a high-dimensional matrix through a multi-layer 3D convolutional network, and then interpolate and output the rough features of each preset spatial point;

[0008] S3, upsample the feature maps of the first four blocks saved by ResNet50 in S2 to the original image size, and find the position of each preset spatial point on the feature map through standard viewpoint conversion to obtain 1×1 local features with different numbers of channels, concatenate the obtained local features into high-dimensional features with a dimension of 1856, and then reduce the high-dimensional features to the final local features of 1024 dimensions through the convolution layer;

[0009] S4, encode the preset spatial point coordinates p(x, y, z) through a multi-layer perceptron (MLP) to obtain a point feature with a dimension of 512, concatenate it with the structural feature, the rough feature and the local feature respectively, let the concatenation result be decoded by a three-branch decoder respectively, add the three parts of the decoding results to obtain the final SDF value of each preset spatial point;

[0010] S5. The final SDF values ​​of all preset spatial points are input into the Marching Cube algorithm to extract the zero-value surface, and finally the three-dimensional mesh shape of the object is reconstructed.

[0011] The hierarchical structure reconstruction network described in step S1 is composed of a convolutional neural network (ResNet50) and a multi-layer perceptron (MLP). Specifically, ResNet50 is first used to extract hierarchical structure information from the input image, and then the hierarchical structure information is encoded into structural features through MLP;

[0012] The preset spatial points in step S2 are 256×256×256 sampling points with uniform density preset in a standard three-dimensional space;

[0013] The number of channels of the local features obtained in step S3 is 64, 256, 512, and 1024 respectively. The local features of each preset spatial point contain the local information of the position. The dimension of the concatenated feature vector is 64 + 256 + 512 + 1024 = 1856. Since the high-dimensional feature vector after concatenation is too large, it is not conducive to subsequent processing. Therefore, the convolution layer is used to reduce the dimension from 1856 to 1024.

[0014] The three-branch decoder described in step S4 includes three different MLP networks, which are responsible for generating SDF values ​​of topological structure, global rough shape and local details respectively, and the three SDF values ​​are added together to obtain the final SDF value.

[0015] Furthermore, the present invention provides a single-view method based on structural features and implicit features. Figure 3 In a specific implementation case of the present invention, the expression of the total loss function is as follows:

[0016] ,

[0017] in, L S It is a weighted loss function used to recover the details near and inside the isosurface to ensure the shape details of the reconstructed object; L V is a binary cross entropy loss function, which is used to supervise the prediction of the probability occupancy grid to ensure its accuracy; L CD is the Chamfer distance loss function, which is used to constrain the reconstruction error from image to point cloud and ensure the consistency between the point cloud and the input image.

[0018] Beneficial effects: The present invention introduces hierarchical features to correct the topological shape of the object, combines rough features and local features, and adopts signed distance function (SDF) as a representation form to effectively improve the reconstruction accuracy and topological structure of the object; the present invention does not need to rely on camera parameters, and effectively integrates SDF information at different levels to help relevant personnel better reconstruct the surface of objects with complex shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is based on the single view of structural features and implicit features Figure 3 Schematic diagram of the general process of maintenance and reconstruction;

[0020] Figure 2 It is a schematic diagram of the reconstruction result of the present invention on the chair category of the ShapeNet dataset;

[0021] Figure 3 It is a schematic diagram of the reconstruction results of different viewing angles of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0023] like Figure 1 , Figure 2 , Figure 3 As shown, the present invention provides a single-view method based on structural features and implicit features. Figure 3 The reconstruction method comprises the following steps:

[0024] S1, input the two-dimensional image to the hierarchical structure reconstruction network, extract the hierarchical structure information and encode it to obtain the structural features;

[0025] S2. Preset sampling spatial points in the standard three-dimensional space, use the ResNet50 model to extract the global features of the input image, save the feature map during the extraction process, and convert the global features into a rough point cloud through a tree-structured graph convolutional network (TreeGCN). Convert the rough point cloud into a probabilistic occupancy grid through 3D convolution, map the grid to a high-dimensional matrix through a multi-layer 3D convolutional network, and use interpolation to generate rough features for each preset spatial point;

[0026] S3, upsample the feature maps of the first four blocks saved by ResNet50 in S2 to the original image size, find the position of each preset spatial point on the feature map through standard viewpoint conversion, obtain 1×1 local features with 64, 256, 512, and 1024 channels respectively, concatenate the local features into high-dimensional features with a dimension of 1856, and then reduce their dimension to the final local features of 1024 dimensions through the convolution layer;

[0027] S4, the coordinates p(x, y, z) of the preset spatial point are processed by MLP to obtain a point feature with a dimension of 512, and are spliced ​​with the structural features extracted in step S1, the rough features extracted in step S2, and the local features extracted in step S3, respectively. After splicing, the different splicing results are sent to the three-branch decoder for decoding and addition, so as to obtain the final SDF value of each preset spatial point for reconstruction;

[0028] S5. The final SDF values ​​of all preset spatial points are input into the Marching Cube algorithm to extract the zero-value surface, thereby reconstructing the three-dimensional mesh shape of the object.

[0029] The hierarchical structure reconstruction network described in step S1 is composed of a convolutional neural network (ResNet50) and a multi-layer perceptron (MLP), specifically, ResNet50 is used to extract hierarchical structure information from the input image, and then the hierarchical structure information is encoded into structural features representing the hierarchical structure through MLP;

[0030] The preset sampling space points in step S2 are 256×256×256 sampling points with uniform density preset in the standard three-dimensional space;

[0031] The rough features in step S2 are specifically obtained by extracting the global features of the image from ResNet50 and then decoding them using the TreeGCN network to obtain a rough point cloud, discretizing the rough point cloud and generating a probabilistic occupancy grid through 3D convolution, then using multi-layer 3D convolution to map the occupancy grid to a high-dimensional latent variable matrix, and extracting the rough features of each preset spatial point from the mapped features through multi-scale trilinear interpolation;

[0032] The standard viewpoint transformation described in step S3 is specifically to send the global features and the global point features after encoding the rough point cloud into the space transformer to predict a transformation parameter (similar to the projection parameter), and present the rough prediction of the target object as a standard viewpoint through the transformation parameter, and use the standard viewpoint to define the three-dimensional points corresponding to the rough prediction, so as to accurately map these three-dimensional points to the feature map;

[0033] Step S4: The three-branch decoder is composed of three different MLP networks, which are used to respectively decode the SDF representing the topological structure of the object, the SDF representing the global rough shape of the object, and the SDF representing the local details of the object, and add the three SDF values ​​to obtain the final SDF prediction value;

[0034] The Marching Cube algorithm used in step S5 is implemented by the PyMcube library of Python.

[0035] Furthermore, in a specific implementation case of the present invention, the total loss function is given by L S , L V and L CD composition, L S is the weighted loss function, L S The goal of optimization is to reconstruct the shape details of the object; L V is the binary cross entropy loss, L V The optimization objective is the prediction accuracy of the supervised probability occupancy grid; L CD is the Chamfer distance loss, L CD The optimization objective is the loss from image to point cloud.

[0036] In a specific implementation case of the present invention, all training processes are carried out on the ShapeNet dataset, and the hierarchical data is based on the PartNet dataset of ShapeNet. In the specific implementation case of the present invention, the training process is divided into two stages: hierarchical structure reconstruction network training and SDF prediction network training. The optimization of the hierarchical structure reconstruction network adopts the SGD stochastic gradient descent method, the learning rate is 0.01, and the batch size is 16; the optimization of the SDF prediction network adopts the Adam optimizer, the learning rate is set to 0.0001, and the batch size is 8.

[0037] The corresponding reconstruction results are shown in Figure 2. Figure 2 As shown in Figure 1, the first row of images are input images of the chair category in the ShapeNet dataset, the second row of images are the final reconstruction results, and the third row of images are real objects. Figure 2 As shown in FIG. 1 , the reconstruction method of the present invention has a more outstanding reconstruction effect when the object has a complex topological structure. The reconstruction results at different viewing angles are shown in FIG. Figure 3 As shown: the first row of pictures are input pictures at different viewing angles, and the second row of pictures are the reconstruction results of the present invention, such as Figure 3 As shown, the reconstruction method of the present invention does not depend on camera parameters, and the reconstruction result will not be affected by changes in viewing angle.

[0038] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A single-view 3D reconstruction method based on structural features and implicit features, characterized in that: The method comprises the following steps:

1. Input the image to the hierarchical structure reconstruction network composed of convolutional neural network ResNet50 and multi-layer perceptron MLP, use ResNet50 of the hierarchical structure reconstruction network to extract the hierarchical structure information of the image, and then encode the hierarchical structure information through the MLP of the hierarchical structure reconstruction network to obtain the structural features representing the hierarchical structure; Second, preset spatial points in the standard three-dimensional space, use the convolutional neural network ResNet50 to extract global features from the input image, save the feature map in the extraction process, input the extracted global features into the tree-structured graph convolutional network TreeGCN for decoding to obtain a rough point cloud, discretize the rough point cloud and generate a probability occupancy grid through 3D convolution, then use multi-layer 3D convolution to map the probability occupancy grid to a high-dimensional latent variable matrix to generate mapping features, and extract coarse features for each preset spatial point from the mapping features through multi-scale trilinear interpolation; 3. After upsampling the feature maps of the first 4 blocks saved in ResNet50 in step 2, find the position of each preset spatial point on the feature map through standard viewpoint conversion, and obtain local features with different numbers of channels. These local features are spliced ​​into high-dimensional features, and then the high-dimensional features are reduced to the final local features through the convolution layer; Fourth, the preset spatial point coordinates are processed by a multi-layer perceptron to obtain point features, and the point features are respectively concatenated with the structural features extracted in step one, the rough features extracted in step two, and the final local features extracted in step three, and then decoded and added by a three-branch decoder to obtain the final signed distance field value of each preset spatial point; 5. Use the Marching Cube algorithm to extract the zero-value surface from the final signed distance field value of each preset spatial point obtained in step 4, and reconstruct the explicit three-dimensional mesh model of the object.

2. The single-view 3D reconstruction method based on structural features and implicit features according to claim 1, characterized in that: The hierarchical structure reconstruction network described in step 1 is composed of a convolutional neural network ResNet50 and a multi-layer perceptron MLP. Specifically, ResNet50 is first used to extract hierarchical structure information of the input image, and then the hierarchical structure information is encoded into structural features representing the hierarchical structure through MLP.

3. The single-view 3D reconstruction method based on structural features and implicit features according to claim 1, characterized in that: The number of local feature channels obtained after the standard viewpoint conversion described in step 3 is 64, 256, 512, and 1024, respectively. The obtained local features are then concatenated to obtain 1856-dimensional high-dimensional features, and then the convolution layer is used to reduce the dimension to obtain the final local features of 1024 dimensions.

4. The single-view 3D reconstruction method based on structural features and implicit features according to claim 1, characterized in that: A three-branch decoder will be used in the decoding operation described in step 4. The three-branch decoder is composed of three different multi-layer perceptrons, which are used to decode the signed distance field values ​​representing the topological shape, the global rough shape, and the local detailed shape respectively. The structural features, the rough features, and the final local features are respectively concatenated with the point features and put into the three-branch decoder, and the corresponding signed distance field values ​​are obtained by decoding. The decoding results are added to obtain the final predicted signed distance field value.

Images