A three-dimensional reconstruction method of a multi-view P2VNet network

Abstract

This invention relates to a 3D reconstruction method using a multi-view P2VNet network. Traditional multi-view 3D reconstruction involves acquiring 2D features of an object using complex equipment and then using algorithms to recover the 3D features. These algorithms suffer from high computational cost, poor real-time performance, and susceptibility to environmental factors, leading to unstable 3D reconstruction accuracy. This invention first redesigns the encoding and decoding networks to achieve better end-to-end mapping from 2D to 3D features. Second, it uses 3DFocal Loss as the model's loss function to address the issues of uneven spatial distribution in the mesh and the difficulty in partitioning parts of the mesh. Finally, it proposes a multi-view adaptive fusion network to fuse 3D features generated from different views and uses an adaptive feature perception network to generate the final voxels from the fused features. The average accuracy of the P2VNet network for target reconstruction in this invention reaches 68.2%, which is 9.5% higher than the average accuracy of 3D-R2N2 and 1.5% higher than the average accuracy of Pix2Vox-A.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and more specifically to a 3D reconstruction method using a multi-view P2VNet network. Background Technology

[0002] 3D reconstruction is a significant problem in robotics, CAD, virtual reality, and augmented reality. Traditional methods, such as SfM (Structured for Motion) and SLAM (Simultaneous Localization and Mapping), struggle to establish feature correspondences when performing cross-view matching of image features.

[0003] To address these issues, scholars have proposed several deep learning-based 3D reconstruction networks in recent years. For example, networks such as 3D-R2N2, Pix2Vox, and PSGN have achieved 3D voxel reconstruction from single or multiple views.

[0004] In the aforementioned networks, PSGN cannot robustly reconstruct complete, high-quality shapes from a single image, resulting in low accuracy. Secondly, the recurrent neural network used in 3D-R2N2 suffers from long-term memory loss; when the same set of images is input in different orders, the network cannot estimate consistent 3D voxels. Simultaneously, the large number of parameters in the LSTM network leads to significant time consumption. To address these problems in reconstruction networks, Xie et al. proposed Pix2Vox. First, the Pix2Vox network consists of multiple parallel encoder-decoder blocks, each predicting a coarse volumetric grid from the input view, thus eliminating the influence of the input image order. Second, a context-aware fusion module selects high-quality features from the coarse 3D voxels for reconstruction and fuses them to generate a refined 3D voxel. However, in the process of generating 3D voxels from 2D images, the end-to-end mapping of features cannot be directly reflected. Furthermore, because the context fusion module only uses a 3×3 receptive field for contextual feature perception, the target edge effect after single-view reconstruction is poor, resulting in poor edge effect after multi-view fusion. Summary of the Invention

[0005] In view of this, in order to solve the problems of large computational load, poor real-time performance, and easy susceptibility to environmental factors affecting accuracy in the prior art, the present invention provides a three-dimensional reconstruction method for multi-view P2VNet networks.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a three-dimensional reconstruction method for a multi-view P2VNet network, characterized by the following steps:

[0007] Step 1: Dataset Construction

[0008] Two-dimensional image data of multiple targets are collected. For the same target, the three-dimensional voxel features of the target are calculated using CAD software. All two-dimensional images and corresponding three-dimensional voxel features are divided into training set and test set in a 3:1 ratio and transformed into input data required by the deep model.

[0009] Step 2: Extract features from the two-dimensional images in the training and test sets using the encoding module to generate 2D feature maps at different scales;

[0010] Step 3: The depth estimation module fuses multiple 2D features to generate 3D features.

[0011] By using the 2D features extracted in Step 2 as the front-layer features, using Sigmoid to generate 2D feature space probabilities, then using a residual network to perform depth perception on the front-layer features, then multiplying the depth-perceived features with the feature space probabilities to obtain the back-layer features, and finally concatenating the front and back-layer features to generate the final 3D features.

[0012] Step 4: The decoding module generates coarse 3D voxels from the 3D features generated by the depth estimation module through a 3D decoding network;

[0013] Step 5: Use the multi-scale context feature perception module to perform multi-view fusion, which merges the 2D view reconstruction results from different angles to enrich the final 3D voxel features and generate accurate reconstruction results.

[0014] Step 6: Repeat steps 2 through 5 on the training set until the entire model converges;

[0015] Step 7: Save the multi-view data obtained after training in Step 6. Figure 3 The system reconstructs a 3D model and performs 3D reconstruction tests on 2D images in the test set to generate corresponding 3D voxel models.

[0016] The specific method for Step 2 above is as follows:

[0017] The encoding module extracts features from the 2D images in the training and test sets from step 1. 2D feature extraction uses the convolutional module of the ResNet50 network to extract features from a 256×256×3 image, resulting in a 128×32 image. 2 256×16 2 512×8 2 1024×4 2 1024×2 2 Size feature map.

[0018] The specific method for Step 3 above is as follows:

[0019] The calculation process for depth estimation is shown in formulas (1) and (2).

[0020] F′ i =D(F i )×S(F i (i=1,2,...m) (1)

[0021] Where m is the total number of channels, F′ i Let D(F) represent the two-dimensional feature of the i-th subsequent layer. i S(F) represents the i-th depth-sensing 2D feature. i ) represents the two-dimensional spatial probability of the i-th feature;

[0022] M i =F i +F′ i (i=1,2,...m) (2)

[0023] Where F′ i Let F′ represent the two-dimensional feature of the i-th depth estimate. i M represents the i-th fused two-dimensional feature. i Let represent the three-dimensional spatial probability of the i-th feature.

[0024] The specific method for Step 4 above is as follows:

[0025] The decoding module generates coarse 3D voxels from the 3D basic unit group generated in Step 3 through a 3D feature upscaling network and a 3D feature perception network.

[0026] (1) 3D Feature Upgrading Network

[0027] The 3D feature generation network consists of four upsampling networks. The upsampling block is a 3D deconvolutional network that upscales the 3D features. Then, Reshape converts the 3D features generated by depth estimation into 3D features. Finally, a 3D convolutional network is used to fuse the 3D features generated by the encoding network with the upscaled features.

[0028] (2) 3D Feature Perception Network

[0029] After 3D feature upscaling, a perceptual network is first used to adjust the generated 3D features to match those generated in the frontal view. This perceptual network uses 1×1 3D convolutions and then performs batch normalization and Leaky ReLU activation to generate 3D features through three depth perception passes. The features from each pass are then fused along the channel boundaries. Finally, two 3D convolutional networks are used on the fused features to change the feature channels to 8 and 1, selecting the optimal channel feature. The first convolutional network uses 3×1 kernels... 3The first convolutional network consists of 3D convolutions with padding of 1, 3D batch normalization, and ReLU activation; the second convolutional network consists of convolutional kernels with 3... 3 It consists of 3D convolutions with padding of 1, 3D batch normalization, and a Sigmoid activation function.

[0030] The specific method for Step 5 above is as follows:

[0031] (1) The rough 3D voxels generated for each view are perceptualized through 4 multi-scale feature blocks;

[0032] (2) Then add the features of the four feature perception blocks together along the channel dimension;

[0033] (3) Then use a three-dimensional convolutional network to extract the perceptual features of each view, and use SoftMax to normalize the probability of the extracted features;

[0034] (4) The rough voxel generated for each view is multiplied by the context-aware probability of that view to generate the context-aware spatial probability of each view.

[0035] (5) The context-aware spatial probabilities of all views are summed to obtain the 3D voxel features of multi-view fusion.

[0036] The specific method for Step 6 above is as follows:

[0037] The 3D reconstruction model of the multi-view target was trained using the training set data. During training, the FocalLoss loss function was used as the loss function. Focal Loss improves the cross-entropy function by adding a class weight and sample difficulty weight adjustment factor (1-pt)γ to the cross-entropy loss. The Focal Loss loss function is shown in Equation (3):

[0038] FL(p t )=-α t (1-p t ) γ log(p t (3)

[0039] Where y = 1 indicates that the voxel mesh is occupied, and the weight of αt is α; y = 0 indicates that the voxel mesh is not occupied, and αt is αt. t The weight is 1-α, through α t Optimize the problem of voxel imbalance; adjust the sample difficulty weighting factor p. t When y = 1, p t The weight is set to p. When y = 0, p is set to p. t The weights are set to 1-p, and finally passed through (1-p) tThe combination factor of α and γ is used to adjust the grids in the voxel that are difficult to distinguish as occupied or not. Both α and γ are gradually adjusted based on prior knowledge and experimental results, and finally set parameter values ​​suitable for the model.

[0040] Based on the above conditions of y being 1 and 0, we can apply the following to αt and p. t The formula for 3DFocal Loss is obtained by setting the weights, as shown in formula (4).

[0041]

[0042] The specific steps of step 7 above are as follows:

[0043] During training, the weights of models with higher average accuracy are saved. After training is completed, the trained model parameters are saved. When using the test set for 3D reconstruction, the trained model is used directly to generate 3D voxels, and the average accuracy of each model and the average accuracy of the entire test set are obtained.

[0044] Compared with the prior art, the present invention has the following advantages and effects:

[0045] 1) In the depth estimation calculation process given in this invention, a depth estimation network structure is introduced. By taking the extracted 2D features as the previous layer features, the Sigmoid is used to generate the 2D feature space probability of the previous layer features, and then the residual network is used to perform depth perception on the previous layer features, and formulas (1) and (2) are proposed. Compared with the existing depth estimation methods, it can achieve a smoother conversion from 2D features to 3D features. Moreover, compared with 3D-R2N2, which generates three-dimensional features from two-dimensional features through LSTM, it has fewer parameters and faster operation speed.

[0046] 2) This invention employs a context-aware multi-scale fusion module, increasing the receptive field and generating richer context-aware features. When target edges are difficult to reconstruct, utilizing fusion features from different receptive fields allows for better selection of higher-scoring 3D voxels, thereby improving the final fusion effect. The average accuracy of P2VNet three-view target reconstruction reaches 0.712. This is 9.5% and 1.5% higher than the average accuracy of 3D-R2N2 and Pix2Vox-A, respectively.

[0047] 3) Experimental results on the ShapeNet dataset show that the proposed method outperforms existing methods in terms of accuracy. Furthermore, the algorithm has strong generalization ability when reconstructing invisible 3D objects. Attached Figure Description

[0048] Figure 1 This is a diagram illustrating the overall structure of the model provided by the present invention.

[0049] Figure 2 The encoder structure diagram of the model provided by this invention;

[0050] Figure 3 A decoder structure diagram of the model provided by this invention;

[0051] Figure 4 A diagram of the model depth perception module provided by this invention;

[0052] Figure 5 A multi-scale context-aware module diagram of the model provided by this invention;

[0053] Figure 6 The reconstruction results of a portion of the model provided by this invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] The principle of the method of this invention is as follows: First, multiple front and rear layer depth estimation modules are added to the encoder to generate corresponding 3D features from 2D features of multiple sizes; Second, in the multi-view feature fusion network, this invention uses multi-size receptive fields, thereby achieving higher quality 3D feature reconstruction.

[0056] This embodiment provides a 3D reconstruction method using a multi-view P2VNet network, comprising four modules: an encoding module, a depth estimation module, a decoding module, and a multi-scale context-aware module. The specific network structure is as follows: Figure 1 As shown, it includes the following steps:

[0057] Step 1: Building the dataset

[0058] Two-dimensional image data of multiple targets are collected. For the same target, the three-dimensional voxel features of the target are calculated using CAD software. All two-dimensional images and corresponding three-dimensional voxel features are divided into training set and test set in a 3:1 ratio and converted into input data required by the deep model.

[0059] In this embodiment, by inputting two-dimensional RGB images of the same target from different angles into the encoding network, the images are preprocessed to change the image pixels to 128×128, and then converted into a tensor of size 3×128×128 through preset parameters to obtain the training set and the test set.

[0060] Step 2, encoding module as follows Figure 2As shown, feature extraction is performed on the two-dimensional images in the training and test sets of step 1. 2D feature extraction uses convolutional blocks to extract features from a 256×256×3 image, resulting in a 256×16 image. 2 512×8 2 1024×4 2 1024×2 2 Size feature map;

[0061] In this embodiment, the feature extraction layer uses the first three residual convolutional modules of ResNet50 to extract the 256×256×3 image into a 512×32 image. 2 2D features; secondly, a convolution kernel of 1 is used. 2 Channel extraction is performed using 2D convolution to generate 128×32 2 The 2D feature map is then extracted; then four convolutional networks are used to extract the features into a 256×16 pixel format. 2 512×8 2 1024×4 2 1024×2 2 Feature maps of various sizes, each convolutional network uses 3 convolutional kernels. 2 A 2D convolutional layer with padding of 1 is followed by batch normalization, ReLU activation, and a MaxPooling layer with a downsampling factor of 2. This extracts a 256×256×3 2D image into a 128×32 image. 2 256×16 2 512×8 2 1024×4 2 1024×2 2 Size feature map.

[0062] Step 3: Use a depth estimation network to fuse multiple two-dimensional features to generate three-dimensional features.

[0063] By using the 2D features extracted in step 2 as the front-layer features, using Sigmoid to generate 2D feature space probabilities, then using a residual network to perform depth perception on the front-layer features, multiplying the depth-perceived features with the feature space probabilities to obtain the back-layer features, and finally concatenating the front and back-layer features to generate the final 3D features.

[0064] The structure diagram of the depth estimation module in step 3 is as follows: Figure 4 As shown, the specific implementation is as follows:

[0065] The calculation process for depth estimation is shown in formulas (1) and (2).

[0066] F′ i =D(F i )×S(F i (i=1,2,...m) (1)

[0067] Where m is the total number of channels, F′ i Let D(F) represent the two-dimensional feature of the i-th subsequent layer. i S(F) represents the i-th depth-sensing 2D feature. i ) represents the two-dimensional spatial probability of the i-th feature.

[0068] M i =F i +F′ i (i=1,2,...m) (2)

[0069] Where F′ i Let F represent the two-dimensional feature of the i-th depth estimate. i M represents the i-th fused two-dimensional feature. i Let represent the three-dimensional spatial probability of the i-th feature.

[0070] Step 4: The decoding module uses the 3D basic unit group generated in Step 3 to generate coarse 3D voxels through a 3D decoding network. The specific structure is as follows: Figure 3 As shown, specifically, this is achieved through the operation of a 3D feature upscaling network and a 3D feature perception network:

[0071] (1) 3D Feature Upgrading Network

[0072] The 3D feature generation network consists of four upsampling networks. The upsampling networks first use deconvolutional networks to upsize the 3D features, and then use reshape to sequentially transform the depth-estimated 3D features into 4D features. 3 8 3 16 3 32 3 The 3D features generated by the encoding network are then fused with the upsampling features using a 3D convolutional network. To achieve a 643-voxel feature size, five upsampling networks are needed to upsample the 3D features from 23 to 4. 3 8 3 16 3 32 3 64 3 .

[0073] (2) 3D Feature Perception Network

[0074] After 3D feature upscaling, a perceptual network is first used to adjust the generated 3D features to match those generated in the frontal view. This perceptual network uses 1×1 3D convolutions, followed by batch normalization and Leaky ReLU activation, performing three depth sensing passes to generate 8×32 features. 3 8×32 3 8×32 3The 3D features are then fused together in each channel to generate a 32×32. 3 The features are then processed using two 3D convolutional networks to vary the feature channels to 8 and 1, selecting the optimal channel feature. The first convolutional network consists of a 3-kernel network. 3 The first convolutional network consists of 3D convolutions with padding of 1, 3D batch normalization, and ReLU activation; the second convolutional network consists of convolutional kernels with 3... 3 It consists of 3D convolutions with padding of 1, 3D batch normalization, and a Sigmoid activation function.

[0075] Step 5: Utilize the context feature awareness module, the structure of which is as follows: Figure 5 As shown, multi-view fusion is performed to combine the reconstruction results of 2D views from different angles, enriching the final 3D voxel features and thus generating an accurate reconstruction effect. Specifically:

[0076] (1) The rough 3D voxels generated for each view are perceptualized through 4 multi-scale feature blocks;

[0077] (2) Then add the features of the four feature perception blocks together along the channel dimension;

[0078] (3) Then use a three-dimensional convolutional network to extract the perceptual features of each view, and use SoftMax to normalize the probability of the extracted features;

[0079] (4) The rough voxel generated for each view is multiplied by the context-aware probability of that view to generate the context-aware spatial probability of each view.

[0080] (5) The context-aware spatial probabilities of all views are summed to obtain the 3D voxel features of multi-view fusion.

[0081] Step 6: Repeat steps 2 to 5 on the training set until the entire model converges. Specifically, use the training set data to train the 3D reconstruction model of the multi-view target. The model training parameters are shown in Table 3.

[0082] Using Focal Loss as the loss function during training can solve the problem of unbalanced target voxel proportions. Focal Loss improves the cross-entropy function by adding a class weight and sample difficulty weight adjustment factor (1-pt)γ to the cross-entropy loss. The Focal Loss function is shown in Equation (3):

[0083] FL(p t )=-α t (1-p t ) γ log(p t(3)

[0084] Where y = 1 indicates that the voxel mesh is occupied, and the weight of αt is α; y = 0 indicates that the voxel mesh is not occupied, and αt is αt. t The weight is 1-α, through α t This can better optimize the problem of voxel imbalance. Then there's the sample difficulty weight adjustment factor p. t When y = 1, p t The weight is set to p. When y = 0, p is set to p. t The weights are set to 1-p, and finally passed through (1-p) t The combination factor γ regulates the grid cells in a voxel that are difficult to distinguish as occupied or not. Both α and γ are gradually adjusted based on prior knowledge and experimental results to ultimately set parameter values ​​suitable for the model.

[0085] Based on the above conditions of y being 1 and 0, we can apply the following to αt and p. t We can obtain the formula for 3DFocal Loss under the weight setting, as shown in formula (4).

[0086]

[0087] Step 7: Save the training parameters with the best accuracy for validation on the test set. The model training parameters are shown in Table 1.

[0088] Table 1 Model Training Parameters

[0089]

[0090]

[0091] During training, the weights of models with higher average accuracy are saved. After training, the trained model parameters are saved. When using the test set for 3D reconstruction, the trained model is directly used to generate 3D voxels, obtaining the average accuracy of each model class and the average accuracy of the entire test set. The reconstruction results are as follows. Figure 6 As shown in Table 2, the accuracy rates after reconstruction are as follows.

[0092] Table 2 compares the average IoU of the model with that of 3D-R2N2 and Pix2Vox-A.

[0093]

[0094] It can be seen that the deep learning model of this invention can reconstruct the approximate outline of the target quite well. Testing on all datasets yielded a final average accuracy of 0.667, which was compared with 3D-R2N2 and pix2Vox-A. The final results are shown in Table 2. As can be seen from Table 2, the model in this paper has the highest average accuracy, 0.6% higher than Pix2Vox-A.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method using a multi-view P2VNet network, characterized in that: The steps are as follows: Step 1: Building the dataset; Two-dimensional image data of multiple targets are collected. For the same target, the three-dimensional voxel features of the target are calculated using CAD software. All two-dimensional images and corresponding three-dimensional voxel features are divided into training set and test set in a 3:1 ratio and transformed into input data required by the deep model. Step 2: Extract features from the two-dimensional images in the training and test sets using the encoding module to generate 2D feature maps at different scales; Step 3: The depth estimation module fuses multiple two-dimensional features to generate three-dimensional features; By using the 2D features extracted in Step 2 as the front-layer features, using Sigmoid to generate 2D feature space probabilities, then using a residual network to perform depth perception on the front-layer features, then multiplying the depth-perceived features with the feature space probabilities to obtain the back-layer features, and finally concatenating the front and back-layer features to generate the final 3D features. Step 4: The decoding module generates coarse 3D voxels from the 3D features generated by the depth estimation module through a 3D decoding network; Step 5: Use the multi-scale context feature perception module to perform multi-view fusion, which merges the 2D view reconstruction results from different angles to enrich the final 3D voxel features and generate accurate reconstruction results. Step 6: Repeat steps 2 through 5 on the training set until the entire model converges; Step 7: Save the multi-view 3D reconstruction model obtained after training in Step 6, and perform 3D reconstruction tests on the 2D images in the test set to generate the corresponding 3D voxel model. The specific method for Step 2 is as follows: The encoding module extracts features from the 2D images in the training and test sets from step 1. 2D feature extraction uses the convolutional module of the ResNet50 network to extract features from a 256×256×3 image, resulting in a 128×32 image. 2 256×16 2 512×8 2 1024×4 2 1024×2 2 Size feature map; The specific method for Step 3 is as follows: The calculation process for depth estimation is shown in formulas (1) and (2). （1） Where m is the total number of channels. Indicates the first A fusion of two-dimensional features, , Indicates the first A depth-sensing two-dimensional feature Indicates the first The two-dimensional spatial probability of each feature; （2） in Indicates the first A fused 3D feature; The specific method for step 4 is as follows: The decoding module uses the 3D features generated in Step 3 to generate coarse 3D voxels through a 3D feature upscaling network and a 3D feature perception network. (1) 3D feature upscaling network; The 3D feature upsampling network consists of four upsampling networks. The upsampling block is performed by a 3D deconvolutional network to upsize 3D features. Then, Reshape transforms the 3D features generated by depth estimation into 3D features. Finally, a 3D convolutional network is used to fuse the 2D features generated by the encoding network with the upsized features. (2) 3D feature perception network; After 3D feature upscaling, a perceptual network is first used to adjust the generated 3D features to match those generated in the frontal view. This perceptual network uses 1×1 3D convolutions and then performs batch normalization and Leaky ReLU activation to generate 3D features through three depth perception passes. The features from each pass are then fused along the channel boundaries. Finally, two 3D convolutional networks are used on the fused features to change the feature channels to 8 and 1, selecting the optimal channel feature. The first convolutional network uses 3×1 kernels... 3 The first convolutional network consists of 3D convolutions with padding of 1, 3D batch normalization, and ReLU activation; the second convolutional network consists of convolutional kernels with 3... 3 It consists of 3D convolutions with padding of 1, 3D batch normalization, and a Sigmoid activation function.

2. The three-dimensional reconstruction method of a multi-view P2VNet network according to claim 1, characterized in that: The specific method for step 5 is as follows: (1) The rough 3D voxels generated for each view are processed through 4 multi-scale feature perceptrons; (2) Then add the features of the four feature perceptrons together along the channel dimension; (3) Then use a three-dimensional convolutional network to extract the perceptual features of each view, and use SoftMax to normalize the probability of the extracted features; (4) The rough voxel generated for each view is multiplied by the context-aware probability of that view to generate the context-aware spatial probability of each view; (5) The context-aware spatial probabilities of all views are summed to obtain the 3D voxel features of multi-view fusion.

3. A three-dimensional reconstruction method for a multi-view P2VNet network according to claim 1 or 2, characterized in that: The specific method for step 6 is as follows: A 3D reconstruction model of multi-view objects is trained using training set data. Focal Loss is used as the loss function during training. Focal Loss improves the cross-entropy function by adding class weights to the cross-entropy loss. and sample difficulty weight adjustment factor The Focal Loss function is shown in equation (3): (3) Where y=1 indicates that the voxel mesh is occupied, at this time The value of y is α; y=0 indicates that the voxel mesh is not occupied. The value is 1−α, through Optimize the problem of unbalanced voxel proportions; adjust the sample difficulty weight factor. When y=1, put The value of is set to p. When y=0, The value of is set to 1-p, and finally through The combination factor adjusts the grids in the voxel that are difficult to distinguish as occupied or not. α and γ are gradually adjusted based on prior knowledge and experimental results, and finally set parameter values ​​suitable for the model. Based on the above analysis of the cases where y is 1 and 0... as well as The formula for 3DFocal Loss under specific value settings is shown in formula (4). (4)。 4. The three-dimensional reconstruction method of a multi-view P2VNet network according to claim 3, characterized in that: The specific steps of step 7 are as follows: During training, the weights of the model with the highest average accuracy are saved. After training is completed, the trained model parameters are saved. When using the test set for 3D reconstruction, the trained model is used directly to generate 3D voxels, and the average accuracy of each type of model and the average accuracy of the entire test set are obtained.

Images