A multi-view three-dimensional reconstruction method and apparatus

Abstract

The application discloses a multi-view three-dimensional reconstruction method and device, and the method comprises the following steps: extracting picture features of three scales of an input image by using a two-dimensional UNet network; optimizing a feature sequence by using a self-attention mechanism; performing two-dimensional to three-dimensional conversion by using homography transformation, and constructing an initial cost volume; inputting the initial cost volume into a three-dimensional UNet network to perform softmax and soft-argmin operations to obtain an initial depth map D k，i Wherein k represents three stages, and the depth map generated in the previous stage is taken as an additional input of homography transformation in the next stage; the depth information entropy of the initial probability volume is taken as a weight to fuse the initial cost volume; and the fused cost volume is subjected to the softmax and soft-argmin operations of the three-dimensional UNet network to generate a depth map D0. The application improves the integrity and accuracy of multi-view three-dimensional reconstruction of an object.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a multi-view... Figure 3 Reconstruction methods and equipment. Background Technology

[0002] 3D reconstruction, as a key technology for environmental perception, can be used in autonomous driving, virtual reality, moving target detection, behavior analysis, security monitoring, and more. From a computer vision perspective, 3D reconstruction aims to estimate dense representations from overlapping images based on a given image dataset, thereby recovering the geometric model of the corresponding object. Traditional 3D reconstruction methods use handcrafted similarity metrics and engineered regularizations (such as normalized cross-correlation and semi-global matching) to compute dense correspondences and recover 3D point clouds. While these methods achieve good results under ideal conditions, they still face some limitations. For example, low-texture, occlusion, specular highlights, and interacting regions in a scene make dense matching difficult to handle, leading to a decrease in the integrity and robustness of the reconstruction (see the literature "Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2018."). In recent years, the rapid development of deep learning technology and computer hardware has greatly promoted academic research on 3D reconstruction, resulting in encouraging achievements. To date, methods based on deep convolutional neural networks (CNNs) have achieved state-of-the-art accuracy in 3D reconstruction. CNNs employ an encoder-decoder structure, where the encoder extracts features from image data and fuses them using an attention mechanism, before the decoder performs deconvolution to generate a predicted 3D model. However, these techniques primarily rely on extrinsic information about features, failing to explore the relationships between features, especially the internal correlations between data or features. Current methods employ different convolutional layers to acquire features, but the representation of contextual information in these features is not sufficiently explicit. 3D reconstruction involves source and reference images, which vary in size, lighting, and field of view. Convolutional feature extraction methods are affected by the receptive field of the convolutional kernel, resulting in feature maps with significant differences, leading to inconsistencies in stereo matching and impacting reconstruction accuracy. Summary of the Invention

[0003] Purpose of the invention: To address the problems of existing technologies, this invention proposes a multi-view... Figure 3 To improve reconstruction methods and equipment, and enhance reconstruction accuracy.

[0004] Technical solution: A multi-view Figure 3 The 3D reconstruction method includes the following steps:

[0005] (1) The three scales of image features of the input image are extracted using a two-dimensional UNet network, which are represented as follows: , and ;

[0006] (2) The obtained feature sequences at three scales , and The inputs are fed into the corresponding self-attention layers, self-attention scores are calculated, and new feature sequences are generated based on the self-attention scores. , and ;

[0007] (3) Use homography transformation to transform the feature sequences respectively , and Map the coordinates to the reference view and construct the corresponding initial cost volume;

[0008] (4) Input the initial cost volume into the 3D UNet network and regularize it to obtain the cost volume. Then, a softmax operation is performed to obtain the probability volume. The initial depth map is obtained by performing a soft-argmin operation on the probability volume. ,in This represents three stages, with the depth map generated in the first stage. As an additional input to the homography transformation in the second stage, the depth map generated in the second stage As an additional input to the homography transformation in the third stage, i represents the i-th view;

[0009] (5) Calculate the depth information entropy of the initial probability volume as the uncertainty of the pixel at different depths. Uncertainty is used as a weight to fuse the initial cost body;

[0010] (6) The fused cost volume is processed by the 3D UNet network to perform softmax and soft-argmin operations to generate a depth map. .

[0011] Furthermore, feature sequences , and The scales of the representation are equivalent to 1 / 8, 1 / 4, and 1 / 2 of the scale of the original image, respectively.

[0012] Further, step (2) includes:

[0013] by express , and Any one of them, first The input is fed into three different linear layers, resulting in three different sets of feature sequences. , , As shown in formula (1):

[0014] (1)

[0015] in, , and This represents the weight factor of the linear layer, which is obtained by training the network multiple times to obtain the optimal value.

[0016] Then, calculate the self-attention score according to formula (2):

[0017] (2)

[0018] in, The feature dimension is represented by the self-attention score, which represents the correlation between features.

[0019] Finally, based on the self-attention score, a new feature sequence is generated, as shown in formula (3):

[0020] (3)

[0021] Furthermore, the homography transformation in step (3) is as follows:

[0022]

[0023] in This represents the homography between the feature map of the i-th view and the reference feature map at the assumed depth d. , , Let these represent the camera intrinsics, rotation, and translation of the i-th view, respectively. Indicates the principal axis parameters of the reference camera. , , These represent the reference camera intrinsic parameters, rotation, and translation, respectively. It is an identity matrix, and the superscript T indicates transpose.

[0024] Furthermore, the uncertainty in step (5) The mathematical expression is as follows:

[0025]

[0026] Represents pixels Uncertainty at different depth values, For mapping functions, This represents the information entropy of pixel s. Represents pixels At the corresponding depth The probability at that location. For depth range.

[0027] The cost body fusion formula is as follows:

[0028]

[0029] in, .

[0030] The present invention also provides a computer device, comprising:

[0031] One or more processors;

[0032] Memory; and

[0033] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the programs, when executed by the processors, implement the multi-view as described above. Figure 3 The steps of the dimensional reconstruction method.

[0034] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-view as described above. Figure 3 The steps of the dimensional reconstruction method.

[0035] Beneficial Effects: Based on a three-layer network architecture, this invention transforms image feature extraction into a sequence problem by utilizing a self-attention mechanism in the feature extraction section. Based on the self-attention score, it further explores the intrinsic connections between image features, balances the differences between features, and considers image feature extraction from a global perspective, thereby improving multi-view object recognition. Figure 3 The integrity of 3D reconstruction is improved. By exploring the use of attention mechanisms to expand the correlation between feature extractions, global search, and adaptive learning of contextual information help capture rich correlations in the image, obtain the connections between feature maps of each pixel, improve the representation of image features, implicitly incorporate prior information from the scene, and ultimately make the reconstruction more purposeful. In the cost volume construction stage, uncertainty is used as the weight for cost volume fusion, which effectively reduces the impact of the "occlusion" problem during matching, improves the accuracy of 3D reconstruction, and further enhances the model's reconstruction effect. Attached Figure Description

[0036] Figure 1 Multi-view based on self-attention mechanism Figure 3 Reconstructing the network architecture diagram;

[0037] Figure 2 This is a two-dimensional UNetCNN network diagram according to the present invention;

[0038] Figure 3 This is a partial reconstruction effect diagram of the DTU test set according to the present invention. Detailed Implementation

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0040] Figure 1 This is the overall network architecture diagram of the present invention. It can be divided into three identical stages, each consisting of a feature extraction part and a two-step cost volume regularization construction part. Simultaneously, the output of the previous stage imposes stricter constraints on the depth range and depth interval of the next stage, making the extracted features more accurate. First, image data is initialized and processed by a 2D UNetCNN network for feature extraction, outputting image features at three scales. Second, a self-attention mechanism (SA) is introduced, transforming the extraction of image features into a serialization problem, mining the correlation between feature data, optimizing feature representation, and further improving the completeness and accuracy of the extracted image features. Then, homography transformation (H) is used to map the regularized 2D image feature information to 3D space. An initial cost volume is constructed through group correlation and the first 3D UNetCNN network, and a softmax operation is used to transform the initial cost volume into an initial probability volume. The depth entropy of the initial probability volume is used as the weight for the initial cost volume fusion. The weighted fusion cost volume (F) is then regularized by the second 3D UNetCNN network to obtain the final cost volume. By utilizing depth entropy to reflect the uncertainty of the matching process, the adverse effects of the "occlusion problem" on the 3D reconstruction of objects can be effectively suppressed. Finally, the cost volume is subjected to a soft-agrmin operation to regress and obtain the final depth map (i.e., D, DepthRegression, and U, Uncertainty Regression), and the obtained depth is used as reference information input into the next stage. This invention weightedly considers the results of the three stages and uses a loss function related to the network to obtain the prediction result that is closest to the real 3D model.

[0041] The detailed process of feature extraction from training data is as follows: Figure 2As shown. For network integrity, image initialization is also incorporated into the UNet network. Multiple 2D images are input into the network for initialization, and feature extraction is performed through downsampling and upsampling in the 2D UNetCNN network. To ensure the accuracy of the extracted features, the downsampling convolution results and downsampling deconvolution results are concatenated. Then, the CNN network outputs image features at three scales relative to the initial image resolution: 1 / 8, 1 / 4, and 1 / 2. Figure 2 The black layer in the middle is the output layer of the desired result. Considering the relationships between features, a self-attention mechanism is used to calculate the self-attention score, regularize the feature representation, and further improve the completeness and accuracy of the extracted image features.

[0042] The input image feature resolutions for stages one, two, and three, respectively, are 1 / 8, 1 / 4, and 1 / 2 of the initial image resolution. Furthermore, the output of each previous stage imposes stricter constraints on the depth range and depth interval of the next stage, resulting in more accurate feature extraction in the next stage. It can be seen that this multi-layered structure is a coarse-to-fine learning and training network, which improves the network reconstruction effect while reducing the requirements for training time and memory.

[0043] The specific implementation process is as follows:

[0044] 1) Take a reference image and N source images The data is input into the network for initialization, and then the obtained data is input into a two-dimensional UNet network to extract image features at three scales, which are represented as follows: , and Their scales are equivalent to 1 / 8, 1 / 4, and 1 / 2 of the original image scale, respectively.

[0045] 2) The obtained feature sequences at three scales , and The inputs are fed into three self-attention layers respectively. Considering the similarity between the network processes, the following description is provided. The processing procedure and Similarly. First, [the text abruptly ends here, likely due to an incomplete sentence or a formatting error.] The input is fed into three different linear layers, resulting in three different sets of feature sequences. , , As shown in formula (1):

[0046] (1)

[0047] in, , and These represent the weight factors of the three linear layers, which are obtained by training the network multiple times to obtain the optimal values.

[0048] Then, calculate the self-attention score according to formula (2):

[0049] (2)

[0050] in, The feature dimension is represented by the embedding dimension of the self-attention input, and the self-attention score represents the correlation between features.

[0051] Finally, based on the self-attention score, a new feature sequence is generated, as shown in formula (3):

[0052] (3)

[0053] The introduction of the self-attention mechanism explores the potential connections between image features, further mines the data representation of image features, and improves the completeness of object 3D reconstruction.

[0054] 3) Construct a reference-source pair cost volume. The key to mapping two-dimensional image information to three-dimensional space is to use homography transformation, as shown in formula (4):

[0055] (4)

[0056] In this case, assuming the depth d is in a relatively large range, This represents the homography between the feature sequence of the source image of the i-th view and the feature sequence of the reference image at the assumed depth d. , , Let these represent the camera intrinsics, rotation, and translation of the i-th view, respectively. Indicates the principal axis parameters of the reference camera. , , These represent the reference camera intrinsic parameters, rotation, and translation, respectively. It is an identity matrix, and the superscript T indicates transpose.

[0057] In the reference-source cost volume, "reference" refers to the reference image and "source" refers to the source image. Homography transformation is used to map the feature information of N source images to the coordinate system of the reference view. Furthermore, considering the group correlation between the feature information of the reference image and the transformed source image, cost maps are calculated. Then, cost maps based on all depth assumptions are stacked together to form the initial cost volume.

[0058] 4) First Cost Volume Regularization. The initial cost volume is input into the 3D UNetCNN network for regularization to obtain the cost volume. Then, a softmax operation is performed to obtain the probability volume. .in, This represents three stages. This invention derives information entropy from the probability volume and then maps it to uncertainty. Based on the concept of information entropy, the mathematical expression of uncertainty is as follows:

[0059] (5)

[0060] Represents pixels Uncertainty at different depth values, For mapping functions, This represents the information entropy of pixel s. Represents pixels At the corresponding depth The probability at that location. For depth range.

[0061] Finally, a soft-argmin operation is performed on the probabilistic volume to obtain the initial depth map. Let represent the depth map obtained from the source image of the i-th view in the k-th stage. The loss function that constrains the training results of this part is defined as follows:

[0062] (6)

[0063] in, The true value in the training data, i.e., the depth map. The truth value of , This represents the discrete depth sampling interval. This represents the valid pixels in the depth map within the assumed range for depth estimation.

[0064] 5) Cost body fusion. Based on the uncertainty obtained above, the cost body fusion formula (7) is defined as follows:

[0065] (7)

[0066] in, . Let represent the cost volume of the i-th view. In 3D reconstruction, objects may be occluded at different shooting angles. Occlusion in the image will cause uncertainty in the network parameters. This invention introduces uncertainty based on information entropy and uses this uncertainty as a weight for cost volume fusion, reducing the impact of the "occlusion" problem during matching and significantly improving the accuracy of 3D object reconstruction. The loss function that constrains this part of the training results is defined as follows:

[0067] (8)

[0068] It is the kth stage. Uncertainty of the source feature map of a view.

[0069] 6) Second Cost Volume Regularization. Similar to the first cost volume regularization process, the fused cost volume undergoes softmax and soft-argmin operations through a 3D UNetCNN network to generate a depth map. The loss function that constrains the training results for this part is defined as follows:

[0070] (9)

[0071] This represents the ground truth depth value of the reference feature map in the k-th stage. At this point, one stage of the network architecture has ended.

[0072] 7) Depth Range Constraints. This invention uses the depth map generated in the previous stage as an additional input for the homography transformation in the subsequent stage, such as... Figure 1 As shown, the depth map generated in the first stage serves as additional input to the homography transformation part in the second stage, further strictly constraining the assumed range of depth d. Similarly, the depth map generated in the second stage serves as additional input to the homography transformation part in the third stage, further constraining the depth range of the third stage. Furthermore, the image feature resolutions of the main inputs (i.e., the inputs from the self-attention module) in the three stages are 1 / 8, 1 / 4, and 1 / 2 of the original image feature data, respectively. The overall network is a coarse-to-fine construction process. The loss function for training the overall network is defined as follows:

[0073] (10)

[0074] in, Let $\mathbf{k}$ be the proportion of the error in the k-th stage in the overall network architecture learning and training. The training parameters of the overall network architecture are constrained by formula (10). Through iteration with a large amount of data, we strive to obtain a network parameter that can represent the multiple views of the network. Figure 3 The error between the 3D reconstruction result and the true value is minimized.

[0075] This invention uses the DTU dataset for training, and some training results on the DTU test set are as follows: Figure 3 As shown, the accuracy and completeness of (a) to (f) are as follows:

[0076] Test graphics Acc Comp (a) 0.2464 0.2918 (b) 0.3637 0.4337 (c) 0.2433 0.4700 (d) 0.5546 0.5900 (e) 0.2142 0.3255 (f) 0.6000 0.2566

[0077] Where Acc. represents accuracy, indicating the distance between the reconstructed point cloud generated by the present invention and the reference point cloud in the dataset, and Comp. represents completeness, indicating the distance between the reference point cloud and the reconstructed point cloud. Both metrics are in millimeters. The smaller the metric, the higher the accuracy. Figure 3 It can be seen that the reconstruction results obtained by the method of the present invention are highly accurate.

[0078] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A multi-view 3D reconstruction method, characterized in that, Includes the following steps: (1) The three scales of image features of the input image are extracted using a two-dimensional UNet network, which are represented as follows: , and ; (2) The obtained feature sequences at three scales , and The inputs are fed into the corresponding self-attention layers, self-attention scores are calculated, and new feature sequences are generated based on the self-attention scores. , and ; (3) Use homography transformation to transform the feature sequences respectively , and Mapping to the coordinate system of the reference view and constructing the corresponding initial cost volume; the homography transformation is as follows: ;in This represents the homography between the feature map of the i-th view and the reference feature map at the assumed depth d. , , Let these represent the camera intrinsics, rotation, and translation of the i-th view, respectively. Indicates the principal axis parameters of the reference camera. , , These represent the reference camera intrinsic parameters, rotation, and translation, respectively. It is an identity matrix, and the superscript T indicates transpose; (4) Input the initial cost volume into the 3D UNet network and regularize it to obtain the cost volume. Then, a softmax operation is performed to obtain the probability volume. The initial depth map is obtained by performing a soft-argmin operation on the probability volume. ,in This represents three stages, with the depth map generated in the first stage. As an additional input to the homography transformation in the second stage, the depth map generated in the second stage As an additional input to the homography transformation in the third stage, i represents the i-th view; (5) Calculate the depth information entropy of the initial probability volume as the uncertainty of the pixel at different depths, and use the uncertainty as a weight to fuse the initial cost volume; the mathematical expression of the uncertainty is as follows: , Represents pixels Uncertainty at different depth values, For mapping functions, This represents the information entropy of pixel s. Represents pixels At the corresponding depth The probability at that location. For depth range; the cost volume fusion formula is as follows: ,in, ; (6) The fused cost volume is processed by the 3D UNet network to perform softmax and soft-argmin operations to generate a depth map. .

2. The multi-view three-dimensional reconstruction method according to claim 1, characterized in that, Feature sequence , and The scales of the representation are equivalent to 1 / 8, 1 / 4, and 1 / 2 of the scale of the original image, respectively.

3. The multi-view three-dimensional reconstruction method according to claim 1, characterized in that, Step (2) includes: by express , and Any one of them, first The input is fed into three different linear layers, resulting in three different sets of feature sequences. , , As shown in formula (1): (1) in, , and This represents the weight factor of the linear layer, which is obtained by training the network multiple times to obtain the optimal value. Then, calculate the self-attention score according to formula (2): (2) in, The feature dimension is represented by the self-attention score, which represents the correlation between features. Finally, based on the self-attention score, a new feature sequence is generated, as shown in formula (3): (3)。 4. The multi-view three-dimensional reconstruction method according to claim 1, characterized in that, The loss function for training the entire network is defined as follows: ； in, Let be the proportion of the error in the k-th stage in the overall network architecture learning and training, and we have: ； ； ； in, This represents the depth map obtained from the source map of the i-th view in the k-th stage. For depth map The true value, It is the kth stage. Uncertainty of the source feature map of each view, This represents the ground truth value of the depth of the reference feature map in the k-th stage. Represents the discrete depth sampling interval. This represents the valid pixels in the depth map within the assumed range for depth estimation.

5. A computer device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the multi-view three-dimensional reconstruction method as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-view three-dimensional reconstruction method as described in any one of claims 1-4.

Images