A multi-view 3D reconstruction method based on an epipolar transformer and implicit neural optimization enhancement MVSNet network

Abstract

This invention discloses a multi-view 3D reconstruction method based on an epipolar transformer and implicit neural optimization enhancement MVSNet network, belonging to the field of multi-view 3D reconstruction technology in computer vision. The basic process of its network model is: coarse depth estimation based on an epipolar transformer-enhanced MVS module, depth super-resolution based on an implicit neural optimization module, and depth refinement based on an enhanced Gaussian-Newton module. The proposed method addresses various problems caused by the imbalance between reconstruction efficiency and reconstruction quality in existing technologies. It significantly improves the quality of predicted depth maps and reconstructed point clouds, while requiring less memory and exhibiting faster inference speed compared to other learning-based progressive refinement MVS methods. Therefore, it provides an effective technical solution for high-precision, high-completeness, and high-efficiency 3D reconstruction.

Description

Technical Field

[0001] This invention relates to multiview in computer vision. Figure 3 In the field of 3D reconstruction technology, and more specifically, in the context of a multi-view MVSNet network based on an epipolar transformer and implicit neural optimization enhancement. Figure 3 Dimensional reconstruction method. Background Technology

[0002] As a fundamental task in computer vision, Multi-View Stereo (MVS) has been extensively studied for decades. Its core idea is to establish pixel-level correspondences between different views. MVS methods aim to reconstruct the dense 3D geometry of a target scene from multiple 2D RGB images with known camera parameters. Because humans naturally prefer to perceive the 3D world, MVS methods play a crucial role in many fields such as national defense, disaster relief, agriculture, urban construction, autonomous navigation, archaeological research, art design, and virtual / augmented reality. Based on different scene representations, MVS methods can be categorized into voxel-based, point spread-based, patch-based, and depth map-based methods. Depth map-based methods are the most lightweight and accurate, applicable to a wide range of scene scales, with leading methods such as COLMAP, Gipuma, and ACMM emerging.

[0003] However, traditional MVS methods use hand-designed similarity metrics, which struggle to adequately address challenging issues in the target scene, such as lighting variations, weakly textured areas, reflective surfaces, and repetitive patterns, easily leading to point cloud voids and blurriness. Furthermore, the engineered regularization methods employed in traditional MVS cannot eliminate the negative impact of invisible pixels, including those located in occluded areas between views and outside the field of view, resulting in a degraded point cloud quality in these invisible regions.

[0004] In the era of deep learning, Yao et al., leveraging the powerful semantic information enrichment and noise filtering capabilities of Convolutional Neural Networks (CNNs), pioneered the end-to-end trainable network MVSNet for the MVS task in 2018. MVSNet uses 2D CNNs to extract image features as a similarity metric and feeds the matching cost volume into a 3D CNN for regularization, significantly mitigating the adverse effects of challenging scenarios and invisible regions. By utilizing the efficient computational power of the graphics processing unit (GPU) in a computer's graphics card, MVSNet achieves significantly better reconstruction quality than traditional state-of-the-art MVS methods with higher reconstruction efficiency. Therefore, deep learning-based MVS methods have rapidly become the preferred approach for multi-view MVS. Figure 3Research in 3D reconstruction has focused primarily on MVSNet, with the vast majority using it as a practical network baseline to address various issues within the MVSNet pipeline. For instance, to address the heavy memory and computational burden of 3D CNNs, recursive methods such as R-MVSNet, D2HC-RMVSNet, and AA-RMVSNet propose using recursive convolutional units to sequentially regularize 2D slices of the 3D cost volume, sacrificing runtime to overcome the memory bottleneck in high-resolution reconstruction. Conversely, multi-stage cascaded methods such as CasMVSNet, CVP-MVSNet, and UCS-MVSNet, based on residual estimation, infer depth maps in a coarse-to-fine manner. While these cascaded frameworks differ in their principles for determining the overall range and sampling interval of depth assumptions in the refinement stage, they all significantly reduce the memory and time required by learning-based MVS methods.

[0005] However, for high-precision, high-completeness, and high-efficiency 3D reconstruction, the learning-based MVS method still has some problems: (1) It still has significant limitations when reconstructing non-ideal scenes and invisible areas; (2) Some complex network structures introduced to improve prediction accuracy result in long running time and heavy memory burden; (3) Most current methods adopt a multi-stage cascaded framework from the perspective of efficiency, and the error in the first stage can be propagated to the last stage through residual estimation, thereby deteriorating the final prediction result. Summary of the Invention

[0006] In view of this, the present invention proposes a multi-view MVSNet network based on an epipolar transformer and implicit neural optimization enhancement. Figure 3 The proposed 3D reconstruction method employs a network-based approach: coarse depth estimation using an epipolar transformer-enhanced MVS module, depth super-resolution using an implicit neural optimization module, and depth refinement using an enhanced Gaussian-Newton module. This method addresses various issues arising from the imbalance between reconstruction efficiency and quality in existing technologies, thus providing an effective technical solution for high-precision, high-completeness, and high-efficiency 3D reconstruction.

[0007] A multi-view MVSNet network based on epipolar transformer and implicit neural optimization enhancement Figure 3 The 3D reconstruction method includes the following steps:

[0008] Step S1: Obtain a multi-view image set of the target scene. Each time, select one image as a reference image and select (N-1) source images with the smallest image angle. Use these N images as input. Input the full-resolution reference image and source images into a scene-aware feature extractor with an embedded view epipolar transformer to obtain a quarter-resolution feature map of the reference image and source images, i.e., the reference feature map and the source feature map.

[0009] Step S2: Through differentiable homography transformation, map N-1 source feature maps onto the forward parallel plane in the reference camera's view frustum, divided by several depth assumptions; calculate the inter-group correlation similarity between the reference feature map and the twisted source feature map, forming N-1 similarity volumes; through a view aggregation module embedded with an inter-view epipolar transformer, aggregate the similarity volumes into a unified matching cost volume; use a 3D CNN to perform a regularization operation on the cost volume to obtain a probability volume along the depth direction; then, through depth regression, calculate a quarter-resolution depth map from the probability volume;

[0010] Step S3: Construct an implicit neural optimization module to implement the depth map super-resolution process. Take the quarter-resolution pixel coordinates, half-resolution pixel coordinates, quarter-resolution depth map, and full-resolution reference image as inputs, and output the optimized half-resolution depth map.

[0011] Step S4: Replace the feature pyramid used in the traditional Gaussian-Newton module with the context-aware feature extractor described in step S1 to construct an enhanced Gaussian-Newton module; use this module to further refine the half-resolution depth map to obtain the final depth map, which is still half-resolution in size;

[0012] Step S5: Replace the reference image and repeat the above steps until the depth maps corresponding to all images of the target scene are obtained.

[0013] Furthermore, the context-aware feature extractor described in step S1 is constructed by embedding a view-inside-outside epipolar transformer within a feature pyramid-based recurrent feature extractor: the feature pyramid-based recurrent feature extractor has a top-down-bottom-up recurrent structure, which can extract context-aware features that simultaneously contain low-level spatial information and high-level semantic information; the view-inside-outside epipolar transformer has a multi-head self-attention layer and a multi-head cross-attention layer, which can aggregate non-local features along the epipolar line, thereby further enhancing the distinguishability and describability of the features.

[0014] Furthermore, the view aggregation module embedded with an inter-view epipolar transformer, described in step S2, aggregates the similarity volumes into a unified matching cost volume. The process is as follows:

[0015] First, using an epipolar transformer between views, and based on the cross-attention concept of the transformer, long-distance 3D associations along the depth direction are constructed between matching pixels on the epipolar lines in different views, and the weight w of each similarity volume is calculated. i :

[0016]

[0017] In the formula, F0 and W i Let W0 represent the reference feature and the source feature body after being twisted according to the differentiable homography transformation, respectively. i = 1, ..., N-1 represent the corresponding source view number, and Ch represents the number of feature channels. In the cross-attention mechanism, W0 acts as the query vector, and W... i The role is the key vector, and w is calculated from both as described above. i This can be used for attention-weighted value vectors; the unified matching cost C is calculated as follows:

[0018]

[0019] In the formula S i For similarity volume, its role is a value vector.

[0020] Furthermore, the implicit neural optimization process described in step S3 is as follows:

[0021] Implicit neural function f implemented by a multilayer perceptron θ It is possible to jointly estimate the interpolation depth value v q,p and interpolation weight w q,p :

[0022] w q,p ,v q,p =f θ (d p ,g p ,g q -g p ,x q -x p )

[0023] In the formula, x p and x q G represents the pixel coordinates at quarter and half resolution, respectively. p and g q These represent quarter-resolution and half-resolution reference feature maps extracted from the full-resolution reference image using the feature pyramid, respectively. p This indicates the use of a feature pyramid to obtain a coarse depth map D at quarter resolution. 1 Extracted quarter-resolution depth feature map; finally, optimized half-resolution depth map D. 2 The following calculation is performed using interpolation:

[0024]

[0025] Where D 2 (x q ) represents D 2 Mid-coordinate x q At depth value, This represents the set of neighboring pixels of pixel q in the quarter-resolution image domain.

[0026] At this point, by filtering and fusing the final depth map of all reference images of the target scene or object, a dense point cloud can be obtained, completing the multi-view process. Figure 3 Reconstruction.

[0027] The beneficial effects of this invention are: Based on the various problems faced by existing deep learning-based multi-view stereo vision 3D reconstruction technologies, this invention proposes a multi-view stereo vision 3D reconstruction method based on an epipolar transformer and an implicit neural optimization-enhanced MVSNet network. Figure 3 This invention proposes a depth reconstruction method. The network model of this method features a progressively refined architecture. The basic process involves coarse depth estimation based on an epipolar transformer-enhanced MVS module, depth super-resolution based on an implicit neural optimization module, and depth refinement based on an enhanced Gaussian-Newton module, thus avoiding the error propagation inherent in cascaded architectures. First, this invention constructs a scene-aware feature extractor with embedded view epipolar transformers, whose extracted more expressive features enable the network to infer more accurate depth values ​​in challenging regions. Second, this invention constructs a view aggregation module with embedded inter-view epipolar transformers, which utilizes cross-attention reasoning for view weights to improve point cloud reconstruction in invisible areas. The MVS module enhanced by these two epipolar transformers effectively improves the quality of depth map and point cloud reconstruction. Then, this invention constructs an implicit neural optimization module, using implicit neural functions to achieve coarse depth map super-resolution, effectively improving the accuracy of foreground object edge regions in the depth map and point cloud. Finally, this invention constructs an enhanced Gaussian-Newton module to further refine the depth map as a whole, thereby comprehensively improving the reconstruction quality. The MVSNet network, trained and tested on the DTU dataset, experimentally validates the multi-view capability of the proposed method based on epipolar transformers and implicit neural optimization enhancements. Figure 3 The dimensional reconstruction method can significantly improve the quality of predicted depth maps and reconstructed point clouds, while consuming less memory and having a faster inference speed compared to other progressive refinement methods. Attached Figure Description

[0028] Figure 1 For multi-view MVSNet networks optimized and enhanced with epipolar transformers and implicit neural networks Figure 3 Flowchart of 3D reconstruction method;

[0029] Figure 2 Schematic diagram of the inner-outer polarimeter converter and the outer polarimeter converter between views;

[0030] Figure 3 For multi-view MVSNet networks optimized and enhanced with epipolar transformers and implicit neural networks Figure 3 A visualization of the point cloud reconstruction results of the entire scene in the DTU dataset evaluation set using the 3D reconstruction method;

[0031] Figure 4 A comparison chart of point cloud visualization results for reconstructing challenging scenes from the DTU dataset using different methods;

[0032] Figure 5 A comparison chart of GPU memory consumption, runtime, and overall reconstruction quality for different methods;

[0033] Figure 6 Ablation experiments were performed on two types of epipolar transducers, an implicit neural optimization module, and an enhanced Gauss-Newton module. Detailed Implementation

[0034] To make the intended purpose, technical means, and effects of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0035] It should be noted that, unless otherwise stated, all technical and scientific terms used in this invention have the same meaning as those in the technical field to which this invention pertains. Methods in the following embodiments that do not specifically specify experimental conditions employ conventional techniques within that technical field. It should be understood that the described embodiments are only a part of the examples of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on these embodiments of this invention without creative effort are within the protection scope of this invention.

[0036] Example 1

[0037] To address the various problems caused by the imbalance between reconstruction efficiency and reconstruction quality in existing technologies, and to provide an effective technical solution for high-precision, high-completeness, and high-efficiency 3D reconstruction, this invention proposes a multi-view MVSNet network based on epipolar transformers and implicit neural optimization enhancement. Figure 3 A depth reconstruction method. The basic network process includes coarse depth estimation based on an epipolar transformer-enhanced MVS module, depth super-resolution based on an implicit neural optimization module, and depth refinement based on an enhanced Gaussian-Newton module, such as... Figure 1 As shown. This multi-view MVSNet network, based on an epipolar transformer and implicit neural optimization enhancement, is shown. Figure 3 The 3D reconstruction method specifically includes the following steps:

[0038] Step S1: Obtain a multi-view image set of the target scene. Each time, select one image as a reference image and select (N-1) source images with the smallest image angle. Use these N images as input. Input the full-resolution reference image and source images into a scene-aware feature extractor with an embedded view epipolar transformer to obtain quarter-resolution feature maps of the reference image and source images, i.e., the reference feature map and the source feature map.

[0039] Step S1.1: Obtain a multi-view image set of the target scene. Each time, select one image as a reference image and select (N-1) source images with the smallest angle between the images. Use these N images as input.

[0040] For example, by selecting the DTU public dataset, multi-view images of a certain scene can be selected from the corresponding training and evaluation sets for different processes of network training and evaluation. Based on the image angle score defined in its pair.txt, (N-1) source images with the best matching degree can be selected for each reference image. In this embodiment, the value is N = 5.

[0041] Step S1.2: Input the full-resolution reference image and the source image into a scene-aware feature extractor embedded with an epipolar transformer to obtain quarter-resolution feature maps of the reference image and the source image, i.e., the reference feature map and the source feature map.

[0042] The context-aware feature extractor is constructed by embedding a view-inside-outside epipolar transformer within a feature pyramid-based recurrent feature extractor. The feature pyramid-based recurrent feature extractor has a top-down-bottom-up repetitive structure, capable of extracting context-aware features that simultaneously contain low-level spatial information and high-level semantic information. The view-inside-outside epipolar transformer has multi-head self-attention layers and multi-head cross-attention layers, capable of aggregating non-local features along the epipolar line, such as... Figure 2 As shown, this further enhances the distinguishability and describability of the features. This scene-aware feature extractor extracts features from a reference image I0 of size W×H and the source image... In this process, reference features and source features of size (W / 4)×(H / 4) are extracted. In this embodiment, the values ​​are: for the DTU training set, W×H = 640×512; for the DTU evaluation set, W×H = 1600×1152.

[0043] Step S2: Through differentiable homography transformation, map N-1 source feature maps onto the forward parallel plane in the reference camera's view frustum, which is divided by several depth assumptions; calculate the inter-group correlation similarity between the reference feature map and the twisted source feature map to form N-1 similarity volumes; through a view aggregation module with an embedded inter-view epipolar transformer, aggregate the similarity volumes into a unified matching cost volume; use a 3D CNN to perform a regularization operation on the cost volume to obtain a probability volume along the depth direction; and then use depth regression to calculate a quarter-resolution depth map from the probability volume.

[0044] Step S2.1: Through differentiable homography transformation, map N feature maps onto the forward parallel plane in the view frustum of the reference camera, which is divided by several depth assumptions.

[0045] The forward parallel plane, defined by several depth assumptions, i.e., the depth assumption plane, is defined by a predefined depth range (d). min ,d max Uniform sampling depth assumption d j get:

[0046] d j =d min +j / D(d max -d min )

[0047] In the formula, j = 0, 1, ..., D-1, and D is the total number of depth hypothesis planes. In this embodiment, the value is: for the DTU dataset, (d min ,d max ) = (425mm, 935mm); for training, D = 48; for evaluation, D = 96.

[0048] Differentiable homography transformations essentially describe the underlying pixel-level correspondence between a reference view and a source view. Specifically, given the corresponding camera parameters... This could be pixel p in the reference view, i.e., the depth assumption plane d in the reference camera's view frustum after twisting. j For pixel p, calculate its corresponding source view pixel p. i (d j ):

[0049]

[0050] Then, by using the calculated sampling grid coordinates, differentiable bilinear sampling is performed on the source feature map, for example, using the F.grid_sample function in Python, to complete the deformation operation of the source feature map.

[0051] Step S2.2: Calculate the inter-group correlation similarity between the reference feature map and the twisted source feature map to form N-1 similarity volumes.

[0052] In the field of deep learning-based MVS, feature maps are first transformed into feature volumes or similarity volumes, and then the cost volume is calculated.

[0053] By combining the reference feature map F0 and the torsional source feature map W i (d j The feature channels Ch are uniformly divided into G groups, and the similarity between the g-th groups can be calculated as follows:

[0054]

[0055] In the formula, g = 0, 1, ..., G-1, and <·,·> are inner product operations. By calculating the similarity of G groups, N-1 similarity maps with G channels are obtained; by calculating the similarity maps at all depth assumption planes, N-1 similarity volumes S are obtained. i In this embodiment, the value is G = 4.

[0056] Step S2.3: Through the view aggregation module with an embedded inter-view epipolar transformer, the similarity volumes are aggregated into a unified matching cost volume.

[0057] The similarity volumes are aggregated into a unified matching cost volume using a view aggregation module embedded with an inter-view epipolar transformer, as follows:

[0058] First, such as Figure 2 As shown, by using an epipolar transformer between views and based on the cross-attention concept of the transformer, a long-distance 3D association relationship along the depth direction is constructed between matching pixels on the epipolar lines in different views, and the weight w of each similarity volume is calculated. i :

[0059]

[0060] In the formula, F0 and W i Let W0 represent the reference feature and the source feature body after being twisted according to the differentiable homography transformation, respectively. i = 1, ..., N-1 represent the corresponding source view number, and Ch represents the number of feature channels. In the cross-attention mechanism, W0 acts as the query vector, and W... i The role is the key vector, and w is calculated from both as described above. i This can be used for attention-weighted value vectors; the unified matching cost C is calculated as follows:

[0061]

[0062] In the formula S i For similarity volume, its role is a value vector.

[0063] Step S2.4: Use 3D CNN to perform regularization on the cost volume to obtain the probability volume along the depth direction.

[0064] The cost volume is fed into a U-Net-based multi-scale 3D CNN, and its powerful noise filtering capability is used to regularize the cost volume. At the end of the 3D CNN, a convolutional unit with one output channel is set, and the output volume undergoes a softmax operation, resulting in the probability volume P along the depth direction.

[0065] Step S2.5: Then, calculate the quarter-resolution depth map from the probability volume through depth regression.

[0066] Corresponding to the sampling of depth assumptions, by calculating the probability weighted sum of all depth assumptions, a coarse depth map at one-quarter resolution can be calculated from the probability volume. The formula is as follows:

[0067]

[0068] In the formula, p represents the corresponding pixel, and D represents the coarse depth map. 1 Resolution and Features Figure 1 Therefore, it is one-quarter the size of the full-resolution input image, that is, its size is (W / 4)×(H / 4).

[0069] Step S3: Construct an implicit neural optimization module to implement the depth map super-resolution process. Take quarter-resolution pixel coordinates, half-resolution pixel coordinates, quarter-resolution depth map and full-resolution reference image as input, and output the optimized half-resolution depth map.

[0070] First, a coarse depth map D at quarter resolution is extracted using a feature pyramid. 1 quarter-resolution features d p Extract quarter-resolution features g from the full-resolution reference image p and its half-resolution feature g q The goal is to... p and g q Under guidance, from d p Start by obtaining the depth value of the half-resolution depth map.

[0071] The depth map is viewed as a two-dimensional image with a single channel. In implicit neural representation, all images can be represented by the same implicit neural function f with parameter θ. θ express:

[0072] s = f θ (c,x)

[0073] In the formula, s is the estimated signal value (depth value or RGB value), c is the latent code (usually a feature vector), and x is the two-dimensional coordinate in the image domain. Based on this, we can extend the definition to the depth value v in the half-resolution depth map, corresponding to the pixel coordinates at a quarter-resolution location. q,p for:

[0074] v q,p =f θ (d p ,g p ,x q -x p )

[0075] In the formula, x p and x q These represent the coordinates of a quarter-resolution and a half-resolution pixel, respectively. Based on the edge weight calculation method in the graph attention mechanism, if the depth value v is used... q,p The depth value at half-resolution pixel coordinates is interpolated, with the following weights:

[0076] w q,p =f η (g p ,g q -g p )

[0077] Since the interpolation depth and interpolation weights have the same representation, an implicit neural function f can be defined. θ To jointly estimate the interpolation depth value v q,p and interpolation weight w q,p :

[0078] w q,p ,v q,p =f θ (d p ,g p ,g q -g p ,x q -x p )

[0079] This implicit neural function can be implemented using a multilayer perceptron (MLP). In this embodiment, to improve efficiency, the number of hidden layers in the MLP is set to [256, 128, 64, 32].

[0080] Finally, the optimized half-resolution depth map D 2 The following calculation is performed using interpolation:

[0081]

[0082] Where D 2 (x q ) represents D 2 Mid-coordinate x q At depth value, This represents the set of neighboring pixels of pixel q in the quarter-resolution image domain.

[0083] Step S4: Replace the feature pyramid used in the traditional Gaussian-Newton module with the context-aware feature extractor described in step S1 to construct an enhanced Gaussian-Newton module; use this module to further refine the half-resolution depth map to obtain the final depth map, which is still half-resolution in size.

[0084] The traditional Gaussian-Newton module can minimize the error function at pixel p through an iterative optimization algorithm:

[0085]

[0086] In the formula, p i′ This represents the reprojected pixel of p. and Let represent the half-resolution reference features and source features extracted from the full-resolution reference image using the feature pyramid, respectively. During this error minimization process, the optimal depth value for pixel p is determined. Clearly, using more expressive features enhances the identification of differences between the source and reference features, thereby improving the accuracy of the final depth value obtained through the Gauss-Newton iterative optimization process.

[0087] Therefore, by using the context-aware feature extractor described in step S1 instead of the feature pyramid used in the traditional Gaussian-Newton module, that is, by using enhanced context-aware features instead of the original ordinary semantic features, an enhanced Gaussian-Newton module can be constructed to optimize the half-resolution depth map D. 2 Further refinement yields the final half-resolution depth map D. 3 .

[0088] Step S5: Replace the reference image and repeat the above steps until the depth maps corresponding to all images of the target scene are obtained.

[0089] By employing a supervised learning strategy, the network model is trained using the L1 loss between the effective values ​​of the predicted depth map and the effective values ​​of the ground truth depth map. Subsequently, the trained network model is used to evaluate each scene, predicting the corresponding depth maps for all its reference images. By filtering and fusing the final depth maps of all reference images of the target scene or object, a dense point cloud can be generated, thus achieving multi-view processing. Figure 3 Multi-view reconstruction. Utilizing a multi-view MVSNet network based on epipolar transformers and implicit neural optimization enhancements. Figure 3The point cloud visualization results of the 3D reconstruction method for the entire scene reconstruction of the DTU dataset evaluation set are as follows: Figure 3 As shown

[0090] As described above, the embodiments of the present invention utilize a multi-view MVSNet network optimized and enhanced by an epipolar transformer and implicit neural network. Figure 3 The method is a 3D reconstruction method that reconstructs the dense 3D geometry of a target scene from multiple 2D RGB images with known camera parameters.

[0091] Example 2

[0092] Experiment setup details

[0093] The DTU dataset is a large-scale indoor multi-view stereo vision dataset acquired under controlled laboratory conditions with fixed camera trajectories. The dataset contains 124 different scenes, each scanned from 49 or 64 views under 7 different lighting conditions. To ensure fair comparison, the same scheme as MVSNet is used to obtain the base ground truth depth maps and corresponding mask images, and the DTU dataset is divided into training, validation, and evaluation sets following previous methods. Official MATLAB evaluation code is provided for calculating quantitative evaluation metrics for point clouds: average accuracy (Accenture) and average integrity (Comp), in mm. Accuracy measures the distance from the reconstructed point cloud to the base ground truth point cloud; integrity measures the distance from the base ground truth point cloud to the reconstructed point cloud. The quality of the reconstructed point cloud is measured by overall quality, also known as total quality or overall reconstruction quality, calculated as the average of accuracy and integrity.

[0094] Overall = (Acc. + Comp.) / 2

[0095] The experimental software environment configuration used in this embodiment of the invention is as follows: the operating system is Ubuntu 20.04; the GPU is NVIDIA GeForce RTX 3090; in the deep learning framework, PyTorch version 1.8.1, CUDA version 11.7, and cuDNN version 8.6.0 are used.

[0096] The specific experimental settings of this invention embodiment are as follows: the total number of training epochs is 16; the batch size is 4; the RMSProp (root mean square propagation) optimizer is used; the initial learning rate is 0.00025, and the learning rate is multiplied by 0.9 to decay every two training epochs.

[0097] Comparison experiment with other network models

[0098] First, the method proposed in this invention is quantitatively compared with the traditional MVS method, the learning-based progressive optimization MVS method, and the learning-based cascaded MVS method. As shown in Table 1, the method proposed in this invention is superior to other learning-based methods in terms of completeness and overall performance, especially in the progressive optimization method with similar architecture.

[0099] Table 1. Results of the reconstruction quality comparison experiment (lower distance index means better reconstruction quality).

[0100]

[0101]

[0102] To verify the robustness of this algorithm, three challenging scenarios from the DTU dataset—Scan11, Scan24, and Scan77—were selected. Qualitative reconstruction comparisons were performed using different methods: R-MVSNet, ACINR-MVSNet, and the proposed method. The results are as follows: Figure 4 As shown. The reconstruction results of R-MVSNet are obtained through optimization of its additionally designed complete point cloud post-processing steps, which are not included in most other methods (including the method proposed in this invention); ACINR-MVSNet is the best recent learning-based progressive refinement MVS method. From Figure 4 As can be seen, the method proposed in this invention, by embedding two epipolar transformers and employing an implicit neural optimization module and an enhanced Gaussian-Newton module to progressively refine the inference depth map, reconstructs a more complete 3D point cloud for the aforementioned scene compared to R-MVSNet and ACINR-MVSNet. This finding is also supported by the corresponding integrity index comparison results in Table 1.

[0103] In practical applications, the efficiency of learning-based MVS methods mainly refers to their efficiency during the testing and evaluation phase. Table 2 compares the GPU memory and runtime required to predict a depth map for different progressive refinement methods. The table also lists quantitative evaluation metrics for the reconstruction quality of each method, as well as network configurations or experimental settings that directly impact efficiency, including input image resolution and the number of depth hypothesis planes.

[0104] Table 2. Results of the efficiency comparison experiment.

[0105]

[0106] Given the high computational costs of MVSNet and Transformer in terms of both time and memory, the method proposed in this invention is carefully designed at each stage of the network to reduce computational requirements. Specifically, by utilizing the most fundamental epipolar geometry principle in the MVS algorithm, two epipolar transformers restrict the aggregation of nonlocal features and the construction of long-range 3D associations to corresponding points on the epipolar lines of the reference and source views, rather than to all pixels in the view, such as... Figure 2 As shown in Table 2, the implicit neural optimization module configures a smaller number of hidden layers to construct a lightweight implicit neural function decoder. Figure 5 As shown, the method proposed in this invention is superior to three representative progressive refinement methods in terms of efficiency, while requiring less memory and significantly faster inferring a single depth map.

[0107] Compared to MVSNet and VA-Point-MVSNet, the proposed method achieves a 51.4% and 84.8% reduction in runtime on the DTU evaluation set, respectively, while also achieving a 33.3% and 14.2% relative performance improvement in overall point cloud reconstruction. These experiments validate the multi-view performance of the MVSNet network based on epipolar transformers and implicit neural optimization enhancement proposed in this invention. Figure 3 The dimensional reconstruction method can significantly improve the quality of reconstructed point clouds, while consuming less memory and having a faster inference speed compared to other progressive refinement methods.

[0108] ablation experiment

[0109] In the ablation experiments, a network model with 3 training views was used for unified evaluation, and the number of evaluation views was 5. To verify the effectiveness of the epipolar transformer-enhanced MVS module, implicit neural optimization module, and enhanced Gaussian-Newton module constructed in this invention, the enhanced Gaussian-Newton module and implicit neural optimization module were gradually removed from the network of the proposed method, and finally, the epipolar transformer-enhanced MVS module was replaced with a regular MVS module for ablation experiments. For fair comparison, the depth maps predicted by different ablation network models were upsampled to the same resolution as the input image before depth fusion. The quantitative comparison results are shown in Table 3. With the addition of the epipolar transformer-enhanced MVS module, implicit neural optimization module, and enhanced Gaussian-Newton module to the method, the reconstructed point cloud showed a significant performance improvement in terms of completeness and overall accuracy.

[0110] Table 3 Ablation Experiment Results

[0111]

[0112] Qualitative comparison results as follows Figure 6As shown, the constructed context-aware feature extractor with an embedded view epipolar transform enables more expressive features, allowing for more accurate depth value inference and facilitating more complete reconstruction, especially in challenging areas such as reflective surfaces. The constructed view aggregation module with an embedded inter-view epipolar transform utilizes cross-attention reasoning for view weights, achieving more accurate depth estimation in invisible areas and improving point cloud reconstruction in corresponding regions. The constructed implicit neural optimization module generates clearer boundaries in the depth map and recovers finer details in the point cloud. The constructed enhanced Gaussian-Newton module further refines the depth map overall, thus comprehensively improving reconstruction quality. These components collectively enhance the overall performance of the proposed method.

[0113] in conclusion

[0114] In summary, this invention proposes a multi-view MVSNet network based on an epipolar transformer and implicit neural optimization enhancement. Figure 3 This invention proposes a depth reconstruction method. The network model of this method features a progressively refined architecture. The basic process involves coarse depth estimation based on an epipolar transformer-enhanced MVS module, depth super-resolution based on an implicit neural optimization module, and depth refinement based on an enhanced Gaussian-Newton module, thus avoiding the inherent error propagation of cascaded architectures. First, this invention constructs a scene-aware feature extractor with embedded view epipolar transformers, whose extracted more expressive features enable the network to infer more accurate depth values ​​in challenging regions. Second, this invention constructs a view aggregation module with embedded inter-view epipolar transformers, which utilizes cross-attention reasoning for view weights to improve point cloud reconstruction in invisible areas. The MVS module enhanced by these two epipolar transformers effectively improves the quality of depth map and point cloud reconstruction. Then, this invention constructs an implicit neural optimization module, using implicit neural functions to achieve coarse depth map super-resolution, effectively improving the accuracy of foreground object edge regions in the depth map and point cloud. Finally, this invention constructs an enhanced Gaussian-Newton module to further refine the depth map as a whole, thereby comprehensively improving the reconstruction quality. The MVSNet network, trained and tested on the DTU dataset, experimentally validates the multi-view capability of the proposed method based on epipolar transformers and implicit neural optimization enhancements. Figure 3 The dimensional reconstruction method can significantly improve the quality of predicted depth maps and reconstructed point clouds, while consuming less memory and having a faster inference speed compared to other progressive refinement methods.

[0115] It will be readily understood by those skilled in the art that, without conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0116] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multi-view 3D reconstruction method based on an epipolar transformer and implicit neural optimization enhancement MVSNet network, characterized in that, Includes the following steps: Step S1: Obtain a multi-view image set of the target scene. Each time, select one image as a reference image and select (N-1) source images with the smallest image angle. Use these N images as input. Input the full-resolution reference image and source images into a scene-aware feature extractor with an embedded view epipolar transformer to obtain a quarter-resolution feature map of the reference image and source images, i.e., the reference feature map and the source feature map. Step S2: Through differentiable homography transformation, map N-1 source feature maps onto the forward parallel plane in the reference camera's view frustum, divided by several depth assumptions; calculate the inter-group correlation similarity between the reference feature map and the twisted source feature map, forming N-1 similarity volumes; through a view aggregation module embedded with an inter-view epipolar transformer, aggregate the similarity volumes into a unified matching cost volume; use a 3D CNN to perform a regularization operation on the cost volume to obtain a probability volume along the depth direction; then, through depth regression, calculate a quarter-resolution depth map from the probability volume; Step S3: Construct an implicit neural optimization module to implement the depth map super-resolution process. Take the quarter-resolution pixel coordinates, half-resolution pixel coordinates, quarter-resolution depth map, and full-resolution reference image as inputs, and output the optimized half-resolution depth map. Step S4: Replace the feature pyramid used in the traditional Gaussian-Newton module with the context-aware feature extractor described in step S1 to construct an enhanced Gaussian-Newton module; use this module to further refine the half-resolution depth map to obtain the final depth map, which is still half-resolution in size; Step S5: Replace the reference image and repeat the above steps until the depth maps corresponding to all images of the target scene are obtained.

2. The multi-view 3D reconstruction method based on an epipolar transformer and implicit neural optimization enhancement MVSNet network according to claim 1, characterized in that: The context-aware feature extractor described in step S1 is constructed by embedding a view-inside-outside epipolar transformer in a feature pyramid-based recurrent feature extractor: the feature pyramid-based recurrent feature extractor has a "top-down-bottom-up" recurrent structure, which can extract context-aware features that simultaneously contain low-level spatial information and high-level semantic information; the view-inside-outside epipolar transformer has a multi-head self-attention layer and a multi-head cross-attention layer, which can aggregate non-local features along the epipolar line, thereby further enhancing the distinguishability and describability of the features.

3. The multi-view 3D reconstruction method based on an epipolar transformer and implicit neural optimization enhancement MVSNet network according to claim 1, characterized in that: The similarity volumes are aggregated into a unified matching cost volume by the view aggregation module embedded with the inter-view epipolar transformer, as described in step S2. The process is as follows: First, using an epipolar transformer between views, and based on the cross-attention concept of the transformer, long-distance 3D associations along the depth direction are constructed between matching pixels on the epipolar lines in different views, and the weight w of each similarity volume is calculated. i : In the formula, F0 and W i Let W0 represent the reference feature and the source feature body after being twisted according to the differentiable homography transformation, respectively. i = 1, ..., N-1 represent the corresponding source view numbers, and Ch represents the number of feature channels. In the cross-attention mechanism, W0 acts as the query vector, and W... i The role is the key vector, and w is calculated from both as described above. i This can be used for attention-weighted value vectors; the unified matching cost C is calculated as follows: In the formula S i For similarity volume, its role is a value vector.

4. The multi-view 3D reconstruction method based on an epipolar transformer and implicit neural optimization enhancement MVSNet network according to claim 1, characterized in that: The implicit neural optimization process described in step S3 is as follows: Implicit neural function f implemented by a multilayer perceptron θ It is possible to jointly estimate the interpolation depth value v q，p and interpolation weight w q，p : w q，p ,v q，p =f θ (d p ,g p ,g q -g p ,x q -x p ) In the formula, x p and x q G represents the pixel coordinates at quarter and half resolution, respectively. p and g q These represent quarter-resolution and half-resolution reference feature maps extracted from the full-resolution reference image using the feature pyramid, respectively. p This indicates the use of a feature pyramid to obtain a coarse depth map D at quarter resolution. 1 Extracted quarter-resolution depth feature map; finally, optimized half-resolution depth map D. 2 The following calculation is performed using interpolation: Where D 2 (x q ) represents D 2 Mid-coordinate x q At depth value, This represents the set of neighboring pixels of pixel q in the quarter-resolution image domain.

Images