A three-dimensional reconstruction method and system based on improved MVSNet

Abstract

The application discloses a kind of multi-view three-dimensional reconstruction method and system based on improved MVSNet, belong to three-dimensional reconstruction technical field.The method includes the following steps: reference image and source image are input into improved MVSNet network, the features of image are extracted by convolution layer and ECA module, and the feature map of reference image and source image is obtained;Feature volume is obtained by differentiable homography transformation, and matching cost volume is aggregated by group similarity measurement module;Matching cost volume is regularized to obtain probability volume by 3D convolution module;Depth map is obtained by depth regression from probability volume;The final depth map is obtained by optimizing depth map through Gauss Newton optimization module;Fusion all obtained depth map and coordinate change obtains dense point cloud and carries out the visualization of three-dimensional model.Compared with traditional MVSNet method, the application sufficiently improves the feature expression ability of image, and the three-dimensional point cloud generated has higher precision and completeness, and has higher robustness to weak texture, repeated texture and other areas.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and in particular relates to a 3D reconstruction method and system based on an improved MVSNet. Background Technology

[0002] 3D reconstruction is the process of converting real-world objects or scenes into 3D models. It is a crucial technology in computer vision and computer graphics, with wide-ranging applications, including virtual reality, augmented reality, game development, and digital artifact preservation. 3D reconstruction typically relies on sensors, image processing algorithms, and computer graphics techniques to acquire and process information about objects or scenes. Some commonly used 3D reconstruction techniques include point cloud reconstruction and multi-view reconstruction. Figure 3 3D reconstruction, structured light reconstruction, RGB-D reconstruction, etc.

[0003] Multi-view stereo (MVS) refers to observing and acquiring images of a scene from multiple perspectives, and then using these images and corresponding camera parameters to reconstruct a 3D representation of the scene, completing stereo matching and depth estimation. In essence, it utilizes image information from multiple perspectives to reconstruct a 3D model, typically including steps such as image feature extraction, camera pose estimation, and 3D point cloud reconstruction. Multi-view stereo reconstruction technology is widely used in photogrammetry, 3D reconstruction, and virtual reality.

[0004] Traditional multi-view stereo reconstruction methods primarily utilize geometric and optical consistency to calculate the matching cost volume before depth estimation. Traditional MVS stereo reconstruction algorithms include Colmap, Furu, and Gipuma, which recover 3D scenes from moving structures. However, they suffer from long runtimes, low accuracy in areas with no texture, weak texture, or repetitive textures, and poor robustness. With the development of deep learning, deep learning-based multi-view stereo reconstruction algorithms can extract image features through convolutional neural networks, learn global semantic information, and fully utilize the contextual information of semantics. This significantly improves reconstruction accuracy and completeness in areas with repetitive textures, weak textures, and non-Lambertian surfaces. Furthermore, models trained with large datasets exhibit good generalization ability and can be applied to various scenarios. Currently, deep learning-based multi-view... Figure 3 Reconstruction has become a research focus for many scholars.

[0005] MVSNet, proposed by Yao et al. in 2018, is the first end-to-end multi-view based deep learning. Figure 3The 3D reconstruction framework takes a reference image and multiple source images as input. After feature extraction, matching cost volume construction, cost volume regularization, and depth regression, a depth map is obtained. However, cost volume regularization consumes too much GPU memory. Therefore, in 2019, Yao et al. proposed R-MVSNet, an improvement on MVSNet, which incorporates a recurrent neural network during cost volume regularization. This reduces memory consumption without compromising accuracy and completeness, but also increases runtime. Point-MVSNet proposes operating directly on the point cloud, first converting a coarsely generated depth map into a point cloud to guide depth optimization. Cascade-MVSNet proposes a cascaded approach, using upsampling of depth maps predicted from low-resolution images to guide the generation of depth maps from high-resolution images. PatchMatchNet introduces the PatchMatch concept to multi-view applications. Figure 3 In 3D reconstruction, the concept of propagation is used to allow each pixel in the image to explore the depth values ​​of the surrounding surfaces of the same object, which improves the accuracy and completeness of the results while reducing the running time. CVP-MVSNet is currently the most accurate deep learning network, but its use of image pyramids to construct the matching cost volume makes the network model complex, computationally expensive, and inefficient in reconstruction. Currently, multi-view reconstruction based on deep learning... Figure 3 The main direction of 3D reconstruction is to reduce the cost of 3D convolution in terms of computation and memory consumption, optimize the network structure, and improve the accuracy and completeness of the reconstructed 3D model under limited resource conditions. Summary of the Invention

[0006] In order to address the problems and shortcomings of existing technologies and improve the accuracy and completeness of depth estimation, this invention proposes a 3D reconstruction method and system based on an improved MVSNet.

[0007] To achieve the objectives of this invention, the technical solution adopted is as follows:

[0008] This invention provides a 3D reconstruction method based on an improved MVSNet, comprising the following steps:

[0009] Step 1: Obtain a set of N multi-view images of the same scene and the same object, and select one as the reference image and (N-1) images as the source images.

[0010] Step 2: Input the reference image and the source image into the improved MVSNet network, extract features from each image, and obtain feature maps of the reference image and the source image;

[0011] Step 3: Perform differentiable homography transformation on the feature map at different depth hypothetical planes to obtain the feature volume; divide the obtained feature volume into g channel groups, calculate the group similarity through the group similarity measurement module, and obtain the matching cost volume based on the group similarity;

[0012] Step 4: Use 3D convolution to perform regularization on the matching cost volume to obtain the probability volume;

[0013] Step 5: Calculate the expected depth value along the depth direction of the probability volume to obtain the initial estimated depth map;

[0014] Step 6: Optimize the initially estimated depth map using the Gauss-Newton optimization method to obtain the optimized depth map;

[0015] Step 7: Fuse the optimized depth map, obtain a dense point cloud through coordinate transformation, visualize the dense point cloud as a 3D model, and obtain a 3D reconstructed model.

[0016] According to the improved MVSNet-based 3D reconstruction method, preferably, the specific operation of feature extraction for each image is as follows: the input source image and reference image are processed through 8 convolutional layers and 2 ECA modules to extract features, thereby obtaining feature maps of the reference image and the source image.

[0017] According to the improved MVSNet-based 3D reconstruction method, preferably, the feature map is subjected to differentiable homography transformation on different depth hypothesis planes to obtain the feature volume. Specifically, the process is as follows: using a plane perpendicular to the principal optical axis of the reference camera as the depth hypothesis plane, the feature map is distorted into different front planes of the reference camera. All pixel coordinates on the source image are mapped to pixel coordinates on the reference image, starting from the minimum depth d. min According to the depth assumption, the interval is continuously mapped to the maximum depth d. max , thus obtaining the feature body.

[0018] According to the improved MVSNet-based 3D reconstruction method, preferably, the distortion process is as shown in equation (1):

[0019]

[0020] In equation (1), K i K ref Camera intrinsics for the source and reference images, respectively, R ref，i Let t be the corresponding rotation matrix. ref，i Let d be the corresponding translation vector. j p represents the corresponding sampling depth value. i，j For I i The coordinates of pixel p in the source image.

[0021] According to the improved MVSNet-based 3D reconstruction method, preferably, the group similarity is obtained by equation (2):

[0022]

[0023] In equation (2), the range of values ​​for g is [0, G-1]; yes The g-th feature body corresponding to the feature body; yes The corresponding feature body of the g-th group; <·> is the inner product operation.

[0024] According to the improved MVSNet-based 3D reconstruction method, preferably, the depth expectation value in step 5 is obtained from equation (3):

[0025]

[0026] In equation (3), P(d) represents the estimated probability of all pixels with a depth assumed to be d, [d min ,d max [ ] represents the range of sampling depth values.

[0027] According to the improved MVSNet-based 3D reconstruction method, preferably, the Gauss-Newton optimization method in step 6 includes: using 2D convolution to extract features from the reference image and the source image to obtain feature map A; the depth value predicted by the initially estimated depth map is denoted as... The depth value of pixel p is denoted as Project point p from the reference image onto p' in each source image. i At this point, calculate p and p'. i The difference in the value of feature map A at the location is minimized, and the objective function is shown in equation (4):

[0028]

[0029] In equation (4), F i F0 is the feature map of the source image; F0 is the feature map of the reference image; p' i Let p be the projection point of pixel p in the i-th source image.

[0030] According to the aforementioned 3D reconstruction method based on improved MVSNet, preferably, p' i From equation (5), we get:

[0031]

[0032] In formula (5) These represent the camera intrinsic parameters, rotation matrix, and translation vector of the corresponding image, respectively.

[0033] According to the improved MVSNet-based 3D reconstruction method, preferably, the optimized depth map in step 6 is obtained from equation (6):

[0034]

[0035] In formula (6) This represents the optimized depth map. δ represents the depth value of pixel p in the initial estimated depth map, and δ represents the increment of the current depth.

[0036] According to the improved MVSNet-based 3D reconstruction method, preferably, the incremental value δ of the current depth is calculated by equation (7):

[0037] δ=-(J T J) -1 J T r (7)

[0038] In (7), J is a stacked Jacobian matrix. r is the stacked residual vector

[0039] A second aspect of the present invention provides a 3D reconstruction system based on an improved MVSNet, comprising:

[0040] The data acquisition module is used to acquire the image set corresponding to the target object or scene;

[0041] The data processing module is used to input the image set obtained by the data acquisition module into the neural network model to obtain a depth map. The neural network model is the improved MVSNet network described in the first aspect of this invention.

[0042] The 3D reconstruction module is used to generate a 3D point cloud of the target object or scene by fusing all the depth maps obtained by the data processing module through preset algorithms and coordinate transformations, thereby obtaining a 3D model.

[0043] According to the aforementioned 3D reconstruction system based on improved MVSNet, the preferred data processing module includes a feature extraction unit, a matching cost volume construction unit, a matching cost volume regularization unit, a depth regression unit, and a depth map optimization unit, with each unit described in detail below:

[0044] The feature extraction unit is used to extract features from the input source image and reference image. It extracts features from each image to be processed through 2D convolution and ECA module to obtain feature maps of the source image and reference image.

[0045] The matching cost body construction unit is used to further process the feature map obtained by the feature extraction unit and construct the matching cost body. First, a differentiable homography transformation is used to obtain the feature body, and then the feature body is aggregated by the group similarity measurement module to form the final matching cost body.

[0046] The matching cost volume regularization unit is used to remove noise contained in the matching cost volume obtained by the matching cost volume construction unit due to non-Lambertian surfaces, occlusion, etc. The matching cost volume is regularized by 3D-UNet at four scales to obtain the probability volume.

[0047] The depth regression unit is used to perform depth regression on the probability volume obtained from the matching cost volume regularization unit to obtain the initial estimated depth map.

[0048] The depth map optimization unit is used to further optimize the initial estimated depth map obtained by the depth regression unit. The Gauss-Newton optimization module iteratively optimizes the initial estimated depth map to generate a more refined depth map.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] 1. This invention proposes a 3D reconstruction method based on an improved MVSNet network architecture. The main steps of this method include feature extraction, matching cost volume construction, matching cost volume regularization, depth regression, depth map optimization, 3D point cloud transformation, and 3D model visualization. To improve the completeness and accuracy of the reconstruction results, this invention integrates an ECA (Efficient Channel Attention Module) module in the feature extraction stage. Through 8 convolutional layers and 2 ECA modules, features are extracted from the image, enabling more effective learning of channel attention and improving the expressive power of the features.

[0051] 2. This invention incorporates a group similarity measurement module during the cost volume construction process to construct a lightweight cost volume, reducing memory consumption. Furthermore, it includes a Newton-Gaussian optimization module to optimize the initially estimated depth map, resulting in a better depth map. The 3D model reconstructed by this invention was tested on the DTU dataset, demonstrating higher accuracy and completeness compared to the 3D point cloud reconstructed by the traditional MVSNet network. Attached Figure Description

[0052] Figure 1 A schematic diagram of the improved MVSNet network structure;

[0053] Figure 2 This is a schematic diagram of the ECA module;

[0054] Figure 3 Schematic diagram of the Gauss-Newton optimization module;

[0055] Figure 4 This is a schematic diagram of the structure of the 3D reconstruction system based on the improved MVSNet of this invention. Detailed Implementation

[0056] The following embodiments are only for further elaboration of the present invention. It should be noted that all techniques and scientific terms used in this invention, unless otherwise stated, have the same meaning as those in the technical field to which this invention pertains. Experimental methods in the following embodiments that do not specify specific conditions all employ conventional techniques in this technical field or follow the conditions recommended by the manufacturer; reagents or instruments whose manufacturers are not specified are all commercially available conventional products.

[0057] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0058] Example 1

[0059] In order to improve multi-view Figure 3 To improve the accuracy and completeness of 3D point cloud generation in 3D reconstruction, this embodiment proposes a 3D reconstruction method based on an improved MVSNet, whose network structure is as follows: Figure 1 As shown. The main process of this method includes feature extraction, matching cost volume construction, matching cost volume regularization, depth regression, depth map optimization, 3D point cloud transformation, and 3D model visualization. This 3D reconstruction method based on improved MVSNet includes the following steps:

[0060] Step 1: Obtain a set of multiple images of the same scene and the same object from multiple perspectives, and select one reference image and (N-1) source images from it; the specific process is as follows:

[0061] A total of N images are selected, each with a height of H and a width of W. One of these images is used as the reference image, denoted as I0, and the remaining (N-1) images are used as the source images, denoted as I1. As input to the network.

[0062] Step 2: Input the above reference image and source image into the improved MVSNet network, and extract features from each image through the ECA module and convolutional layers to obtain feature maps for all images; the specific process is as follows:

[0063] The input source and reference images are 3×H×W in size, and are processed through 8 convolutional layers and 2 ECA modules (e.g., ...). Figure 2 As shown), the final output size is The convolutional layers share weights. The third and sixth convolutional layers are downsampled to half their original height and width, and after downsampling, the ECA module is used to learn channel attention. Learning channel attention before further feature extraction allows for better feature aggregation and improved feature expressiveness.

[0064] Typically, the development of attention modules can be divided into two directions: enhancing feature aggregation, i.e., improving feature convergence strategies, and combining channel and spatial attention. However, adding attention mechanisms generally makes the model more complex, increasing computational cost. The ECA module, a lightweight channel attention module, aims to learn channel attention at a low cost, improving the efficiency of learning channel attention. Therefore, the ECA module adds only a few parameters and accelerates convergence. In this embodiment, the ECA module is added to the feature extraction part to improve the feature representation capabilities of the source and reference image feature maps, which is helpful for subsequent steps such as constructing the matching cost body.

[0065] The implementation process of the ECA module is as follows: After aggregating convolutional features using GAP (Global Average Pooling) without dimensionality reduction, the ECA module adaptively determines the kernel size k, then performs 1D convolution, and finally executes the Sigmoid function to learn channel attention. The ECA module does not change the size of the input and output tensors. A schematic diagram of the ECA module is shown below. Figure 2 As shown, the ECA module implements a local cross-channel interaction strategy without dimensionality reduction through 1D convolution. This avoids the benefit of channel dimensionality reduction for learning effective channel attention, as channel dimensionality reduction disrupts the correspondence between channels and weights. Cross-channel interaction is achieved by fusing neighborhood channel features through 1D convolution, with all channels sharing the same learnable parameters. To avoid the significant computational cost of manually adjusting k through cross-validation, the kernel size of the 1D convolution is adaptively selected, where kernel size k represents the coverage of the local cross-channel interaction.

[0066] The basic structure of the entire feature extraction network is shown in Table 1. The final output reference feature map can be denoted as f0, and the source feature map can be denoted as f0.

[0067] Table 1 Basic Structure of Feature Extraction Network

[0068]

[0069] The feature maps extracted by the 8 convolutional layers and 2 ECA modules significantly enhance feature aggregation compared to the original method, bringing a significant performance gain while almost not increasing the complexity of the model.

[0070] Step 3: Perform differentiable homography transformation on the feature maps extracted in Step 2 at different depth hypothetical planes, calculate the matching cost at different depths, and obtain the feature volume; uniformly divide the feature volume into g channel groups, calculate the group similarity using the group similarity measurement module, and construct and aggregate the matching cost volume based on the group similarity. The specific process is as follows:

[0071] Based on the principle of planar scanning algorithm, a plane perpendicular to the principal optical axis of the reference camera is used as the depth assumption plane. All feature maps are distorted into different front planes of the reference camera, mapping all pixel coordinates on the source image to pixel coordinates on the reference image. The minimum depth is denoted as d. min According to the depth assumption, the interval is continuously mapped to the maximum depth d. max The twisting process is called a differentiable homography transformation, and the corresponding calculation is given by equation (1):

[0072]

[0073] In equation (1), K i K ref Camera intrinsics for the source and reference images, respectively, R ref，i Let t be the corresponding rotation matrix. ref，i Let d be the corresponding translation vector. j p represents the corresponding sampling depth value. i，j For I i The coordinates of pixel p in the source image.

[0074] By twisting the pixel coordinates of all source images to the coordinates of the reference image using equation (1), the depth features of all source images can be denoted as follows: Depth d j The depth features of all source images can be denoted as: The matching cost body at this point has dimensions C×D×H×W, where C represents the channel dimension and D represents the depth dimension. It is then evenly divided into g groups according to the channel dimension. The group similarity between them is calculated by equation (2):

[0075]

[0076] In equation (2), the range of values ​​for g is [0, G-1]; yes The g-th cost body corresponding to the matching cost body; yes The corresponding cost body of the g-th group; <·> is the inner product operation.

[0077] The feature volume dimension output by the group similarity measurement module is: Form N feature bodies, denoted as To accommodate input from any source image, the feature volume is averaged, and the final matching cost volume can be calculated using equation (3):

[0078]

[0079] In this embodiment, setting G=8 allows for a lightweight 8-channel cost body, which significantly reduces memory consumption.

[0080] Step 4: Regularize the matching cost body using 3D convolution to obtain the probability body; the specific process is as follows:

[0081] The aggregated matching cost volume contains noise due to non-Lambertian surfaces, occlusion, etc., so it is regularized using 3D convolution to obtain the probability volume. The probability volume can be used not only for pixel-by-pixel depth prediction but also to measure the confidence of the estimate. This embodiment uses a 3D-UNet network structure with four scales to encode and decode the original matching cost volume, and finally compresses each channel to 1. Finally, the probability values ​​are normalized using a SoftMax operation in the depth direction. The output probability volume after regularization has the following dimensions:

[0082] Step 5: Calculate the expected depth value along the depth direction using the above probability volume to obtain the initial estimated depth map; the specific process is as follows:

[0083] The regularized probability volume is then used to regress the probability of each pixel at depth d using the SoftArgMin function. This yields the confidence score of each pixel along the depth direction in the reference image, corresponding to the pixel's initial depth value. A higher confidence score indicates a greater probability of the pixel at that depth. The output depth map size is... Meanwhile, to avoid gaps and unevenness along the depth direction, the depth expectation value is used as the depth estimate for this pixel to smooth the entire depth map. The depth expectation value D is calculated by equation (4):

[0084]

[0085] In equation (4), P(d) represents the estimated probability of all pixels at depth d, [d min ,d max [] represents the range of depth values.

[0086] Step 6: Optimize the depth map obtained in Step 5 using the Gauss-Newton optimization module to obtain the optimized depth map. A schematic diagram of the Gauss-Newton optimization module is shown below. Figure 3 As shown. The specific process is as follows:

[0087] Feature extraction is performed on the original reference image and multiple source images using 2D convolution to obtain corresponding feature maps. The depth value predicted from the initial estimated depth map obtained after depth regression is denoted as . The depth value of pixel p is denoted as Project point p from the reference image onto p' in each source image. i At this point, calculate p and p'. i The difference in the corresponding feature map values ​​is used to minimize the objective function, which is shown in equation (5):

[0088]

[0089] In equation (5), F i F0 is the feature map of the source image, and F0 is the feature map of the reference image. i Let p' be the projection point of pixel p in the i-th source image. i Calculated using equation (6):

[0090]

[0091] In formula (6) These represent the camera intrinsic parameters, rotation matrix, and translation vector of the corresponding image, respectively.

[0092] Specifically, from Start by calculating the residual r of pixel p in each source image. i (p), the residual is described by equation (7):

[0093] r i (p)=F i (p' i )-F0(p) (7)

[0094] For each residual r i (p), calculate about The first derivative is calculated by equation (8):

[0095]

[0096] Finally, the incremental value δ of the current depth can be obtained, as shown in equation (9):

[0097] δ=-(J T J) -1 J T r (9)

[0098] In (9), J is a stacked Jacobian matrix. r is the stacked residual vector The optimized depth map is calculated from this, and the optimized depth map is described by equation (10):

[0099]

[0100] Step 7: Fuse the optimized depth map, obtain a dense point cloud through coordinate transformation, and visualize the dense point cloud as a 3D model to obtain a 3D reconstructed model. The specific process is as follows: transform each pixel on the optimized depth map from the pixel coordinate system to the camera coordinate system. At this time, the camera coordinate system is the world coordinate system, and the constraints of the coordinate transformation are the camera parameters. The transformation relationship is shown in Equation (11):

[0101]

[0102] In equation (11), u and v are arbitrary coordinate points in the image coordinate system; f / dx and f / dy are the focal lengths of the camera along the x and y directions, respectively; u0 and v0 are the center coordinates of the image; x w y w z w These represent the three-dimensional coordinates of a point in the world coordinate system; z c The z-axis value represents the camera coordinates, i.e., the distance (depth) from the target to the camera; R and T are the 3×3 rotation matrix and 3×1 translation matrix of the extrinsic parameter matrix, respectively.

[0103] Since the world coordinate origin and the camera origin coincide, meaning there is no rotation or translation, the external parameter matrix is ​​set as follows:

[0104]

[0105] The optimized depth map is converted into a dense point cloud through coordinate transformation and saved as a ply file. The saved 3D point cloud can be visualized as a 3D model using MeshLab software.

[0106] The 3D point cloud models generated by the improved MVSNet-based 3D reconstruction method were evaluated by comparing the accuracy and completeness of the generated 3D point clouds. Eight Scans (Scan1, Scan4, Scan9, Scan10, Scan11, Scan12, Scan13, and Scan15) were selected in the validation set of the DTU dataset. The experimental results are shown in Table 2.

[0107] Table 2 Accuracy Results

[0108] MVSNet 0.3969 0.3767 0.3868 Improved MVSNet 0.3248 0.3426 0.3337

[0109] As can be seen from Table 2, compared with the original MVSNet method, the 3D point cloud model generated by the 3D reconstruction method based on the improved MVSNet proposed in this invention has improved accuracy by 22.23%, completeness by 9.96%, and overall point cloud reconstruction accuracy and pixel completeness by 15.92%.

[0110] In summary, the proposed 3D reconstruction method based on improved MVSNet achieves higher accuracy and completeness while suppressing model complexity growth. The model was trained and tested on the DTU dataset, and experiments verified that the 3D reconstruction method based on improved MVSNet described in this invention improves the reconstruction results.

[0111] Example 2

[0112] A 3D reconstruction system based on an improved MVSNet is illustrated in the following schematic diagram. Figure 4 As shown, the system includes a data acquisition module, a data processing module, and a 3D reconstruction module, with each module detailed below:

[0113] The data acquisition module is used to acquire the image set corresponding to the target object or scene;

[0114] The data processing module is used to process the obtained image set of the target object or scene, and input the reference image and source image into the neural network model to obtain the depth map. The neural network model is the improved MVSNet model described in Example 1. Specifically, it is obtained by introducing the ECA effective channel attention module, group similarity measurement module and Gauss-Newton optimization module on the basis of the original network.

[0115] The 3D reconstruction module is used to generate a 3D point cloud of the target object or scene by fusing the depth maps obtained from the data processing module through preset algorithms and coordinate transformations, thereby obtaining a 3D model and realizing the 3D reconstruction of the corresponding target or scene.

[0116] The aforementioned data processing module includes a feature extraction unit, a matching cost volume construction unit, a matching cost volume regularization unit, a depth regression unit, and a depth map optimization unit. The specific details of each unit are as follows:

[0117] The feature extraction unit is used to extract features from the input source image and reference image. It extracts features from each image to be processed through 2D convolution and ECA module to obtain feature maps of the source image and reference image.

[0118] The matching cost body construction unit is used to further process the feature map obtained by the feature extraction unit and construct the matching cost body. First, a differentiable homography transformation is used to obtain the feature body, and then the feature body is aggregated by the group similarity measurement module to form the final matching cost body.

[0119] The matching cost volume regularization unit is used to remove noise contained in the matching cost volume obtained by the matching cost volume construction unit due to non-Lambertian surfaces, occlusion, etc. The matching cost volume is regularized by 3D-UNet at four scales to obtain the probability volume.

[0120] The depth regression unit is used to perform depth regression on the probability volume obtained from the matching cost volume regularization unit to obtain the initial estimated depth map.

[0121] The depth map optimization unit is used to further optimize the initial estimated depth map obtained by the depth regression unit. The Gauss-Newton optimization module iteratively optimizes the initial estimated depth map to generate a more refined depth map.

[0122] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A 3D reconstruction method based on an improved MVSNet, characterized in that, Includes the following steps: Step 1: Obtain a set of N multi-view images of the same scene and the same object, and select one as the reference image and (N-1) images as the source images; Step 2: Input the reference image and the source image into the improved MVSNet network, and extract features from each image to obtain feature maps of the reference image and the source image; the feature extraction part of the improved MVSNet network includes 8 convolutional layers and 2 ECA modules, with the 2 ECA modules set after the third and sixth convolutional layers, respectively; Step 3: Perform differentiable homography transformation on the feature map at different depth hypothetical planes to obtain the feature volume; divide the obtained feature volume into g channel groups, calculate the group similarity through the group similarity measurement module, and obtain the matching cost volume based on the group similarity; Step 4: Use 3D convolution to perform regularization on the matching cost volume to obtain the probability volume; Step 5: Calculate the expected depth value along the depth direction of the probability volume to obtain the initial estimated depth map; Step 6: Optimize the initially estimated depth map using the Gauss-Newton optimization method to obtain the optimized depth map; Step 7: Fuse the optimized depth map, obtain a dense point cloud through coordinate transformation, visualize the dense point cloud as a 3D model, and obtain a 3D reconstructed model.

2. The 3D reconstruction method based on the improved MVSNet as described in claim 1, characterized in that, The feature map is subjected to differentiable homography transformation on different depth assumption planes to obtain the feature volume. Specifically, the process is as follows: using a plane perpendicular to the principal optical axis of the reference camera as the depth assumption plane, the feature map is distorted onto different front planes of the reference camera. All pixel coordinates on the source image are mapped to pixel coordinates on the reference image, starting from the minimum depth denoted as... According to the depth assumption, the interval is continuously mapped to the maximum depth. , thus obtaining the feature body.

3. The 3D reconstruction method based on the improved MVSNet according to claim 2, characterized in that, The twisting process is shown in equation (1): （1） In formula (1) , These are the camera intrinsics for the source image and the reference image, respectively. Here is the corresponding rotation matrix. For the corresponding translation vector, For the corresponding sampling depth value, for Pixels in the source image The coordinates.

4. The 3D reconstruction method based on the improved MVSNet according to claim 3, characterized in that, Group similarity is obtained from equation (2): （2） In equation (2), the range of values ​​for g is [0, G-1]; yes The g-th feature body corresponding to the feature body; yes The corresponding feature body of the g-th group; It is an inner product operation.

5. The 3D reconstruction method based on the improved MVSNet according to claim 4, characterized in that, The expected depth value mentioned in step 5 is obtained from equation (3): （3） In formula (3) This represents the estimated probability of all pixels with a depth assumed to be d, [d min ,d max [ ] represents the range of values ​​for the sampling depth.

6. The 3D reconstruction method based on the improved MVSNet according to claim 4, characterized in that, The Gauss-Newton optimization method described in step 6 includes: using 2D convolution to extract features from the reference image and the source image to obtain feature map A; the depth value predicted from the initially estimated depth map is denoted as... The depth value of pixel p is denoted as Points on the reference image Projected onto each source image At the location, calculate and The difference in the value of feature map A at the location is minimized, and the objective function is shown in equation (4): （4） In formula (4) The feature map of the source image; Feature map of the reference image; For pixels In the Projection points of Zhang Yuan's image.

7. The 3D reconstruction method based on the improved MVSNet according to claim 4, 5, or 6, characterized in that, The optimized depth map described in step 6 is obtained from equation (6): （6） In formula (6) This represents the optimized depth map. This represents the depth value of pixel p in the initially estimated depth map. This represents the increment value of the current depth.

8. A 3D reconstruction system based on an improved MVSNet, characterized in that, The three-dimensional reconstruction system employs the method described in any one of claims 1-7, comprising: The data acquisition module is used to acquire the image set corresponding to the target object or scene; The data processing module is used to input the image set obtained by the data acquisition module into the improved MVSNet network to obtain a depth map; The 3D reconstruction module is used to generate a 3D point cloud of the target object or scene by fusing all the depth maps obtained by the data processing module through preset algorithms and coordinate transformations, thereby obtaining a 3D model.

9. The 3D reconstruction system based on the improved MVSNet according to claim 8, characterized in that, The data processing module includes: The feature extraction unit is used to extract features from the input source image and reference image. It extracts features from each image to be processed through 2D convolution and ECA module to obtain feature maps of the source image and reference image. The matching cost body construction unit is used to further process the feature map obtained by the feature extraction unit and construct the matching cost body. First, a differentiable homography transformation is used to obtain the feature body, and then the feature body is aggregated by the group similarity measurement module to form the final matching cost body. The matching cost volume regularization unit is used to remove noise contained in the matching cost volume obtained by the matching cost volume construction unit due to non-Lambertian surfaces and occlusion. The matching cost volume is regularized by 3D-UNet at four scales to obtain the probability volume. The depth regression unit is used to perform depth regression on the probability volume obtained from the matching cost volume regularization unit to obtain the initial estimated depth map. The depth map optimization unit is used to further optimize the initial estimated depth map obtained by the depth regression unit. The Gauss-Newton optimization module iteratively optimizes the initial estimated depth map to generate a more refined depth map.

Images