A multi-view three-dimensional reconstruction method based on deep learning
By constructing a multi-scale feature extraction network and combining an attention mechanism with a residual network, the problem of insufficient reconstruction accuracy of depth estimation in high-complexity scenes is solved, achieving fast and efficient 3D reconstruction results, especially in weak texture regions and large scenes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-02-24
- Publication Date
- 2026-08-04
AI Technical Summary
Existing deep learning methods suffer from insufficient reconstruction accuracy and high GPU memory consumption, especially in high-complexity scenarios, when performing depth estimation. They also perform poorly in weak texture regions, have long running times, high memory consumption, and weak generalization ability.
We employ a deep learning-based multi-view 3D reconstruction method. This method extracts semantic features by constructing a multi-scale feature extraction network, combines an attention mechanism with a residual network for pixel-level evaluation of depth candidate hypotheses, and uses a variable convolutional network for depth sampling and filtering. Finally, we output a high-precision 3D point cloud.
It achieves fast and accurate 3D reconstruction, effectively handles complex large scenes, improves the reconstruction effect of weak texture areas, reduces memory consumption and running time, and enhances the generalization ability of the model.
Smart Images
Figure CN116310095B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of geometric computer vision 3D reconstruction, and relates to a multi-view method based on deep learning. Figure 3 Dimensional reconstruction method. Background Technology
[0002] In recent years, with the development of deep learning, the solution of computer vision problems has made significant progress. Recently, various 2D image processing techniques have been refined for application in 3D problems, and the reconstruction of the 3D shape of objects has become a hot topic. Many previous methods required complete 3D model data for supervision; however, such data is scarce and its acquisition is complex and costly. This has led to the development of multi-view... Figure 3 Deep learning-based 3D reconstruction methods, compared to traditional 3D reconstruction methods, can incorporate global semantic information into image reconstruction, thereby improving the reconstruction results in low-light and low-texture areas.
[0003] Current deep learning methods mostly employ 3D convolutional regularization for cost volume normalization to evaluate and derive the depth map during depth estimation. This evaluation method can maintain good reconstruction accuracy in low-complexity scenes, but the reconstruction process consumes a large amount of GPU memory, and it suffers from insufficient reconstruction accuracy in high-complexity scenes. Summary of the Invention
[0004] This solution addresses the problems and needs mentioned above by proposing a multi-view approach based on deep learning. Figure 3 The reconstruction method, by adopting the following technical solutions, is able to solve the above-mentioned technical problems.
[0005] To achieve rapid and accurate 3D reconstruction of objects, this invention proposes a deep learning-based 3D reconstruction method based on multi-view information.
[0006] The technical solution adopted in this invention is: a multi-view based on deep learning. Figure 3 The 3D reconstruction method is characterized by comprising the following steps:
[0007] Step 1: Construct a multi-scale feature extraction network to extract semantic features from multiple multi-view images at multiple scales, thereby obtaining key target features at multiple scales.
[0008] Step 2: Within a preset depth range, perform depth sampling to obtain sub-pixel level depth candidate hypotheses;
[0009] Step 3: Based on the attention mechanism and residual network, and combined with multi-scale feature maps, perform pixel-level depth evaluation and iterative calculation on depth candidate hypotheses to obtain the depth map;
[0010] Step 4: Input the depth map obtained in Step 3 and the feature map of the reference view into the depth optimization network, and obtain the final depth map after optimization.
[0011] Step 5: Fuse the depth maps of all reference views calculated in Step 4 to obtain the 3D scene reconstruction results and output the 3D point cloud. Attached Figure Description
[0012] Figure 1 This is a schematic diagram illustrating specific steps in an embodiment of the present invention;
[0013] Figure 2 This is a schematic diagram of the process in an embodiment of the present invention;
[0014] Figure 3 This is a reconstruction result of a large-scale outdoor scene in an embodiment of the present invention. Detailed Implementation
[0015] The present invention will now be described in further detail with reference to the accompanying drawings. The examples described are intended to facilitate understanding of the present invention and are not intended to limit the present invention.
[0016] The existing multi-view Figure 3 Traditional 3D reconstruction methods often fail to achieve good results in areas with weak texture or reflective surfaces, and suffer from problems such as long runtime, high memory consumption, and weak generalization ability. Therefore, this invention discloses a processing method based on an attention mechanism and residual networks. This method has strong generalization ability and can quickly and efficiently obtain high-precision depth maps, thereby achieving accurate and complete 3D reconstruction results. Figure 1 , Figure 2 As shown, this invention provides a multi-view based on deep learning. Figure 3 The dimensional reconstruction method of this invention includes the following specific steps:
[0017] Step 1: Construct a multi-scale feature extraction network to extract semantic features from multiple multi-view images at multiple scales, thereby obtaining key target features at multiple scales.
[0018] Specifically, multiple multi-view images are acquired by using drone photography equipment to capture images of the area to be reconstructed from multiple angles and in all directions, thereby obtaining a multi-view image dataset.
[0019] Furthermore, based on the image feature pyramid structure, a convolutional neural network is used to extract features at multiple scales from multiple multi-view images to obtain original feature maps. An upsampling layer is then applied to the low-resolution original feature map, and it is fused with the original feature map at the next scale to obtain target features with rich semantic information. This upsampling fusion process is applied to the original feature maps between adjacent scales to obtain feature maps at multiple scales.
[0020] In this embodiment, the process of training the multi-scale extraction network using a training set includes:
[0021] Input N multi-view images of size H×W, including one reference view. and N-1 source views { }. Mark them separately. , , The original feature maps at three scales are , , , Obtained through upsampling network New feature maps at different scales ,Will and Fusion Final feature map of scale Similarly, Through upsampling network and Fusion, Acquisition Multi-scale feature extraction networks extract key features at different scales. Compared to traditional convolutional networks, multi-scale feature extraction networks obtain feature maps with richer semantic information, which can effectively improve the generalization ability and training accuracy of the trained model.
[0022] Step 2: Within a preset depth range, perform depth sampling to obtain sub-pixel level depth candidate hypotheses.
[0023] Specifically, by projecting a pre-defined depth range into the reciprocal space and sampling in the form of the reciprocal of the depth value, sub-pixel level depth candidate hypotheses are obtained, which helps to improve the accuracy of depth assessment.
[0024] Furthermore, for the initial depth evaluation iteration, within the inverse space of the depth values, given the number of candidate depth hypotheses, uniform sampling is performed, and a random number is added to each candidate depth hypothesis to obtain initial candidate hypotheses. Based on a variable convolutional network, the coordinates of similar pixels for each pixel are obtained for depth sampling, yielding new candidate hypotheses. Performing depth sampling within the inverse space and sampling similar pixels increases the diversity of candidate depth hypotheses and the robustness of model training, effectively improving the accuracy of depth evaluation and the effect of 3D reconstruction.
[0025] Furthermore, for subsequent depth evaluation iterations, based on the depth map generated in the previous iteration, random numbers are added to the candidate depth values of each pixel to perturb them, thus obtaining the initial depth candidate hypotheses for this iteration. Similarly, depth sampling of similar pixels is performed using a variable convolutional network to obtain new depth candidate hypotheses for this iteration.
[0026] In this embodiment, based on a predefined reciprocal depth range [1 / , 1 / Uniform sampling is performed to generate multiple depth hypotheses for each pixel, and random numbers are added to each hypothesis to obtain a random depth hypothesis. A variable convolutional neural network is used to obtain the coordinates (x, y) of similar pixels for each pixel, and depth sampling is performed based on these coordinates to obtain new candidate hypotheses. This allows the network model to handle large-scale complex scenes and has better generalization ability. For subsequent iterations, random numbers are added to the depth map generated in the previous iteration to randomly perturb the depth within a reasonable depth range R, and the depth range R is gradually reduced in subsequent iterations. This further refines the depth evaluation results and corrects erroneous evaluations from previous iterations.
[0027] Step 3: Based on the attention mechanism and residual network, and combined with multi-scale feature maps, perform pixel-level depth evaluation and iterative calculation on depth candidate hypotheses to obtain a depth map.
[0028] For multi-view stereo matching, the correct depth value of any pixel in the reference image is also the correct depth value of the corresponding pixel in any source view. For multi-view depth evaluation, it is necessary to integrate the visible information from multiple source views to evaluate the depth value of each individual pixel in the reference view. To this end, a weighted calculation is performed using an attention mechanism to extract the weight information of different pixels in the reference view for maximum feature extraction. The cost value of each depth hypothesis for each pixel is obtained through grouping operations and weighted calculation. Subsequently, a 3D convolutional network is used for cost volume normalization to obtain the single-value of each depth hypothesis, and depth regression is performed to obtain the depth map. A residual network is then used to extract residual information from the reference view to optimize the depth map. Finally, a variable filtering method is applied based on a variable convolutional network. Compared to a fixed filtering window, variable filtering can obtain pixel coordinates on the same 3D plane. After averaging, unreasonable depth values in the depth map can be effectively smoothed, resulting in a more accurate depth map.
[0029] In this embodiment, the process of calculating the depth map through multi-view depth evaluation includes:
[0030] Obtain the reference view through the convolutional attention module. Up pixel Weight information ; Perform 3D convolution calculations on feature channels to obtain a reference view. With source view In pixels Group similarity Extract residual information from the reference view using a residual network. According to the formula: Calculate and obtain reference view The depth map. Wherein, according to the formula: Calculation and acquisition , and These represent the g-th feature group of the source view and the g-th feature group of the reference view, respectively. Represents the pixels corresponding to the reference view and the source view. , This represents the total number of reference views and source views. Representative reference view The depth hypothesis, through Exponential normalization yields the probability corresponding to each depth hypothesis.
[0031] For the depth map calculated above, a variable convolutional network is used to obtain the pixel coordinates of the same local plane in 3D space. Depth sampling is performed based on these pixel coordinates, and the mean depth of all sampled points is obtained to acquire a smoothed depth value. The variable filtering formula is shown below: .in Represents pixels The depth value, Representation and pixels Depth values of pixels in the same plane This represents the number of sampling points.
[0032] For this iteration, the depth map obtained from the above calculation is used as the input for the next iteration. This allows for continuous correction of erroneous depth assessments in subsequent iterations, improving the accuracy of depth assessment and enabling our model network to better cope with complex 3D reconstruction scenarios, thus greatly enhancing the model's generalization ability.
[0033] Step 4: Input the depth map obtained in Step 3 and the feature map of the reference view into the depth optimization network, and obtain the final depth map after optimization.
[0034] Specifically, to avoid a certain depth scale being shifted, the input depth map is normalized to [0,1].
[0035] Furthermore, the reference view feature maps extracted by the multi-scale feature network are upsampled using a deconvolutional network.
[0036] Similarly, the normalized depth map is upsampled to an H×W scale, and the two features obtained above are fused and processed using residual data.
[0037] The difference network extracts depth residual information, and finally adds the obtained depth residual information to the depth map obtained in step 3 to obtain the final optimized depth map.
[0038] In the experiment, the depth map output at each stage had varying degrees of error compared to the true value. To better reflect the loss changes throughout the training process, the losses from different stages were summed to obtain the final loss. Therefore, the loss function was defined as follows: .
[0039] In this embodiment, The target key features and the depth map obtained in step 3 are used as inputs. The feature map and the depth map normalized to [0,1] are upsampled respectively and combined together. The depth residual information is extracted through the residual network. The depth residual information is added to the normalized depth map. Then the result is denormalized to obtain the optimized depth map.
[0040] Step 5: Fuse the depth maps of all reference views calculated in Step 4 to obtain the 3D scene reconstruction results and output the 3D point cloud.
[0041] Specifically, using depth maps and their corresponding probabilities from different views as input, the depth maps are filtered based on illumination consistency and geometric constraint consistency to obtain the most reliable depth points. A visibility-based fusion algorithm minimizes the impact of occlusion, illumination, and other factors, minimizing depth occlusion and conflicts between different views to obtain a unified point cloud representation.
[0042] In this embodiment, a pre-set illumination consistency threshold α and a geometric constraint consistency threshold β are used to filter out depth points that do not meet the thresholds. To further suppress noise points, the average depth values of similar pixels that meet the threshold requirements are taken as the final visible depth. Finally, the fused depth map is directly projected into 3D space to output a 3D point cloud.
[0043] This application employs a multi-view approach suitable for large-scale 3D reconstruction. Figure 3 The 3D reconstruction algorithm, by introducing an attention mechanism and a residual network structure, effectively solves the problem of unsatisfactory reconstruction results in weak texture regions in current 3D reconstruction algorithms, and can meet the 3D reconstruction needs of complex large scenes.
[0044] The above results demonstrate the feasibility and effectiveness of the deep learning-based large-scene 3D reconstruction algorithm proposed in this application.
[0045] It should be noted that the detailed description of the implementation of the present invention above in conjunction with the accompanying drawings is only a preferred embodiment of the present invention. Within the knowledge scope of those skilled in the art, various changes can be made without departing from the spirit of the present invention. Any obvious modifications should fall within the protection scope of the present invention.
Claims
1. A multi-view 3D reconstruction method based on deep learning, characterized in that, Includes the following steps: Step 1: Construct a multi-scale feature extraction network to extract semantic features from multiple acquired multi-view images at multiple scales, thereby obtaining key target features at multiple scales. Step 2: Within a preset depth range, perform depth sampling in the inverse depth space to obtain sub-pixel level depth candidate hypotheses; Step 3: Based on the convolutional attention mechanism and residual network, the depth map is obtained by performing pixel-level depth evaluation and iterative calculation on the depth candidate hypothesis in combination with multi-scale feature maps, and the depth map is then smoothed by adaptive filtering through a variable convolutional network. The process of calculating the depth map for multi-view depth evaluation includes: obtaining the pixel weight information of the reference view through the convolutional attention module, grouping the feature channels and calculating the group similarity between the reference view and the source view through 3D convolution, extracting the residual information of the reference view through the residual network, and calculating the depth map according to the formula containing the above weight information, group similarity and residual information. The process of adaptive filtering and smoothing the depth map through a variable convolutional network includes: obtaining the pixel coordinates of the same local plane in three-dimensional space through a variable convolutional network, and obtaining the depth value of each sampling point based on the pixel coordinates and taking the average value to obtain the smoothed depth value. Step 4: Input the obtained depth map and the feature map of the reference view into the depth optimization network to obtain depth residual information for depth map optimization; The process of depth map optimization includes: normalizing the input depth map to [0,1], upsampling the reference view feature map and the normalized depth map respectively and then fusing them, and extracting depth residual information through a residual network to optimize the depth map. Step 5: Filter the depth map through illumination consistency constraints and geometric consistency constraints, perform depth map fusion based on visibility fusion algorithm, obtain the 3D scene reconstruction result, and output 3D point cloud; The depth map filtering and fusion process includes: filtering the depth map based on illumination consistency and geometric constraint consistency to obtain reliable depth points; using a visibility-based fusion algorithm to reduce the impact of occlusion and illumination; and obtaining a 3D reconstructed point cloud.
2. The deep learning-based multi-view 3D reconstruction method as described in claim 1, characterized in that, Based on the image feature pyramid structure, a convolutional neural network is used to extract features from multiple multi-view images at multiple scales to obtain the original feature map; an upsampling layer is applied to the low-resolution original feature map and fused with the original feature map at the next scale to obtain target features with rich semantic information. The above upsampling fusion process is applied to the original feature maps between two adjacent scales to obtain feature maps of multiple scales.
3. The deep learning-based multi-view 3D reconstruction method as described in claim 2, characterized in that, During the iterative calculation process, for the first iteration, within the inverse space of the depth value, given the number of candidate depth hypotheses, uniform sampling is performed, and a random number is added to each depth candidate hypothesis to obtain the initial candidate hypothesis; Based on a variable convolutional network, the coordinates of similar pixels for each pixel are obtained, and depth sampling is performed to obtain new candidate hypotheses to enhance the robustness of the model.
4. The deep learning-based multi-view 3D reconstruction method as described in claim 3, characterized in that, During the iterative calculation process, for subsequent iterations, in order to correct the erroneous evaluations in the previous iterations, random number perturbations are added to the candidate depth values of each pixel based on the depth map generated in the previous iteration to obtain the initial depth candidate hypothesis for this iteration; and a variable convolutional network is used to sample the depth of similar pixels to obtain a new depth candidate hypothesis for this iteration.
5. The deep learning-based multi-view 3D reconstruction method as described in claim 4, characterized in that, The process of calculating a depth map for multi-view depth evaluation includes: Obtain the reference view through the convolutional attention module. Up pixel Weight information Perform grouped 3D convolution calculations on the feature channels to obtain a reference view. With source view In pixels Group similarity at location Extract residual information from the reference view using a residual network. According to the formula: Calculate and obtain reference view The depth map; where, according to the formula: Calculation and acquisition , and These represent the g-th feature group of the source view and the g-th feature group of the reference view, respectively. This indicates the corresponding pixels in the reference view and the source view. , This indicates the total number of reference views and source views. Indicates reference view The depth assumption, through Exponential normalization is used to calculate the probability corresponding to each depth hypothesis.
6. The deep learning-based multi-view 3D reconstruction method as described in claim 5, characterized in that, A variable convolutional network is used to obtain the coordinates of pixels in the same local plane in 3D space. The depth values of each sampling point are then obtained based on these pixel coordinates, and the average value is calculated to obtain a smoothed depth value. The variable filtering formula is shown below: ,in, Represents pixels The depth value, Represents pixels Depth values of pixels on the same plane This indicates the number of sampling points.
7. The deep learning-based multi-view 3D reconstruction method as described in claim 6, characterized in that, The input depth map is normalized to [0,1] to avoid depth shift; the reference view feature map extracted by the multi-scale feature network is upsampled through a deconvolution network, and the normalized depth map is upsampled to the H×W scale; the two features are fused and the depth residual information is extracted through the residual network to optimize the depth map.
8. The deep learning-based multi-view 3D reconstruction method as described in claim 7, characterized in that, The depth map is filtered based on illumination consistency and geometric constraint consistency to obtain reliable depth points; a visibility-based fusion algorithm reduces the impact of occlusion and lighting factors, minimizing depth occlusion and conflicts between different views to obtain a 3D reconstructed point cloud.