Deep learning multi-view stereo 3d reconstruction algorithm based on path aggregation
By employing path aggregation strategies and deep learning methods, the problem of poor reconstruction performance in weak and repetitive texture regions by traditional multi-view stereo algorithms has been solved, achieving high-precision and efficient 3D reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-30
- Publication Date
- 2026-03-31
AI Technical Summary
Traditional multi-view stereo algorithms are not effective in reconstructing non-Lambertian regions and regions with low or repetitive textures, and the reconstruction speed is slow.
A deep learning-based multi-view stereo 3D reconstruction algorithm employing a path aggregation strategy utilizes a path aggregation feature extraction network and deformable convolutional layers, combined with deep learning for feature extraction and cost volume regularization, to generate a high-precision depth map and perform 3D reconstruction.
It improves the reconstruction accuracy in areas with weak and repetitive textures, enhances the integrity and speed of reconstruction, and achieves efficient 3D reconstruction results.
Smart Images
Figure CN116778091B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer graphics and image processing technology, specifically to a multi-view stereo 3D reconstruction algorithm based on a deep learning network method using a path aggregation strategy. Background Technology
[0002] The main task of multi-view stereoscopic 3D reconstruction is to reconstruct a 3D model of a scene from multi-view images. With the increasing demand for 3D models of scenes or objects in fields such as autonomous driving, virtual reality, and medical imaging, 3D reconstruction technology is also developing rapidly.
[0003] 3D reconstruction methods can be categorized into active and passive methods based on the type of sensor used for acquisition. Active 3D reconstruction requires specific actions or scans of the target using corresponding sensors, such as structured light and lidar. Active 3D reconstruction methods typically involve actively sending pulsed lasers or light rays to the target and receiving reflected or echoed information to obtain the target's 3D structural information. Active 3D reconstruction is characterized by high accuracy and speed, but due to its reliance on acquisition equipment, it is also characterized by high cost and poor versatility. Passive 3D reconstruction utilizes acquired image sequences to complete 3D reconstruction. Leveraging knowledge from computer graphics, multi-view geometry, and other fields, it uses 2D information from the images to recover depth information and achieve the final 3D reconstruction.
[0004] Passive 3D reconstruction, because it does not rely on special acquisition equipment, has the advantages of easy data acquisition and high versatility, but its accuracy is often limited by the data itself. Since acquiring image sequences is more economical and reliable than using depth sensors to acquire 3D structural information of a scene or target, passive 3D reconstruction is a better choice for tasks with low real-time requirements.
[0005] Multi-view stereo (MVS) uses images acquired from multiple cameras at different viewpoints to reconstruct a 3D model of a scene or target. Current MVS algorithms are mostly based on depth maps. The MVS task involves predicting the depth of a given series of images and their corresponding corrected camera parameters, then fusing the predicted depth maps to reconstruct a dense point cloud model of the scene.
[0006] Therefore, the main research focus of multi-view stereo algorithms is to generate accurate depth maps for each image, so as to perform depth map fusion to generate a dense 3D point cloud model. Traditional multi-view stereo algorithms rely on manually designed feature operators to extract features. They can achieve good reconstruction results under the ideal Lambertian reflection model, but perform poorly in non-Lambertian regions and low-texture or repetitive texture regions, resulting in incomplete reconstruction. As deep learning has demonstrated excellent performance in many computer vision tasks, network structures represented by convolutional neural networks have been proven to be able to extract multi-scale and multi-dimensional features of images. Researchers have begun to introduce deep learning into multi-view stereo tasks. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes a deep learning multi-view stereoscopic reconstruction algorithm based on a path aggregation strategy. It performs well in areas with weak or repetitive textures, and features fast reconstruction speed, convenient implementation, and high completeness and accuracy.
[0008] Technical Solution: This invention discloses a deep learning-based multi-view stereo 3D reconstruction algorithm based on path aggregation. The specific 3D reconstruction algorithm is as follows:
[0009] Step 1: Calibrate the camera to obtain camera parameters, and use the equipment to acquire images of the scene from various angles to determine the internal and external parameters of each view.
[0010] Step 2: For the captured image sequence, calculate the depth map for each image. Specifically:
[0011] Step 2.1: Set this view as the reference view, select n other source views, and determine their intrinsic and extrinsic parameters.
[0012] Step 2.2: Pass the n+1 views through the path aggregation feature extraction network to obtain multi-scale feature maps.
[0013] The path aggregation feature extraction network consists of two bottom-up and one top-down convolutional neural networks, three deformable convolutional layers, and three convolutional block attention modules. The input image sequentially passes through the bottom-up, top-down, and path aggregation networks. Each path is divided into three stages by three residual convolutional blocks, and two paths are connected by lateral links with 1x1 kernel sizes. For ease of explanation, the output of the top layer is named stage 1. The output of stage 1 of the bottom-up path aggregation and the previous stage 2 is added together via a lateral link. This output then passes through stage 2 of the current path, and is added together with the output of stage 3 of the previous stage via a lateral link, serving as input to stage 3 of the current path. Finally, the outputs of the three stages of the path aggregation network are passed through a deformable convolutional layer and a convolutional block attention module to obtain the final feature map. The resolutions of the feature maps output from the three stages are 1 / 4, 1 / 2, and 1 times that of the input reference view, respectively.
[0014] Step 2.3: Name the reference image feature map F0 and the source image feature map F i (i=1,…,n). Based on homography transformation, and with the help of the camera intrinsic and extrinsic parameters of the reference view and the source view, the feature map F of the source view is obtained by formula (1). i The homography matrix H of the reference view feature map F0 i (·)
[0015]
[0016] Since a cascaded, coarse-to-fine cost volume construction method is adopted, the homography matrix formula for each stage needs to take into account the depth sampling offset. Therefore, starting from the second stage, formula (2) is adopted.
[0017]
[0018] in This represents the predicted depth of the m-th pixel in the k-th stage. This represents the residual disparity of the m-th pixel to be learned in the (k+1)-th stage. It is determined by the number of depth sampling planes and the sampling interval and sampling interval attenuation coefficient at each stage.
[0019] Specifically, three stages are pre-defined with resolutions ranging from low to high, and the depth sampling planes are assumed to be 48, 32, and 8. For the first stage, the depth sampling planes are selected as 48 depth planes uniformly sampled within a given depth range. After the first stage depth prediction is completed, 16 depth planes are sampled above and below the predicted depth, satisfying a total of 32 sampling planes. After the second stage depth prediction is completed, the depth sampling planes for the third stage are determined in the same way as the second stage, with 4 depth planes sampled above and below the predicted depth, satisfying a total of 8 depth sampling planes. This cascaded cost volume construction significantly reduces memory usage during the cost volume regularization stage, enabling this invention to be used for high-resolution vision. Figure 3 Reconstruction.
[0020] Step 2.4: Calculate each F according to formula (2) i After obtaining the homography matrix of (i=1,…,n) to F0, F is obtained using formula (3). i Projecting the assumed depth planes onto the reference view yields n feature volumes F. i ′.
[0021] F i ′=H i (d)F i (3)
[0022]
[0023] Obtain each feature body F i After that, the n feature bodies are aggregated into a cost body C using formula (4) based on variance. This process is carried out in three stages.
[0024] Step 2.5: After constructing the cost body in the three stages, the cost body is regularized using 3D-Unet. The cost body C first passes through conv0, conv1, conv2, conv3, conv4, conv5, and conv6, which are 3D convolutional modules. Each module contains a 3D convolutional layer, a batch normalization layer, and a ReLU layer. The output of conv6 is then passed through conv7; the output of conv7 is added to the output of conv4 and input into conv8; the output of conv8 is added to the output of conv2 and input into conv9; the output of conv9 is added to the output of conv0, and then passed through an outline layer to obtain the final probability body P. conv7, conv8, and conv9 are 3D deconvolutional modules, each containing a deconvolutional layer, a batch normalization layer, and a ReLU layer. The parameters of the cost body regularization network in the three stages are not shared.
[0025] Step 2.6: After cost volume regularization, the probability volume P of the current stage resolution is obtained. Softmax operation is performed on the probability volume P, and the expected value of the probability of each pixel at each depth is calculated to obtain a preliminary depth map.
[0026] Step 2.7: The initial depth map is further refined using a depth map refinement network. First, the input image and depth information are concatenated along the channel dimension. Then, the concatenated result is passed to a series of convolutional layers, RefineNet_conv1, RefineNet_conv2, and RefineNet_conv3, for feature extraction and learning. Next, depth residual prediction is performed using RefineNet_res to obtain a residual map representing the depth improvement. Finally, the initial depth estimate is added to the residual map to obtain the final depth estimate result, depth_refined, which is returned to obtain a more accurate depth estimate, completing the depth estimation of the reference view.
[0027] Step 3: Based on the depth map obtained from the depth estimation of all views completed in Step 2, perform depth map fusion to generate a spatial point cloud 3D representation of the scene.
[0028] Beneficial effects
[0029] This invention utilizes a path-aggregation-based deep learning method to extract multi-scale features of images, fully leveraging the high- and low-level semantic features of images at various scales to achieve end-to-end 3D reconstruction of scenes, with simple deployment and implementation.
[0030] This invention employs a path-aggregation feature extraction network, which enhances the propagation of low-level semantic features through a path aggregation strategy. Furthermore, it further strengthens these features using deformable convolutional layers and convolutional block attention modules. This improves reconstruction accuracy in areas with weak or repetitive textures. Moreover, by leveraging deep learning for feature extraction and cost volume regularization, it achieves higher reconstruction efficiency compared to traditional multi-view stereoscopic 3D reconstruction techniques. Attached Figure Description
[0031] To more clearly illustrate the technical description and effects of the present invention, the accompanying drawings will be described in detail below.
[0032] See Figure 1 This is the overall flowchart of the three-dimensional reconstruction algorithm of the present invention.
[0033] See Figure 2 This is a schematic diagram of the network structure for multi-view depth estimation in the 3D reconstruction method of this invention.
[0034] See Figure 3 This is a reconstruction example of the three-dimensional reconstruction algorithm of this invention. Detailed Implementation
[0035] The present invention will now be described in detail with reference to the accompanying drawings and formulas. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0036] See Figure 1 This is a schematic diagram of the deep learning multi-view 3D reconstruction method based on path aggregation provided by the present invention. After capturing multi-view images with a camera, the steps are as follows:
[0037] The camera is calibrated to obtain camera parameters, and images of the scene are acquired from various angles using the device to determine the intrinsic and extrinsic parameters of each view. For the captured image sequence, a depth map is calculated for each image. Specifically:
[0038] Step 1: Set this view as reference view I0, and select n other source views I. i (i = 1, ..., n), and determine their intrinsic and extrinsic parameters. More specifically, in the selection of the source view, the view closest to the reference view should be selected based on the extrinsic parameters, and secondly, n should be greater than or equal to 2 (n ≥ 2).
[0039] Step 2: Process the reference view and the source view, a total of n+1 views, through the path aggregation feature extraction network to obtain multi-scale feature maps, and name the reference image feature map F0 and the source image feature map F10 respectively. i (i = 1, ..., n).
[0040] The path aggregation feature extraction network consists of two bottom-up and one top-down convolutional neural networks, three deformable convolutional layers, and three convolutional block attention modules. The input image passes sequentially through the bottom-up, top-down, and path aggregation networks. Each path is divided into three stages by three residual convolutional blocks, and two paths are connected by lateral links with a convolutional kernel size of 1x1.
[0041] For ease of explanation, we'll name the output of the topmost layer "Stage 1." Starting from the top down, the output of Stage 1 from the bottom-up path aggregation is combined with the output of Stage 2 from the previous path via a lateral link. This combined output then passes through Stage 2 of the current path. The output of Stage 2 is then combined with the output of Stage 3 from the previous path via a lateral link, and used as input to Stage 3 of the current path. Finally, the outputs of the three stages from the bottom-up path aggregation are passed through a deformable convolutional layer and a convolutional block attention module to obtain the final feature map.
[0042] Between each stage, the stride of the last residual convolution kernel is 2, so the resolution of the final output feature map is 1 / 4, 1 / 2, and 1 times that of the input reference view, respectively.
[0043] Step 3: Based on the homography transformation, and using the camera intrinsic and extrinsic parameters of the reference view and the source view, obtain the feature map F of the source view using formula (1). i The homography matrix H projected onto the reference view feature map F0 i (·)
[0044]
[0045] Since a cascaded, coarse-to-fine cost volume construction method is adopted, the homography matrix formula for each stage needs to take into account the depth sampling offset. Therefore, starting from the second stage, formula (2) is used.
[0046]
[0047] in This represents the predicted depth of the m-th pixel in the k-th stage. This represents the residual disparity of the m-th pixel to be learned in the (k+1)-th stage. It is determined by the number of depth sampling planes and the sampling interval and sampling interval attenuation coefficient at each stage.
[0048] Specifically, three stages are pre-defined with resolutions ranging from low to high, and the depth sampling planes are assumed to be 48, 32, and 8. For the first stage, the depth sampling planes are selected as 48 depth planes uniformly sampled within a given depth range. After the first stage depth prediction is completed, 16 depth planes are sampled above and below the predicted depth values, satisfying a 32-sampling-plane requirement. After the second stage depth prediction is completed, the depth sampling planes for the third stage are determined in the same way as the second stage, with 4 depth planes sampled above and below the predicted depth values, satisfying an 8-sampling-plane requirement. This cascaded cost volume construction significantly reduces memory usage during the cost volume regularization stage, enabling this invention to be used for high-resolution viewing. Figure 3 Reconstruction.
[0049] Step 4: Calculate each F according to formula (2) i After obtaining the homography matrix of (i=1,…,n) to F0, F is obtained using formula (3). i Projecting the assumed depth planes onto the reference view yields n feature volumes F. i ′.
[0050] F i ′=H i (d)F i (3)
[0051]
[0052] Obtain each feature body F iAfter that, the n feature bodies are aggregated into a cost body C using formula (4) based on variance. This process is carried out in three stages.
[0053] Step 5: After constructing the cost body in the three stages, the cost body is regularized using 3D-Unet. The cost body C first passes through conv0, conv1, conv2, conv3, conv4, conv5, and conv6, which are 3D convolutional modules. Each module contains a 3D convolutional layer, a batch normalization layer, and a ReLU layer. The output of conv6 is then passed through conv7; the output of conv7 is added to the output of conv4 and input into conv8; the output of conv8 is added to the output of conv2 and input into conv9; the output of conv9 is added to the output of conv0, and then passed through an outline layer to obtain the final probability body P. conv7, conv8, and conv9 are 3D deconvolutional modules, each containing a deconvolutional layer, a batch normalization layer, and a ReLU layer. The parameters of the cost body regularization network in the three stages are not shared.
[0054] Step 6: After cost volume regularization, the probability volume P of the current stage resolution is obtained. Softmax operation is performed on the probability volume P, and the expected value of the probability of each pixel at each depth is calculated to obtain a preliminary depth map.
[0055] Step 7: The initial depth map is further refined using a depth map refinement network. First, the input image and depth information are concatenated along the channel dimension. Then, the concatenated result is passed to a series of convolutional layers, RefineNet_conv1, RefineNet_conv2, and RefineNet_conv3, for feature extraction and learning. Next, depth residual prediction is performed using RefineNet_res to obtain a residual map representing the depth improvement. Finally, the initial depth estimate is added to the residual map to obtain the final depth estimate result, depth_refined, which is returned to achieve a more accurate depth estimate, completing the depth estimation of the reference view.
[0056] By following the steps above, depth estimation is performed on all views to obtain a depth map. Then, the depth maps are fused to generate a spatial point cloud 3D representation of the scene.
[0057] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the invention and is not intended to limit the invention. Although the invention has been described in detail, those skilled in the art can still modify the technical solutions described in the examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention are included within the scope of protection of the invention.
Claims
1. A method for multi-view stereo 3D reconstruction based on path-aggregated deep learning, characterized in that, The method comprises the following steps: Step 1: set the to-be-solved depth map view as a reference image and select n source images, and determine the corresponding camera internal and external parameters; Step 2: the reference image and several source images are respectively subjected to the same path aggregation feature extraction network to obtain multi-scale feature maps of each image; The path aggregation feature extraction network is composed of two bottom-up and one top-down convolutional neural networks, three deformable convolution layers and three convolution block attention modules; the input image is first subjected to a bottom-up structure, which is composed of three residual convolution blocks respectively; then subjected to a top-down structure, which is composed of three residual blocks; and the previous bottom-up structure is connected through a convolutional neural network with a convolution kernel of 1x1 as a horizontal connection to fuse high-level features and low-level features; followed by the last bottom-up network structure, which embodies the path aggregation effect, and the three levels from bottom to top are named N1, N2 and N3 respectively; N1 is directly obtained from the last layer of the previous top-down path, first has N1 pass through residual blocks with two steps of 2 and 1, and then adds the feature maps obtained to the same layer feature maps in the previous top-down path to realize horizontal connection, and then passes through a residual block with a step of 1 to obtain N2; the output of N2 is added to the output of the topmost layer of the previous top-down path through horizontal connection, and is taken as input to N3; thus, three feature maps of different scales are obtained, and the final multi-scale feature maps are extracted through different deformable convolution layers and convolution block attention modules; Step 3: according to the feature map-based homography transformation algorithm, n groups of feature bodies are obtained from the feature maps of the reference image and the source images respectively, and the feature bodies are constructed into cost volumes based on the variance method; Step 4: the constructed cost volume is subjected to cost volume regularization through the 3D-UNet network structure to obtain a probability volume, the probability volume is subjected to softmax operation along the first dimension, and expectation operation is performed to obtain the depth map of the final reference image, and the depth map is subjected to refinement through a refinement network; Step 5: steps 1-4 are repeated for all multi-view images to obtain the depth map of each image, and the point cloud expression of the three-dimensional model is obtained through depth map fusion to realize three-dimensional reconstruction of the target scene.
2. The method of claim 1, wherein the method further comprises: The feature maps obtained in step 2 are multi-scale feature maps at three scales, and steps 3-4 are performed at three scales to realize the construction of the cascade cost volume from coarse to fine; for the cascade structure of the cost volume construction method, the specific steps are as follows: The number of depth sampling planes at each stage is preset as 48, 32 and 8; for the first stage, the depth sampling planes are selected as 48 depth planes uniformly sampled in a given depth range; After the first stage depth prediction is completed, 16 depth planes above and below the first stage depth prediction are sampled to meet the 32 sampling planes; after the second stage depth prediction is completed, the third stage depth sampling plane is determined as the second stage, and 4 depth planes above and below the second stage depth prediction value are sampled to meet the 8 depth sampling planes; The cost volume is constructed in a cascaded manner, and the resolution of the predicted depth map is 1 / 4 of the input view in the first stage, 1 / 2 in the second stage, and the same as the reference view in the third stage.
3. The method of claim 1, wherein the method further comprises: The homographic transformation of the feature map in step 3 is performed according to the input reference image, the camera internal and external parameters corresponding to the source image, and a preset depth interval, and the formula is as follows: where K i , R i , t i , are the camera intrinsic, rotation and translation of the ith view, I is an identity matrix, i is 0, which refers to the reference view, and n1 represents the principal axis of the reference camera; and when it is applied to the cascade structure of the network, the homographic transformation formula of the k+1 stage is: represents the predicted depth of the mth pixel point at the kth stage, represents the residual disparity of the mth pixel to be learned at the k+1th stage.
4. The method of claim 1, wherein the method further comprises: The homographic transformation of the feature map in step 3 is performed according to the input reference image, the camera internal and external parameters corresponding to the source image, and a preset depth interval, and the formula is as follows: Step 4.1: preset the number of depth samples in each stage as 48, 32 and 8; Step 4.2: calculate the homography matrix H of the reference view and each source view according to formula (2); K i ,R i ,t i ,are the camera intrinsic, rotation and translation of the ith view, I is an identity matrix, i is 0 referring to the reference view, n1 represents the principal axis of the reference camera, represents the predicted depth of the mth pixel point at the kth stage, represents the residual disparity of the mth pixel to be learned at the k+1th stage; Step 4.3: Projecting the source view feature map F i obtained from step 2 to each of the preset depth planes by means of the homography matrix obtained from step 4.2 to obtain the perspective-transformed feature volume F i ′: F i ′=H i (d)F i (3) Step 4.4: A number of feature bodies F i are aggregated into one cost body by the following variance-based formula, where denotes the average value of the feature bodies: Step 4.5: perform steps 4.1-4.2 in the three stages of the cascaded structure.
5. The method of claim 1, wherein the method further comprises: The cost volume regularization module in step 4 is as follows: For the three stages of the cascaded structure, the cost volume regularization modules are independent of each other and are not shared, and each cost volume regularization module is a 3D-Unet structure, specifically: The input cost volume is first sequentially passed through conv0, conv1, conv2, conv3, conv4, conv5 and conv6, which are 3D convolution modules, each of which includes a 3D convolution layer, a batch normalization layer and a Relu layer; then the output of the conv6 layer is passed through conv7; the output of conv7 is added to the output of conv4 and input to conv8; the output of conv8 is added to the output of conv2 and input to conv9, and the output of conv9 is added to the output of conv0, and then passed through an outlay layer to obtain the final probability body P; wherein conv7, conv8 and conv9 are 3D deconvolution modules, each of which includes a deconvolution layer, a batch normalization layer and a Relu layer.
6. The method of claim 1, wherein the method further comprises: The depth map in step 4 is refined by a refinement network, specifically: First, the input image and depth information are spliced along the channel dimension, and then the spliced result is passed to a series of convolution layers RefineNet_conv1, RefineNet_conv2 and RefineNet_conv3 for feature extraction and learning; next, a residual map representing depth improvement is obtained through RefineNet_res for depth residual prediction; finally, the initial depth estimation and the residual map are added to obtain the final depth estimation result depth_refined, which is returned to obtain a more accurate depth estimation result.