Multi-view three-dimensional reconstruction method based on feature enhancement

Through the method of feature enhancement and adaptive cost volume generation, the existing multi-view 3D reconstruction method has solved the problems of large computing volume and high memory consumption, and achieved high fidelity feature extraction and accurate 3D reconstruction.

CN119006714APending Publication Date: 2024-11-22NORTHEASTERN UNIV AT QINHUANGDAO
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411095971.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing multi-view 3D reconstruction method has high hardware requirements in the cost-based regularization stage, resulting in large amounts of computing and high memory consumption.

Method used

Using a multi-view three-dimensional reconstruction method based on feature enhancement, multi-scale features are extracted through the bottom-up feature extraction module and the top-down feature aggregation part, combining Hal wavelet downsampling and pixel attention mechanism to generate high-fidelity feature maps , and use GRU iterative optimizer and weighted network to generate adaptive cost bodies to reduce the computational volume.

Benefits of technology

It effectively reduces video memory consumption, improves the fidelity and integrity of feature extraction, enhances the accuracy of pixel matching, and improves reconstruction accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006714A_ABST
    Figure CN119006714A_ABST
Patent Text Reader

Abstract

The invention provides a multi-view three-dimensional reconstruction method based on feature enhancement, and the method comprises the steps: obtaining a plurality of images of a to-be-reconstructed object at different view angles and a camera pose, extracting a multi-scale feature map through a feature map extraction network, and obtaining a three-dimensional cost body through homography transformation and variance-based cost measurement; the three-dimensional cost body passes through a weight network to obtain an adaptive cost body, and initial depth map prediction is carried out through a lightweight 3DCNN; the adaptive cost body and the depth map are connected through a two-dimensional convolutional layer, the adaptive cost body and the depth map are connected with context features to form a dynamic cost body, the dynamic cost body is input into a gate control loop unit GRU, and an updated depth map is obtained; and through a fusion algorithm based on visibility, fusing the depth maps of different visual angles and projecting the fused depth maps into a 3D space to obtain three-dimensional point cloud data. According to the method, the parameters and the calculation amount of the network can be reduced, and the reconstruction precision and the integrity are effectively improved under the condition that fewer GPU video memories are used.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a multi-view reconstruction method based on feature enhancement. Figure 3 Dimensional reconstruction method. Background Art

[0002] 3D reconstruction technology is ubiquitous in modern scientific and technological applications. With the rapid development of computer vision and graphics technology, 3D reconstruction has become a hot topic in scientific research and engineering applications. 3D reconstruction converts 2D image data into 3D models. Its core lies in using computer vision and graphics algorithms to extract spatial information from multi-view 2D images and construct accurate 3D models. In the medical field, images are used to achieve detailed pathological analysis and surgical planning; in archaeology, historical sites and cultural relics are restored to help protect cultural heritage; in architecture and engineering, accurate building models are generated to improve design and construction efficiency; in film and television production, realistic virtual scenes and characters are created; in robot navigation, environmental perception is provided to enhance autonomous navigation capabilities; in geographic information systems, detailed terrain and city models are constructed to support urban planning and environmental monitoring. 3D reconstruction has not only improved the accuracy and efficiency of work in various fields, but also promoted the development of virtual reality and augmented reality technologies, becoming an important support for modern scientific and technological innovation.

[0003] Traditional 3D reconstruction methods mostly rely on expensive hardware devices, such as laser scanners or structured light sensors, which are not only costly but also complex to operate. With the rapid development of computer vision technology, image-based 3D reconstruction methods have gradually become a research hotspot. The rise of MVSNet (Multi-View Stereo Matching) networks has made multi-view stereo matching a Figure 3 The 3D reconstruction technology has been further developed. By fusing multiple images taken from different angles, the 3D structure of the target object can be reconstructed, which has the advantages of low cost and high flexibility. Later, a series of 3D reconstruction networks for multi-view stereo matching were derived, but because 3D convolution is used in the cost volume regularization stage, the hardware requirements are relatively high. Summary of the invention

[0004] Aiming at the shortcomings of the existing technology, this paper proposes a multi-view Figure 3 The 3D reconstruction method can effectively reduce the consumption of video memory.

[0005] A multi-view feature enhancement Figure 3 The reconstruction method comprises the following steps:

[0006] S1. Establish a feature map extraction network, extract multi-scale features from image data through feature extraction and feature fusion networks, output feature maps of three scales, with resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image respectively, and extract multi-scale context features for the reference image at the same time.

[0007] The feature extraction network consists of two parts, the first part is a bottom-up feature extraction module, and the second part is a top-down multi-scale feature aggregation part.

[0008] The bottom-up feature extraction module uses a two-dimensional convolutional layer to extract feature maps and a downsampling module based on Haar wavelet to perform high-fidelity downscaling transformation on feature maps. First, the convolutional layer is used to extract feature maps with 8 channels and the resolution of the original image size from N H×W images. Then, the feature maps are decomposed into two parts: low-frequency and high-frequency details. H The high-frequency components in different directions in the L It is concatenated with the high-frequency components obtained previously to achieve lossless encoding of features. Finally, a convolution layer, a batch normalization layer, and a ReLU activation layer are used to obtain the final feature map. The downsampled feature map is downsampled to half of the original with high fidelity, and the number of channels is doubled. Repeat the above steps, with changes in the parameter settings and input and output of the convolution layer. Finally, feature maps with 8, 16, and 32 channels and resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image are obtained.

[0009] The top-down feature aggregation part is mainly completed by the collaboration of the convolution layer and the pixel attention guidance module to complete the feature fusion. First, the output feature map of each layer must first pass through a 1×1 convolution kernel to make the number of channels of each feature map the same, and the image resolution remains unchanged. Then, the features of different levels are fused through the pixel attention guidance module. First, the dot product of the pixel feature vectors corresponding to the upper and lower feature maps is calculated and a possibility value σ is obtained through the activation function, indicating the degree to which the two pixels may belong to the same object. If σ is higher, it is considered that the features provided in the high-level feature map are more valuable for reference, and vice versa. The final output fusion result is the dynamic fusion of the pixel features of the two feature maps according to the σ weight coefficient.

[0010] S2. Establish a cost volume generation network, transform the feature map of the source image through homography, construct a feature volume, and then use the variance-based cost metric to generate a primary cost volume.

[0011] First, establish a depth hypothesis plane with a depth hypothesis range of d min to d max , for each pixel p at stage k and iteration t, in the inverse depth range R k Internal uniform sampling D k Deep hypothesis:

[0012]

[0013] Among them, I m represents the minimum depth assumption plane interval, is the depth value of pixel p after iteration t in stage k, d min is the minimum depth value, d max is the maximum depth value.

[0014] The homography transformation is based on the extracted feature map and camera parameters, and projects the feature map of the source image onto several parallel planes under the reference image. The coordinate transformation formula on the feature map is:

[0015] x′~H i (d)·x

[0016] Among them, ~ represents the projection equation, x represents a point at the depth value d on the original feature map of the source image, x′ represents the point after the projection transformation of x, and H i (d) represents the homography transformation matrix between the source image feature map and the reference image feature map at depth value d, and the formula is as follows:

[0017]

[0018] Where n1 is the main axis direction of the reference camera, d represents the uniform sampling value calculated from the predicted depth range, Represents the camera intrinsic parameter matrix, rotation matrix, and translation vector corresponding to each view.

[0019] The variance-based cost metric is used to measure the similarity between N views and can be adapted to any number of input views. The cost metric defines the mapping relationship:

[0020]

[0021] Among them, W, H, D, and F are the input image width, height, number of depth samples, and number of channels of the feature map, respectively. is the size of the feature volume, and C is the obtained cost volume.

[0022] S3. Introduce a weighted network in the depth dimension, perform weighted averaging on each voxel of the depth layer hypothesis d, and combine the weight map with the cost volume to generate an adaptive cost volume. The specific formula is as follows:

[0023]

[0024] Where C′ represents the adaptive cost volume, ω h,w represents the weight, and C represents the primary cost body.

[0025] Then, two 2D convolutional layers are used to extract geometric and depth features from the adaptive cost volume and depth map, and the output is concatenated with the context features to construct the final dynamic cost volume.

[0026] S4. Initial depth map generation. First, a lightweight 3D CNN is used to regularize the cost volume to obtain the probability volume P corresponding to each depth hypothesis d. d , the initial depth map D is obtained by weighted averaging the probability volume init , the formula is as follows:

[0027]

[0028] S5. The GRU (Gated Recurrent Unit) based iterative optimizer updates the depth map and iteratively processes the dynamic cost volume. In each iteration t, the GRU inputs the dynamic cost volume C t-1 and the potential hidden state h t-1 , using a deep head module consisting of two convolutional layers and a tanh activation function to predict the residual depth Δd t , using formula D t =D t-1 +Δd t Update the input depth map and upsample it to D t+1 As the input of the next iteration. After the last iteration of each stage, the updated depth map is upsampled using the mask upsampling module. In addition, at each stage, the initial hidden state h0 is initialized by the context feature network, and the dynamic cost body C t-1 Through the previous depth map D t+1 to update.

[0029] The loss of the gated recurrent unit-based depth map iterator is as follows:

[0030] The depth maps output from the initial depth prediction module and the GRU-based multi-level optimization module at different iterations. The L on the output depth map is calculated using the true depth map of the corresponding resolution. 0.5 Loss. The final loss is the weighted sum of all losses:

[0031]

[0032] Among them, L init L is the initial depth map obtained by the initial depth prediction module 0.5 Loss. k is the number of optimization iterations in stage k, T k The loss of the output depth map and the k-stage upsampled depth map, is the corresponding weight.

[0033] l 0.5 The norm and its gradient formula can be expressed as:

[0034]

[0035] Applying this norm to the photometric consistency loss turns common points into accurate points and reduces the focus on bad points, L 0.5 The photometric consistency loss is:

[0036]

[0037] Among them, P valid is the effective real pixel set, d(p) is the real depth value of pixel p, is the depth estimate of the network.

[0038] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0039] (1) A multi-view feature enhancement method provided by the present invention Figure 3 The dimensional reconstruction method uses a lightweight network to reduce the consumption of video memory in order to solve the problem of excessive computational complexity of existing high-performance networks.

[0040] (2) A multi-view based feature enhancement method provided by the present invention Figure 3 The proposed method uses a downsampling module based on Haar wavelet transform and a feature fusion module based on pixel attention mechanism to obtain a high-fidelity and high-integrity feature extraction module, which can reduce the image size while ensuring that information is not lost, and adaptively fuses features effectively to enhance the features.

[0041] (3) A multi-view based feature enhancement method provided by the present invention Figure 3 The method uses an adaptive cost volume generation structure to learn the influence of images of each view in multiple matching volumes, enhance the parts with better pixel matching, and suppress the parts with failed matching.

[0042] (4) A multi-view based feature enhancement method provided by the present invention Figure 3 dimensional reconstruction method, using L 0.5 Photometric consistency loss makes the model pay more attention to accuracy during training and improves the accuracy of reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a diagram of the overall structure of the network reconstructed in an example of the present invention.

[0044] Figure 2 This is a structural diagram of the feature enhancement module in an example of the present invention.

[0045] Figure 3 This is an overall structural diagram of updating the depth map in an example of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0047] A multi-view feature enhancement Figure 3 Dimensional reconstruction methods, such as Figure 1 As shown, the following steps are included:

[0048] The first step is to obtain multi-view images of the object to be reconstructed. Use a camera or other device to take multiple images of the object from different angles, and use the motion recovery structure algorithm to obtain the camera's pose information, including the camera's internal and external parameters and the matching information between views. Select the first image as the reference image, and then find the N images closest to it based on the camera pose information as the source image of the object to be reconstructed.

[0049] The second step is to extract the multi-scale feature map of the image. The image is input into the feature map extraction module to extract the multi-scale feature map. The network model framework is as follows Figure 2 As shown. The image input passes through multiple layers of two-dimensional convolutional layers and downsampling modules based on Haar wavelet transform to extract multi-scale features with 16, 32, and 64 channels, and sizes of 1 / 2, 1 / 4, and 1 / 8 of the original image. The three-scale features are then passed through convolutional layers to change the number of channels to 64. The pixel attention guidance module is used to adaptively fuse the features into the previous layer in turn, and the two-dimensional convolution transform is used to return the original number of channels to obtain feature maps with 8, 16, and 32 channels and resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image. This is used as a pyramid model from high resolution to low resolution, and the reference image is then input into the feature map extraction module to obtain contextual features.

[0050] The feature extraction network consists of two parts: the first part is the bottom-up feature extraction module, and the second part is the top-down multi-scale feature aggregation part.

[0051] The bottom-up feature extraction module uses a two-dimensional convolutional layer to extract feature maps and a downsampling module based on Haar wavelet to perform high-fidelity downscaling transformation on feature maps. First, the convolutional layer is used to extract feature maps with 8 channels and the resolution of the original image size from N H×W images. Then, the feature maps are decomposed into two parts: low-frequency and high-frequency details. H The high-frequency components in different directions in the LIt is concatenated with the high-frequency components obtained previously to achieve lossless encoding of features. Finally, a convolution layer, a batch normalization layer, and a ReLU activation layer are used to obtain the final feature map. The downsampled feature map is downsampled to half of the original with high fidelity, and the number of channels is doubled. Repeat the above steps, with changes in the parameter settings and input and output of the convolution layer. Finally, feature maps with 8, 16, and 32 channels and resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image are obtained.

[0052] The top-down feature aggregation part is mainly completed by the collaboration of the convolution layer and the pixel attention guidance module to complete the feature fusion. First, the output feature map of each layer must first pass through a 1×1 convolution kernel to make the number of channels of each feature map the same, and the image resolution remains unchanged. Then, the features of different levels are fused through the pixel attention guidance module. First, the dot product of the pixel feature vectors corresponding to the upper and lower feature maps is calculated and a possibility value σ is obtained through the activation function, indicating the degree to which the two pixels may belong to the same object. If σ is higher, it is considered that the features provided in the high-level feature map are more valuable for reference, and vice versa. The final output fusion result is the dynamic fusion of the pixel features of the two feature maps according to the σ weight coefficient.

[0053] The third step is initial depth estimation. The feature map of the source image is transformed by homography using the cost volume generation network, and the feature map under the perspective of the source image is converted into the feature cost under the perspective of the reference image.

[0054] The homography transformation is based on the extracted feature map and camera parameters, projecting the feature map of the source image onto several parallel planes under the reference image. The coordinate transformation formula on the feature map is:

[0055] x′~H i (d)·x

[0056] Among them, ~ represents the projection equation, x represents a point at the depth value d on the original feature map of the source image, x′ represents the point after the projection transformation of x, and H i (d) represents the homography transformation matrix between the source image feature map and the reference image feature map at depth value d, and the formula is as follows:

[0057]

[0058] Where n1 is the main axis direction of the reference camera, d represents the uniform sampling value calculated from the predicted depth range, Represents the camera intrinsic parameter matrix, rotation matrix, and translation vector corresponding to each view.

[0059] Then, the variance-based cost metric is used to obtain the initial three-dimensional cost volume. The cost volume is regularized using a lightweight 3D CNN to obtain the probability volume P corresponding to each depth hypothesis d. d, the initial depth map is obtained by weighted averaging the probability volumes.

[0060] The variance-based cost metric is used to measure the similarity between N views and can be adapted to any number of input views. The cost metric defines the mapping relationship:

[0061]

[0062] Among them, W, H, D, and F are the input image width, height, number of depth samples, and number of channels of the feature map, respectively. is the size of the feature volume, and C is the obtained cost volume.

[0063] The fourth step is to update the iterative depth map.

[0064] First, a weighted network is introduced in the depth dimension, and each voxel of the depth layer hypothesis d is weighted averaged. The weight map is combined with the cost volume to generate an adaptive cost volume. Two 2D convolutional layers are used to extract geometric features and depth features from the adaptive cost volume and depth map. The 2D convolutional layer is used to process the concatenation of geometric features and depth features, canceling the depth dimension and connecting the cost map along the channel dimension. Then, the output is concatenated with the context feature to construct the final dynamic cost volume.

[0065] Then, the dynamic cost volume is iteratively processed using the GRU (Gated Recurrent Unit) structure to obtain the depth map residual and update the original predicted depth map, such as Figure 3 As shown. In each iteration t, GRU inputs the dynamic cost body C t-1 and the potential hidden state h t-1 , using a deep head module consisting of two convolutional layers and a tanh activation function to predict the residual depth Δd t , using formula D t =D t-1 +Δd t Update the input depth map and upsample it to D t+1 As the input of the next iteration. After the last iteration of each stage, the updated depth map is upsampled using the mask upsampling module. In addition, at each stage, the initial hidden state h0 is initialized by the context feature network, and the dynamic cost body C t-1 Through the previous depth map D t+1 to update.

[0066] The Gated Recurrent Unit (GRU) has two main gating units: the reset gate and the update gate. The reset gate determines the hidden state h of the previous time step. t-1 How much information needs to be ignored and the current input How to combine with this retained information to generate candidate hidden states

[0067]

[0068] Among them, σ is the activation function, W r are the parameters of the convolutional network, Indicates h t-1 and concatenation of, tanh is the hyperbolic tangent activation function, W h are the parameters of the hidden convolutional network.

[0069] The update gate determines the hidden state h at the previous time step t-1 and candidate hidden states How to combine, by updating the gate z t Control history information t-1 and candidate information The mixing ratio generates the hidden state h of the current time step t .

[0070]

[0071] Among them, W z is the weight matrix of the update gate.

[0072] The loss function of the depth map iterator based on the gated recurrent unit is as follows:

[0073] The depth maps output from the initial depth prediction module and the GRU-based multi-level optimization module at different iterations. The L on the output depth map is calculated using the true depth map of the corresponding resolution. 0.5 Loss. The final loss is the weighted sum of all losses:

[0074]

[0075] Among them, L init L is the initial depth map obtained by the initial depth prediction module 0.5 Loss. k is the number of optimization iterations in stage k, T k The loss of the output depth map and the k-stage upsampled depth map, is the corresponding weight.

[0076] l 0.5 The norm and its gradient formula can be expressed as:

[0077]

[0078] Applying this norm to the photometric consistency loss turns common points into accurate points and reduces the focus on bad points, L 0.5 The photometric consistency loss is:

[0079]

[0080] Among them, P valid is the effective real pixel set, d(p) is the real depth value of pixel p, is the depth estimate of the network.

[0081] In the fifth step, depth maps from different perspectives are unified into a point cloud representation in three-dimensional space using depth map fusion. d Upsampling is performed as a confidence measure for each pixel. Then, a visibility-based fusion algorithm is used in reconstruction to minimize the depth occlusion between different viewpoints, discarding pixels with an estimated depth probability less than 0.3, and then directly reprojecting the fused depth map into space to generate a point cloud in three-dimensional space.

[0082] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A multi-view 3D reconstruction method based on feature enhancement, characterized in that: The following steps are involved: S1. Obtain multiple images of the object to be reconstructed from different perspectives, obtain the camera pose from the image using the motion recovery structure algorithm, and divide the image into a reference image and a source image; S2, extract multi-scale feature maps from all images through the feature map extraction network, extract context features from the reference image, use homography transformation to obtain multi-view feature volumes, and then obtain a three-dimensional cost volume through a variance-based cost metric; S3, the three-dimensional cost volume is passed through the weight network to obtain an adaptive cost volume. In the initial stage, a lightweight 3DCNN is used to regularize the cost volume to obtain the probability volume corresponding to each depth hypothesis d, and the initial depth map is obtained by weighted average with the depth value; S4, update the iterative depth map, use the two-dimensional convolutional layer to connect the features of the adaptive cost body and the depth map, connect the output with the context features to form a dynamic cost body, input the dynamic cost body into the gated recurrent unit GRU, update iteratively, and obtain the updated depth map; S5. Obtain a three-dimensional reconstructed model. Through a visibility-based fusion algorithm, the depth maps of different perspectives are fused into a unified point cloud representation, and the fused depth map is projected into a 3D space to obtain dense three-dimensional point cloud data and obtain a three-dimensional reconstructed model.

2. The multi-view 3D reconstruction method based on feature enhancement according to claim 1, characterized in that: The feature map extraction network in step S2 is as follows: The multi-scale features in the image data are extracted through the feature map extraction network, and feature maps of three scales are output, with resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image respectively. At the same time, multi-scale context features are extracted for the reference image. The module consists of two parts: the first part is the bottom-up feature extraction module, and the second part is the top-down multi-scale feature aggregation part.

3. The multi-view 3D reconstruction method based on feature enhancement according to claim 2, characterized in that: The specific structure of the bottom-up feature extraction module is as follows: First, a convolutional layer is used to extract a feature map with 8 channels and the resolution of the original image from N H×W images. After a downsampling module based on Haar wavelet transform, the feature map is decomposed into two parts: low frequency and detail high frequency. The high frequency components in different directions of the high frequency are selected, and then the low frequency components are spliced ​​with the high frequency components obtained previously to complete the lossless encoding of the features. Then, a convolutional layer, a batch normalization layer, and a ReLU activation layer are passed to obtain the final feature map. After downsampling, the feature map is downsampled to half of the original with high fidelity, and the number of channels is doubled. The above steps are repeated to obtain feature maps with 8, 16, and 32 channels and resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image. The specific structure of the top-down feature aggregation module is as follows: First, the output feature map of each layer must first pass through a 1×1 convolution kernel to make the number of channels of each feature map the same, while keeping the image resolution unchanged. Then, the features of different levels are fused through the pixel attention guidance module. First, the dot product of the pixel feature vectors corresponding to the upper and lower feature maps is calculated and passed through an activation function to obtain a possibility value σ, which indicates the degree to which the two pixels may belong to the same object. If σ is higher, it is believed that the features provided in the high-level feature map are more valuable for reference, and vice versa. The final output fusion result dynamically fuses the pixel features of the two feature maps according to the σ weight coefficient.

4. The multi-view 3D reconstruction method based on feature enhancement according to claim 2, characterized in that: The specific process of generating the cost volume in step S2 is as follows: First, establish a depth hypothesis plane with a depth hypothesis range of d min to d max , for each pixel p at stage k and iteration t, in the inverse depth range R k Internal uniform sampling D k Deep hypothesis: Among them, I m represents the minimum depth assumption plane interval, is the depth value of pixel p after iteration t in stage k, d min is the minimum depth value, d max is the maximum depth value; The homography transformation is based on the extracted feature map and camera parameters, and projects the feature map of the source image onto several parallel planes under the reference image. The coordinate transformation formula on the feature map is: x′~H i (d)·x Where, ~ represents the projection equation, x is the eigenvalue corresponding to the depth value d in the source image coordinates, x′ represents the point after the projection transformation of x, and H i (d) represents the homography transformation matrix between the source image feature map and the reference image feature map at depth value d, and the formula is as follows: Where n1 is the main axis direction of the reference camera, d represents the uniform sampling value calculated from the predicted depth range, Represents the camera intrinsic parameter matrix, rotation matrix and translation vector corresponding to each view; The variance-based cost metric is used to measure the similarity between N views and can be adapted to any number of input views. The specific formula is as follows: Among them, W, H, D, and F are the input image width, height, number of depth samples, and number of channels of the feature map, respectively. is the size of the feature volume, and C is the obtained cost volume.

5. The multi-view 3D reconstruction method based on feature enhancement according to claim 1, characterized in that: The specific process of generating the adaptive cost volume in step S3 is as follows: A weighted network is introduced in the depth dimension, and each voxel of the depth layer hypothesis d is weighted averaged. The weight map is combined with the cost volume to generate an adaptive cost volume. The specific formula is as follows: Where C′ represents the adaptive cost volume, ω h,w represents the weight, C represents the primary cost body; Then, two 2D convolutional layers are used to extract geometric and depth features from the adaptive cost volume and depth map, and the output is concatenated with the context features to construct the final dynamic cost volume.

6. The multi-view 3D reconstruction method based on feature enhancement according to claim 1, characterized in that: The specific process of generating the initial depth map in step S3 is as follows: First, a lightweight 3D CNN is used to regularize the cost volume to obtain the probability volume P corresponding to each depth hypothesis d d , the initial depth map is obtained by weighted averaging the probability volume, the formula is as follows: The probability volume obtained by the initial depth prediction module is upsampled and used as the confidence measure for each pixel, discarding pixels with estimated depth probability less than 0.

3.

7. The multi-view 3D reconstruction method based on feature enhancement according to claim 1, characterized in that: The specific process of updating the depth map in step S4 is as follows: In each iteration t, the GRU (Gated Recurrent Unit) inputs the dynamic cost volume C t-1 and the potential hidden state h t-1 , using a deep head module consisting of two convolutional layers and a tanh activation function to predict the residual depth Δd t , using formula D t =D t-1 +Δd t Update the input depth map and upsample it to D t+1 As the input of the next iteration, after the last iteration of each stage, the updated depth map is upsampled using the mask upsampling module; in addition, at each stage, the initial hidden state h0 is initialized by the context feature network, and the dynamic cost body C t-1 Through the previous depth map D t+1 Make updates; The loss function of the depth map update part is as follows: Use L between the estimated depth value and the actual depth value 0.5 As the loss function, the depth maps output from the initial depth prediction module and the GRU-based multi-level optimization module at different iterations are used to calculate L on the output depth maps using the true depth maps of the corresponding resolutions. 0.5 Loss, the final loss is the weighted sum of all losses, and the formula of the loss function is as follows: Among them, L init L is the initial depth map obtained by the initial depth prediction module 0.5 Loss, T k is the number of optimization iterations in stage k, T k The loss of the output depth map and the k-stage upsampled depth map, is the corresponding weight; L 0.5 The norm and its gradient formula can be expressed as: Applying this norm to the photometric consistency loss turns common points into accurate points and reduces the focus on bad points, L 0.5 The photometric consistency loss is: Among them, P valid is the effective real pixel set, d(p) is the real depth value of pixel p, is the depth estimate of the network.

Citation Information

Cited By

  • Three-dimensional reconstruction method and device, electronic equipment, storage medium and program product

    CN119540470A

  • Multi-view three-dimensional reconstruction method and system based on GRU and 3DCNN

    CN119600206A

  • Human body depth estimation method and device, electronic equipment and storage medium

    CN119991764A

  • Data processing method for ocean optical cable communication

    CN120812099A

  • A data processing method for marine optical cable communication

    CN120812099B