Dynamic self-adaptive multi-view three-dimensional reconstruction method and model based on double-domain information fusion

Abstract

The invention relates to a three-dimensional reconstruction method and model, in particular to a dynamic self-adaptive multi-view three-dimensional reconstruction method and model based on double-domain information fusion, and aims to solve the problems that in the prior art, the space relation between pixels is explored by utilizing attention along a channel dimension, the self-adaptive long-range information association capability is lacked, and the space relation between pixels is not explored. According to the dynamic adaptive multi-view three-dimensional reconstruction method based on double-domain information fusion, spatial domain detail enhancement and frequency domain dynamic aggregation and enhancement are realized through a multi-scale spatial domain and frequency domain information fusion network, so that multi-scale enhancement characteristics of an input image are obtained; a multi-scale depth prediction network is constructed by adopting a coarse-to-fine framework, depth hypotheses of all scales are obtained, a cost body is generated through adaptive aggregation, finally regularization is carried out to obtain prediction depth, and the prediction depth is converted to obtain a three-dimensional point cloud file of a corresponding scene. Meanwhile, the invention provides a model capable of realizing the method.

Description

Technical Field

[0001] The present invention relates to a three-dimensional reconstruction method and model, and in particular to a dynamic adaptive multi-view stereoscopic reconstruction method and model based on dual-domain information fusion. Background Art

[0002] Multi-view stereo reconstruction is a core task in the field of computer vision. It aims to recover and reconstruct a dense geometric representation of a scene from multiple overlapping perspectives. Its application scope includes autonomous driving, virtual reality, augmented reality, cultural relics protection, etc. It has received extensive attention and research from relevant scholars in the past few decades.

[0003] Multi-view stereo (MVS) methods use hand-crafted feature representations for matching, and therefore face challenges in areas with repeated patterns, weak textures, drastic lighting changes, and reflections. To compensate for the weak representation of MVS methods, learning-based methods have emerged, which replace hand-crafted similarity metrics and cost volume aggregation modules with deep learning-based feature extraction and 3D regularization networks, and thus show excellent performance. In particular, learning-based MVS methods can better introduce global semantic information, such as specular and reflection priors, to provide support for subsequent robust feature matching and depth generation.

[0004] However, existing methods only mine the feature representation of the input image in the spatial domain, and have limited processing capabilities for complex patterns such as rotation and scaling. They are prone to losing high-frequency details, and are particularly difficult to extract robust features from reflective surfaces, weak textures, and textureless areas. In addition, considering computational efficiency and memory limitations, existing methods usually use fixed-range attention or only use attention along the channel dimension to explore the spatial connections between pixels, lacking the ability to adaptively associate long-range information, resulting in reduced image reconstruction performance. Therefore, how to effectively utilize the advantages of frequency domain processing in capturing high-frequency details and complex structures, as well as exploring long-range spatial dependencies in the process of deep propagation, is crucial to improving the reconstruction quality of the scene. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art that the attention along the channel dimension is used to explore the spatial connection between pixels, but the ability to adaptively associate long-range information is lacking, resulting in reduced image reconstruction performance, and to provide a dynamic adaptive multi-view stereo reconstruction method and model based on dual-domain information fusion.

[0006] The inventive concept of the present invention is as follows: In the dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion of the present invention, first, a multi-view image of the target scene and the internal and external parameters of the camera are obtained, and the multi-view image is input, and the multi-view image is divided into a reference image and multiple source images. Next, a multi-scale spatial domain and frequency domain information fusion network is constructed to achieve spatial domain detail enhancement and frequency domain dynamic aggregation and enhancement to obtain multi-scale enhanced features of the input image. Then, a multi-scale depth prediction network is constructed using a coarse-to-fine framework. For the lowest-scale features, a depth hypothesis is obtained by uniform sampling along the inverse depth range; for the features of the remaining scales, an uncertainty hypothesis mechanism based on dynamic range multi-scale adaptive propagation is constructed, and the depth hypothesis of each scale is obtained through the dynamic range multi-scale adaptive propagation mechanism. Using the internal and external parameters of the camera, the multi-scale features are warped to the front parallel plane of the reference camera cone through a differentiable homography transformation, and the reference features and the source features are adaptively aggregated using an adaptive group correlation strategy to generate a cost volume. Finally, the cost volume is regularized using 2D regularization compensated by the probability volume geometry prior to obtain the predicted depth, which is finally converted to obtain the 3D point cloud file of the corresponding scene. In the process of building a model that can implement the above method, the network training is guided by the L1 loss at each stage. This method is accurate, flexible and robust, and provides effective technical support for multi-view stereo matching reconstruction tasks.

[0007] To achieve the above objectives, the technical solutions provided by the present invention are as follows: A dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion, which is special in that it includes the following steps: S1, input N multi-view images of the same scene to obtain multi-scale features ,in, Indicates scale, , Indicates the sequence number of the multi-view image, ranging from 0 to N-1. One of the N multi-view images is used as the reference image. represents the reference image, and the remaining N-1 images are used as source images; S2, for multi-scale features Perform spatial detail enhancement and output the features corresponding to each multi-view image ; For multi-scale features Perform dynamic aggregation and enhancement in the frequency domain and output the features corresponding to each multi-view image ; S3, the corresponding features With features Fusion gets fusion features ,Will Upsampling and corresponding multi-scale features Add together to get multi-scale enhanced features ; S4, determine the probability volume of the reference image at scale k=0 and its corresponding predicted depth ; S5, determine the predicted depth at scales k=1,2,3 ; Prediction depth of corresponding scale Specifically obtained through the following methods: 1) Based on multi-scale enhanced features , using the uncertainty hypothesis mechanism based on dynamic range multi-scale adaptive propagation to determine the corresponding depth hypothesis; 2) Use the depth hypothesis to enhance the features corresponding to the current scale Perform differentiable homography transformation to obtain the transformed feature volume , apply the adaptive group correlation strategy to aggregate the transformed feature volume to obtain the cost volume at the current scale; 3) Use 2D regularization to process the cost volume of the current scale to obtain the corresponding probability volume ; For probability body The weighted sum along the depth direction is used to obtain the predicted depth of the current scale. ; S6, predict the depth at scale k=3 Convert it into a three-dimensional point cloud file of the corresponding scene, and complete dynamic adaptive multi-view stereo reconstruction based on dual-domain information fusion.

[0008] Furthermore, step S2 is specifically as follows: S2.1. Multi-scale features Perform deep convolution to obtain intermediate features ; S2.2. The intermediate features Perform the split operation along the channel in proportion, and get , , ,right , Perform deep convolution to obtain convolution features and convolutional features ; S2.3. , , After cascading along the channel direction, the channel information is integrated through point-by-point convolution to obtain the features corresponding to each multi-view image. ; S2.4. Multi-scale features The features within the corresponding frequency range are adaptively weighted according to the frequency, highlighting the required frequencies and suppressing the useless frequencies, and obtaining the weighted spectrum data corresponding to each multi-view image. ; S2.5. Weighted spectral data Convert from the spatial domain to the frequency domain via Fourier transform and calculate the amplitude and Phase , and dynamically enhance them through the modulation network, and then add residual connections to obtain new amplitudes and new phase ; S2.6. Using the new amplitude and new phase Reconstruct the complex tensor and then perform inverse Fourier transform to obtain the features corresponding to each multi-view image .

[0009] Furthermore, in step S2.1, the depth convolution adopts depth convolution , where d is the expansion rate; In step S2.2, the ratio is: ;right , The specific steps for deep convolution are: , Depthwise convolution and depthwise convolution to process; In step S2.3, the point-by-point convolution is performed using Point-wise convolution; Step S2.4 is to obtain weighted spectrum data by the following formula: : ; in, is a two-dimensional convolutional layer with a Sigmoid activation function for calculating the weights of low-frequency features. is a two-dimensional convolutional layer with a Sigmoid activation function for calculating the weights of high-frequency features. is a low frequency feature, Multi-scale features A simplified representation of ; In step S2.5, the modulation network is composed of , LeakyReLU and Efficient Channel Attention The above mentioned components are dynamically enhanced by the modulation network, and then the residual connection is added to obtain the new amplitude. and new phase , specifically:

[0010] .

[0011] Furthermore, 1) in step S5 is specifically: a. respectively At the scale of Dimensionality reduction to obtain dimensionality reduction features , and then reduce the dimension features Along Axis direction and Perform sorting in the axis direction to obtain the corresponding features and , and record the indexes in the X-axis and Y-axis directions , ; In the X-axis direction and Y-axis direction, the features and Perform layer normalization and one-dimensional convolution, and then according to the index and Restore order to get features and ; The first cross attention is calculated by and the second cross attention :

[0012]

[0013] in, Indicates adjusting the shape of the tensor. is the activation function, represents matrix dot multiplication, T is transpose, , , for The corresponding QKV matrix, , , for The corresponding QKV matrix; First cross-attention and the second cross attention Fusion, and then with dimensionality reduction features Residual connection to obtain the fusion features of the reference image at the corresponding scale ; b. In the At the scale of , for each pixel in the reference image ,use The spatial correlation between pixels in the neighborhood and themselves guides the adaptive propagation of depth, and 8 depth hypotheses at the current scale are obtained; c. At the current scale, in pixels The depth sampling within the depth search range is obtained deep hypothesis, where For the The number of depth hypotheses at each scale is determined by using the Gaussian quantile function, and the following formula is used to obtain the depth hypothesis at each scale. The depth hypothesis corresponding to the depth sampling points is:

[0014] in, For the The scale A deep hypothesis, , To optimize the standard deviation, is the quantile function of the standard normal distribution, For the The predicted depth at each scale is It means the mean The standard deviation is Quantile function of the normal distribution; represents the total probability of coverage within the depth search range, and λ is a scalar parameter that controls the width of the sampling interval.

[0015] Furthermore, in step a, the convolution kernel sizes of the one-dimensional convolution are and ; Step b is as follows: for each pixel in the reference image , using dilated convolution to act on fusion features Learn to use pixels centered The 8 pixels in the neighborhood are and Offset in both directions , according to the offset , for Prediction depth at each scale Apply bilinear interpolation to obtain 8 depth hypotheses for the current scale; In step c, the pixel The depth search range is obtained by: Based on fusion features , get pixel and Normalized weights of the 8 pixels in the neighborhood , and The standard deviation of the predicted depth at each scale ,in, Pixel and The serial numbers of the 8 pixels in the neighborhood; Get the offset pixel and the standard deviation of its neighboring pixels , weighted to obtain the optimal standard deviation : ; Calculate pixels by optimizing the standard deviation The depth search range R is: ,in, is a scalar parameter that controls the width of the sampling interval, For the Prediction depth at each scale.

[0016] Furthermore, in step c, the normalized weight The specific method is as follows: After being processed by a shallow 2D weight learning network with residual connections, the pixels are learned and The weight distribution of 8 pixels in the neighborhood, and then the Softmax operation is applied to get the normalized weight ; No. The standard deviation of the predicted depth at each scale Obtained by the following formula:

[0017] in, For the The number of depth hypotheses at each scale, For the The scale A deep hypothesis, For the The depth at this scale is The probability of For the Prediction depth at each scale; The offset pixel and the standard deviation of its neighboring pixels According to the offset obtained in step b , in standard deviation It is obtained by interpolation based on .

[0018] Furthermore, 2) in step S5 is specifically: The multi-scale enhanced features are After differentiable homography transformation, it is warped to the front parallel plane of the reference camera frustum corresponding to the reference image, and the transformed feature volume corresponding to the current scale is obtained ,definition, The transformed feature body is the reference feature body, The transformed feature body is the source feature body, and the transformed feature body After the 3D weight learning network, the pixel-by-pixel similarity weights of the source feature body relative to the reference feature body are generated at each scale. ; The transformed feature bodies Along the channel direction, The cost body at the current scale is constructed by the following formula :

[0019] in, is the number of feature volume channels, is the matrix dot product, for A simplified representation of The reference feature Group features, For the The source feature Group characteristics; In step S5, 3), 2D regularization is used to process the cost volume of the current scale, specifically: In the corresponding depth prediction process, the probability body After upsampling and cost volume Cascade along the depth dimension, pass through 2D U-Net with batch normalization and ReLU activation function; use Softmax function to normalize the processed feature volume to obtain the probability volume of the current scale .

[0020] Further, step S4 is specifically as follows: S4.1. The lowest scale feature at scale Sampling at equal intervals along the inverse depth interval Deep Hypothesis :

[0021] in, is the preset maximum depth limit, is the pre-set minimum depth limit; S4.2. Using differentiable homography to transform Transformed into the feature volume under the reference image perspective , for the feature body Find the variance to construct the cost volume; S4.3. Steps The obtained cost volume at the scale is normalized along the depth direction to obtain Probability volume at scale , , weighted to obtain the predicted depth at the lowest scale :

[0022] in, Depth Hypothesis probability.

[0023] At the same time, the present invention also provides a dynamic adaptive multi-view stereo reconstruction model based on dual-domain information fusion, which is used to implement the above-mentioned dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion, and its special features are: it includes a multi-scale spatial domain, a frequency domain information fusion network and a multi-scale depth prediction network; the multi-scale spatial domain, frequency domain information fusion network includes a feature extraction module, a spatial domain detail enhancement module, a frequency domain dynamic aggregation module and a dual-domain information fusion module; the feature extraction module is used to perform multi-scale feature extraction on multi-view images to obtain multi-scale features ; The spatial detail enhancement module is used to perform deep convolution on multi-scale features to achieve spatial detail enhancement and output features ; The frequency domain dynamic aggregation module is used to analyze multi-scale features Perform dynamic aggregation and enhancement in the frequency domain and output features ; The dual-domain information fusion module is based on features ,feature and multi-scale features The multi-scale depth prediction network includes a depth initialization module, a dynamic range adaptive propagation module, a dynamic range uncertainty perception module, a feature volume adaptive aggregation module and a cost volume regularization module; the depth initialization module is used to determine Probability volume of reference image at different scales and predicted depth ; The dynamic range adaptive propagation module is used to construct a The dynamic range uncertainty perception module is used to construct a partial depth hypothesis for the reference image. Another part of the depth hypothesis at the scale of k=1,2,3; the feature body adaptive aggregation module is used to perform differentiable homography transformation on the multi-scale enhanced features at the scale of k=1,2,3 to obtain the transformed feature body, and then adaptively aggregate the transformed feature body to obtain the cost body at the current scale; the cost body regularization module is used to process the cost body at the scale of k=1,2,3 to obtain the corresponding probability body and predicted depth.

[0024] Furthermore, the feature extraction module includes 8 2D convolutional layers, each of which is followed by batch normalization and activation functions, where the convolution kernel size of the 3rd, 5th, and 7th convolutional layers is , the stride and padding are 2, and the convolution kernel size of the remaining convolutional layers is , stride and padding are 1.

[0025] Beneficial effects of the present invention: 1. The dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion in this invention adopts a coarse-to-fine framework to construct a multi-scale depth prediction network. The lowest scale feature of the image is used to sample depth hypotheses at equal intervals along the inverse depth interval. In the remaining stages, the corresponding depth hypotheses are learned using the uncertainty hypothesis mechanism based on dynamic range multi-scale adaptive propagation. This significantly improves the adaptability to complex geometric scenes, makes the generated candidate depths more effective, and lays the foundation for the final high-quality predicted depth.

[0026] 2. The multi-scale spatial domain and frequency domain information fusion network of the dynamic adaptive multi-view stereo reconstruction model based on dual-domain information fusion of the present invention is provided with a spatial domain detail enhancement module and a frequency domain dynamic aggregation module, which respectively It optimizes spatial details and dynamically aggregates and enhances the frequency domain to improve the ability to capture and express targets of different scales, and makes full use of the characteristics of frequency domain information to help the multi-scale spatial and frequency domain information fusion network capture richer and more important features. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a structural schematic diagram of an embodiment of a dynamic adaptive multi-view stereo reconstruction model based on dual-domain information fusion of the present invention; Figure 2 It is a flow chart of an embodiment of a dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion of the present invention; Figure 3 (a) to (h) are schematic diagrams of reconstructed point cloud images obtained according to an embodiment of the dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion of the present invention. DETAILED DESCRIPTION

[0028] The dynamic adaptive multi-view stereo reconstruction model based on dual-domain information fusion of the present invention includes a multi-scale spatial domain, a frequency domain information fusion network and a multi-scale depth prediction network. Figure 1 As shown in the figure, the multi-scale spatial and frequency domain information fusion network includes feature extraction module, spatial detail enhancement module (SDDE), frequency domain dynamic aggregation module (FDDA) and dual domain information fusion module (DDF). The multi-scale depth prediction network includes depth initialization module, dynamic range adaptive propagation module (DRAP), dynamic range uncertainty perception module (UA), feature volume adaptive aggregation module and cost volume regularization module.

[0029] Before using the above-mentioned dynamic adaptive multi-view stereo reconstruction model based on dual-domain information fusion for reconstruction, the model needs to be trained. The loss constrains the difference between the predicted depth map and the ground truth depth map, thereby guiding the network training and obtaining a trained dynamic adaptive multi-view stereo reconstruction model.

[0030] loss: ; Overall loss:

[0031] in, Representation scale The true depth value under For scale The predicted depth under Represents the weight coefficient of each scale.

[0032] After training, the image to be reconstructed is input into the model and reconstructed using the dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion of the present invention. Figure 2 As shown, the specific steps include: Step 1: Use multi-scale spatial and frequency domain information fusion network to extract features.

[0033] 1.1. Obtain N multi-view images of the same scene , and the camera intrinsic parameters corresponding to N multi-view images and external parameters , where 3≤N≤6, Indicates the sequence number of the multi-view image, ranging from 0 to N-1. and Respectively represent The rotation and translation components in the camera extrinsic parameters corresponding to the multi-view images. One of the N multi-view images is used as the reference image, and the remaining N-1 images are used as the source images. represents the reference image, They represent N-1 source images respectively. It is defined that the camera corresponding to the reference image is the reference camera, and the camera corresponding to the source image is the source camera.

[0034] 1.2. Multi-scale feature extraction The feature extraction module is used to extract a reference image and Zhang source image, respectively, multi-scale feature extraction to obtain the corresponding multi-scale features ,in, Indicates scale, The feature extraction module contains 8 2D convolutional layers, each of which is followed by batch normalization and activation functions. The convolution kernel size of the 3rd, 5th, and 7th convolutional layers is , the stride and padding are 2, and the convolution kernel size of the remaining convolutional layers is , stride and padding are 1.

[0035] 1.3. Use the spatial detail enhancement module to enhance the spatial details; 1.3.1. Use deep convolution to encode multi-level features; Using depthwise convolution Handling multi-scale features , get the intermediate features As shown in the following formula, d is the expansion rate. For the convenience of expression, the subsequent steps are omitted. The superscripts and subscripts are abbreviated as .

[0036]

[0037] Will Proportional along the channel Perform the split operation to obtain 、 、 ,in, , By depth convolution and Process them separately to get the convolution features , , remain unchanged;

[0038]

[0039] 1.3.2. , , After cascading along the channel direction, it passes through a Point-by-point convolution integrates channel information to obtain the features of each multi-view image output by the spatial detail enhancement module :

[0040] In this embodiment, the intermediate feature Divide into three parts along the channel direction in proportion, and , Convolution kernels of different sizes and dilation rates are used for depth convolution. No processing is done. This way of dividing the channels can effectively avoid repeated calculations of all features and improve calculation efficiency. , Deep convolution processing with different dilation rates and convolution kernel sizes is used to enhance the ability to capture and express objects of different scales. In addition, Directly retaining the original features can provide untransformed detail information and avoid the loss of detail information or feature blurring caused by deep convolution operations.

[0041] 1.4. Dynamic aggregation and enhancement using frequency domain dynamic aggregation module 1.4.1. Multi-scale features Adaptively weight the high-frequency and low-frequency features within the corresponding frequency range to obtain weighted spectrum data corresponding to each multi-view image. Among them, low frequency refers to the global structure outline, and high frequency refers to the edge area and texture details. Adaptive weighting is used to highlight low-frequency and high-frequency features, and to suppress harmful and useless frequency information such as noise and background. Low-frequency features By multi-scale features The weighted spectrum data is obtained by performing pooling operations with different window sizes. The specific formula is as follows:

[0042]

[0043] in, is a two-dimensional convolutional layer with a Sigmoid activation function for calculating the weights of low-frequency features. is a two-dimensional convolutional layer with a Sigmoid activation function for calculating the weights of high-frequency features. Represents the pooling operation. For simplicity, only the formula for pooling once is listed here.

[0044] The multi-scale pooling kernel in the present invention can capture information in different frequency ranges, fully consider the impact of different frequency characteristics on the reconstruction quality, adaptively weight low-frequency characteristics and high-frequency characteristics respectively, highlight the scene-related frequency characteristics, and suppress the interference of useless frequencies such as noise and background on the reconstruction quality. It can improve the generalization ability of the multi-scale spatial and frequency domain information fusion network for different scenes and enhance its robustness.

[0045] 1.4.2. Weighted Spectral Data Perform dynamic aggregation and enhancement Weighted spectral data Convert from the spatial domain to the frequency domain via Fourier transform and calculate the amplitude and Phase , and dynamically enhance them through the same modulation network, and then add residual connections to obtain new amplitudes and new phase , as shown in the following formula, where the modulation network is composed of , LeakyReLU and Efficient Channel Attention constitute.

[0046]

[0047]

[0048] Use the new amplitude and new phase Reconstruct the complex tensor, and then perform inverse Fourier transform to obtain the features of each multi-view image output by the frequency domain dynamic aggregation module .

[0049] The amplitude reflects the global energy distribution and is suitable for capturing overall features, while the phase contains local detail information, which helps to improve the expression of fine-grained features. Dynamic enhancement and residual connection can make full use of the characteristics of frequency domain information and help the multi-scale spatial and frequency domain information fusion network capture richer and more important features. In other embodiments of the present invention, the amplitude and Phase Dynamic enhancement can also be performed through different modulation networks. The modulation network used in the present invention is a lightweight modulation network that can dynamically adjust the weight of the spectrum information, highlight the key frequency bands and suppress useless frequencies, and enhance the robustness and discrimination of the features. Among them, the residual connection is set so that the frequency domain dynamic aggregation module can learn the changes in spectrum features more efficiently, while improving the gradient fluidity of the network, making the optimization process more stable.

[0050] 1.5. Interactive fusion of dual-domain information The corresponding features are fused through the dual-domain information fusion module and Features pass and The operation is fused to obtain the fusion feature , thereby improving the model's ability to capture multi-scale frequency domain details and global information:

[0051] The dual-domain information fusion module fuses the features Upsampling and The corresponding scales are added to obtain the multi-scale enhanced features of dual-domain information fusion .

[0052] Step 2: Determine the multi-scale enhanced features at scale k=0 The probability body .

[0053] 2.1. In order to allocate limited resources to more likely depth hypotheses, the depth sampling initialization module performs The lowest scale feature of is sampled at equal intervals along the inverse depth interval, and the Deep hypothesis:

[0054] in, is the corresponding serial number of the depth hypothesis, and are respectively the maximum depth limit and the minimum depth limit which are preset. and They are set to 935mm and 425mm respectively. The value is 48.

[0055] 2.2. Determine the cost volume at the lowest scale Using differentiable homography transformation Transformed into the feature volume under the reference image perspective ; For feature body Find the variance to construct the cost volume :

[0056] in, represents the feature volume mean, for A simplified representation of .

[0057] 2.3. Depth sampling initialization module for cost volume Normalize along the depth direction and get Probability volume at scale , , and then weighted to obtain the predicted depth at the lowest scale :

[0058] in, Depth Hypothesis probability.

[0059] Steps 2.2 and 2.3 are as follows Figure 1 As shown in the homography transformation and variance aggregation (HW&Variance).

[0060] Step 3: Determine the predicted depth at scales k=1, 2, and 3 . The predicted depth of the corresponding scale Specifically obtained through the following methods: 1) Determine the multi-scale enhancement features when k=1,2,3 The depth assumption The present invention proposes an uncertainty search method of dynamic range adaptive propagation, which utilizes the uncertainty hypothesis mechanism based on dynamic range multi-scale adaptive propagation to learn robust depth hypotheses for uncertainty regions by aggregating cross-scale dynamic range information.

[0061] a. Dynamic range information aggregation Dynamic range adaptive propagation module in At the scale of Enhanced features of scale ,pass Dimensionality reduction to obtain dimensionality reduction features , and then reduce the dimension features Along axis direction (width direction of the image) and The axis direction (the height direction of the image) is sorted from large to small according to the eigenvalues ​​( ), and get the corresponding features and , respectively expressed as: , and record the indexes in the X-axis and Y-axis directions , .

[0062] The above features are respectively and Perform layer normalization and one-dimensional convolution to achieve cross-scale dynamic range feature aggregation in two directions and obtain the feature and The convolution kernel sizes used are and ,use The representation layer is normalized, then:

[0063]

[0064] According to the index and Features and Restore and get the features and ,in Indicates restore operation by index:

[0065]

[0066] Through the above-mentioned sorting and index recovery operations, one-dimensional convolution can operate between similar pixels across regions rather than between adjacent pixels, thereby effectively capturing and focusing on uncertainty areas while maintaining geometric structure consistency.

[0067] Use the reshape operation to and Reshape and obtain the corresponding QKV matrix, which are expressed as , , and , , , where Q, K, and V represent the Query, Key, and Value feature vectors in the attention mechanism, respectively.

[0068] Finally, the dual-axis cross attention is performed, and the first cross attention is calculated by the following formula : ; in, Indicates adjusting the shape of the tensor. is the activation function, represents matrix dot product, and T is the transpose.

[0069] Similarly, the second cross attention is calculated by : .

[0070] The two cross attentions are fused and residual connections are added to obtain the fusion features of the reference image with rich dynamic range global information at the corresponding scale. : .

[0071] The present invention sets a dual-axis cross attention between the dynamic range enhancement features in two directions, which is used to focus on the correlation in the horizontal direction and the vertical direction respectively. The dual-axis cross attention can better capture global information, enhance the modeling ability of the multi-scale depth prediction network for complex structures and textures, and significantly reduce the computational complexity compared with conventional self-attention.

[0072] b. Dynamic Range Adaptive Propagation (DRAP) module builds deep hypothesis The dynamic range adaptive propagation modules are respectively At the scale of , using the spatial correlation between its neighborhood pixels and itself to guide deep adaptive propagation, specifically: introducing dilated convolution to act on the fusion features of the corresponding scale Learn to use pixels centered The 8 pixels in the neighborhood are and Offset in both directions According to the offset , predict the depth of the previous scale Bilinear interpolation is applied to generate 8 depth hypotheses at the current scale based on dynamic range adaptive propagation.

[0073] The present invention integrates the characteristics Applying dilated convolution to calculate adaptive offset can enhance the receptive field while dynamically adjusting the depth sampling position according to the scene, flexibly capturing complex geometric structures and parallax change information, significantly improving the adaptability to complex geometric scenes, and laying the foundation for achieving high-quality depth prediction.

[0074] c. Building deep hypotheses through the dynamic range uncertainty awareness module (UA) The dynamic range uncertainty perception modules are respectively At the scale of After being processed by the existing shallow 2D weight learning network with residual connections, the pixels are learned and The weight distribution of 8 pixels in the neighborhood, and then the Softmax operation is applied to get the normalized weight , which is simplified as ,in, Pixel and The ordinal numbers of the 8 pixels in the neighborhood.

[0075] According to the previous scale probability and predicted depth Calculate the At this scale, pixels Standard deviation of predicted depth For simplicity of expression, The superscript and the content in brackets represent the scale, then pixels The standard deviation of the predicted depth is denoted by , as shown in the following formula:

[0076] in, For the The number of depth hypotheses at each scale, For the The scale A deep hypothesis, For the The depth at this scale is The probability of For the Prediction depth at each scale.

[0077] Then according to the previously learned offset Adaptive propagation of neighborhood coordinates, in standard deviation The offset pixels are obtained by interpolation based on and the standard deviation of its neighboring pixels , weighted to obtain the optimal standard deviation : .

[0078] Therefore, the present invention can adaptively transform pixels based on the fusion features. Different weights are assigned to pixels and their neighboring pixels. Normalized weights ensure that the contribution of each pixel is effectively balanced in the standard deviation calculation, thereby reducing the interference of noise or outliers on depth prediction. In addition, weak texture areas usually lack obvious features, and it is difficult to obtain accurate depth through traditional methods. This method combines pixels by weight. The characteristics and standard deviation information of its neighborhood pixels can effectively supplement the areas with insufficient texture information and improve the reliability of depth prediction.

[0079] The dynamic range uncertainty perception module optimizes the standard deviation Counting pixels Depth of search range : ,in, A scalar parameter that controls the width of the sampling interval.

[0080] At the current scale, in pixels Depth of search range The depth sampling is performed within, and the number of samples is ,in, Indicated in The number of depth hypotheses at each scale. In this embodiment, = Number of depth hypotheses at scales 1, 2, 3 M 1 、M 2 、M 3 Set to 32, 16, and 16 respectively, which can be set and adjusted based on experience. The Gaussian quantile function (i.e., the inverse cumulative distribution function of the Gaussian distribution) is introduced to determine the depth sampling points. The sampling points generated in this way will be The center presents a Gaussian distribution, the closer The denser the position sampling, the farther away The more sparse the position sampling is. The depth hypothesis corresponding to the depth sampling points is: ; in, For the The scale A deep hypothesis, , is the quantile function of the standard normal distribution, It means the mean The standard deviation is Quantile function of the normal distribution; Indicates the depth of the search range The sum of the probabilities of the inner coverage.

[0081] The present invention introduces a Gaussian quantile function to generate depth sampling points, so that the sampling points present a Gaussian distribution with the predicted depth as the center, densely distributed in areas with higher probability, and sparsely covering low-probability areas far from the center. The advantages of this method are: effectively concentrating computing resources on the high confidence range of depth estimation, improving computing efficiency; taking into account the coverage of low-probability areas, enhancing the robustness and global search capabilities of the present invention, thereby reducing computing overhead while maintaining accuracy.

[0082] 2) The feature body adaptive aggregation module performs feature body aggregation; The feature volume adaptive aggregation module respectively converts the multi-scale enhanced features corresponding to the current scale into After the differentiable homography transformation (HW), it is warped to the front parallel plane of the reference camera frustum to obtain the transformed feature volume corresponding to the current scale. For the sake of simplicity, the transformed feature volume is recorded as ,definition, The transformed feature body is the reference feature body, The transformed feature volume is the source feature volume. The differentiable homography transformation formula for differentiable homography transformation is as follows:

[0083] in, represents any pixel in the reference image, express In the The pixels corresponding to the multi-view images; and represents the rotation and translation components in the reference camera extrinsics, and Respectively represent The rotation and translation components in the camera extrinsic parameters corresponding to the multi-view images; are the intrinsic parameters of the reference camera, is the intrinsic parameter of the camera corresponding to the i-th multi-view image; It is The scale A deep hypothesis.

[0084] In order to balance computational efficiency and matching quality, an adaptive group correlation (GC) strategy is applied to construct the cost volume. This paper designs a lightweight 3D weight learning network to generate the pixel-by-pixel similarity weight of the source feature volume relative to the reference feature volume at the current scale. , which helps to reduce the mismatch caused by occlusion between views. Table 1 shows the specific parameters of the lightweight 3D weight learning network. The transformed feature volume After being processed by three 3D convolutional layers with a step size of 1, the Softmax operation generates the pixel-by-pixel similarity weight of each source feature body to the reference feature body. , the first two 3D convolutional layers have batch normalization and activation functions, and the third 3D convolutional layer has no batch normalization and activation function.

[0085] Table 1 Specific parameters of lightweight 3D weight learning network

[0086] Then transform the feature volume at the current scale Along the channel direction, The cost volume that aggregates multiple feature volume information at the current scale is constructed by the following formula : ; in, is the number of feature volume channels, is the matrix dot product, for A simplified representation of The reference feature Group features, The value ranges from 1 to G. For the The source feature Group features.

[0087] 3) The cost volume regularization module of the probability volume geometric prior compensation performs cost volume regularization; In the depth prediction process of the remaining scales except the lowest scale, due to the increase in the resolution of the cost volume, 3D convolution will bring great memory consumption, so a 2D regularization with probability volume geometric prior compensation is introduced. Probability of scale Embed The cost of scale In the current In the depth prediction process corresponding to the scale, the probability body After upsampling and cost volume Cascading along the depth dimension, and processed by a 2D regularized network (U-Net network) with batch normalization and ReLU activation function, not only the model is lightweight, but also a good spatial information integration capability is guaranteed. Finally, the processed cost volume is normalized using the Softmax function to obtain the probability volume of the current scale. .

[0088] The predicted depth of the reference image at the current scale By probability body The weighted sum along the depth direction is obtained as shown in the following formula:

[0089] in, Indicates that the depth is probability.

[0090] Step 4: Use the public Open3d code to Prediction depth at scale Convert to a 3D point cloud file of the corresponding scene. Then use CloudCompare software to check the quality of the reconstructed point cloud image, such as Figure 3 As shown, there are eight reconstructed point cloud images obtained through this embodiment.

Claims

1. A dynamic adaptive multi-view stereo reconstruction method based on dual-domain information fusion, characterized in that: The following steps are involved: S1, input N multi-view images of the same scene to obtain multi-scale features ,in, Indicates scale, , Indicates the sequence number of the multi-view image, ranging from 0 to N-1. One of the N multi-view images is used as the reference image. represents the reference image, and the remaining N-1 images are used as source images; S2, for multi-scale features Perform spatial detail enhancement and output the features corresponding to each multi-view image ; For multi-scale features Perform dynamic aggregation and enhancement in the frequency domain and output the features corresponding to each multi-view image ; S3, the corresponding features With features Fusion gets fusion features ,Will Upsampling and corresponding multi-scale features Add together to get multi-scale enhanced features ; S4, determine the probability volume of the reference image at scale k=0 and its corresponding predicted depth ; S5, determine the predicted depth at scales k=1,2,3 ; Prediction depth of corresponding scale Specifically obtained through the following methods: 1) Based on multi-scale enhanced features , using the uncertainty hypothesis mechanism based on dynamic range multi-scale adaptive propagation to determine the corresponding depth hypothesis; 2) Use the depth hypothesis to enhance the features corresponding to the current scale Perform differentiable homography transformation to obtain the transformed feature volume , apply the adaptive group correlation strategy to aggregate the transformed feature volume to obtain the cost volume at the current scale; 3) Use 2D regularization to process the cost volume of the current scale to obtain the corresponding probability volume ; For probability body The weighted sum along the depth direction is used to obtain the predicted depth of the current scale. ; S6, predict the depth at scale k=3 Convert it into a three-dimensional point cloud file of the corresponding scene, and complete dynamic adaptive multi-view stereo reconstruction based on dual-domain information fusion.

2. The dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion according to claim 1, characterized in that: Step S2 is specifically as follows: S2.

1. Multi-scale features Perform deep convolution to obtain intermediate features ; S2.

2. Intermediate features Perform the split operation along the channel in proportion, and get , , ,right , Perform deep convolution to obtain convolution features and convolutional features ; S2.

3. , , After cascading along the channel direction, the channel information is integrated through point-by-point convolution to obtain the features corresponding to each multi-view image. ; S2.

4. Multi-scale features The features within the corresponding frequency range are adaptively weighted according to the frequency, highlighting the required frequencies and suppressing the useless frequencies, and obtaining the weighted spectrum data corresponding to each multi-view image. ; S2.

5. Weighted spectral data Convert from the spatial domain to the frequency domain via Fourier transform and calculate the amplitude and Phase , and dynamically enhance them through the modulation network, and then add residual connections to obtain new amplitudes and new phase ; S2.

6. Using the new amplitude and new phase Reconstruct the complex tensor and then perform inverse Fourier transform to obtain the features corresponding to each multi-view image .

3. The dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion according to claim 2, characterized in that: In step S2.1, the depth convolution adopts depth convolution , where d is the expansion rate; In step S2.2, the ratio is: ;right , The specific steps for deep convolution are: , Depthwise convolution and depthwise convolution to process; In step S2.3, the point-by-point convolution is performed using Point-wise convolution; Step S2.4 is to obtain weighted spectrum data by the following formula: : ； in, is a two-dimensional convolutional layer with a Sigmoid activation function for calculating the weights of low-frequency features. is a two-dimensional convolutional layer with a Sigmoid activation function for calculating the weights of high-frequency features. is a low frequency feature, Multi-scale features A simplified representation of ; In step S2.5, the modulation network is composed of , LeakyReLU and Efficient Channel Attention The above mentioned components are dynamically enhanced by the modulation network, and then the residual connection is added to obtain the new amplitude. and new phase , specifically: ； 。 4. The dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion according to any one of claims 1 to 3, characterized in that: 1) in step S5 is specifically: a. respectively At the scale of Dimensionality reduction to obtain dimensionality reduction features , and then reduce the dimension features Along Axis direction and Perform sorting in the axis direction to obtain the corresponding features and , and record the indexes in the X-axis and Y-axis directions , ; In the X-axis direction and Y-axis direction, the features and Perform layer normalization and one-dimensional convolution, and then according to the index and Restore order to get features and ; The first cross attention is calculated by and the second cross attention : ； ； in, Indicates adjusting the shape of the tensor. is the activation function, represents matrix dot multiplication, T is transpose, , , for The corresponding QKV matrix, , , for The corresponding QKV matrix; The first cross-attention and the second cross attention Fusion, and then with dimensionality reduction features Residual connection to obtain the fusion features of the reference image at the corresponding scale ; b. Respectively in At the scale of , for each pixel in the reference image ,use The spatial correlation between pixels in the neighborhood and themselves guides the adaptive propagation of depth, and 8 depth hypotheses at the current scale are obtained; c. At the current scale, in pixels The depth sampling within the depth search range is obtained deep hypothesis, where For the The number of depth hypotheses at each scale is determined by using the Gaussian quantile function, and the following formula is used to obtain the depth hypothesis at each scale. The depth hypothesis corresponding to the depth sampling points is: ； in, For the The scale A deep hypothesis, , To optimize the standard deviation, is the quantile function of the standard normal distribution, For the The predicted depth at each scale is It means the mean The standard deviation is Quantile function of the normal distribution; represents the total probability of coverage within the depth search range, and λ is a scalar parameter that controls the width of the sampling interval.

5. The method for dynamic adaptive multi-view stereoscopic reconstruction based on dual-domain information fusion according to claim 4, characterized in that: In step a, the convolution kernel sizes of the one-dimensional convolution are and ; Step b is as follows: for each pixel in the reference image , using dilated convolution to act on fusion features Learn to use pixels centered The 8 pixels in the neighborhood are and Offset in both directions , according to the offset , for Prediction depth at each scale Apply bilinear interpolation to obtain 8 depth hypotheses for the current scale; In step c, the pixel The depth search range is obtained by: Based on fusion features , get pixel and Normalized weights of the 8 pixels in the neighborhood , and The standard deviation of the predicted depth at each scale ,in, Pixel and The serial numbers of the 8 pixels in the neighborhood; Get the offset pixel and the standard deviation of its neighboring pixels , weighted to obtain the optimal standard deviation : ; Calculate pixels by optimizing the standard deviation The depth search range R is: ,in, is a scalar parameter that controls the width of the sampling interval, For the Prediction depth at each scale.

6. The method for dynamic adaptive multi-view stereoscopic reconstruction based on dual-domain information fusion according to claim 5, characterized in that: In step c, the normalized weight The specific method is as follows: After being processed by a shallow 2D weight learning network with residual connections, the pixels are learned and The weight distribution of 8 pixels in the neighborhood, and then the Softmax operation is applied to get the normalized weight ; No. The standard deviation of the predicted depth at each scale Obtained by the following formula: ； in, For the The number of depth hypotheses at each scale, For the The scale A deep hypothesis, For the The depth at this scale is The probability of For the Prediction depth at each scale; The offset pixel and the standard deviation of its neighboring pixels According to the offset obtained in step b , in standard deviation It is obtained by interpolation based on .

7. The dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion according to claim 6, characterized in that: 2) in step S5 is specifically: The multi-scale enhanced features are After differentiable homography transformation, it is warped to the front parallel plane of the reference camera frustum corresponding to the reference image, and the transformed feature volume corresponding to the current scale is obtained ,definition, The transformed feature body is the reference feature body, The transformed feature body is the source feature body, and the transformed feature body After the 3D weight learning network, the pixel-by-pixel similarity weights of the source feature body relative to the reference feature body are generated at each scale. ; The transformed feature bodies Along the channel direction, The cost body at the current scale is constructed by the following formula : ； in, is the number of feature volume channels, is the matrix dot product, for A simplified representation of The reference feature Group features, For the The source feature Group characteristics; In step S5, 3), 2D regularization is used to process the cost volume of the current scale, specifically: In the corresponding depth prediction process, the probability body After upsampling and cost volume Cascade along the depth dimension, pass through 2D U-Net with batch normalization and ReLU activation function; use Softmax function to normalize the processed feature volume to obtain the probability volume of the current scale .

8. The dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion according to claim 7, characterized in that: Step S4 is specifically as follows: S4.

1. The lowest scale feature at scale Sampling at equal intervals along the inverse depth interval Depth Hypothesis : ； in, is the pre-set maximum depth limit, is the pre-set minimum depth limit; S4.

2. Using differentiable homography to transform Transformed into the feature volume under the reference image perspective , for the feature body Find the variance to construct the cost volume; S4.

3. Steps The obtained cost volume at the scale is normalized along the depth direction to obtain Probability volume at scale , , weighted to obtain the predicted depth at the lowest scale : ； in, Depth Hypothesis probability.

9. A dynamic adaptive multi-view stereoscopic reconstruction model based on dual-domain information fusion, used to implement the dynamic adaptive multi-view stereoscopic reconstruction method based on dual-domain information fusion according to any one of claims 1 to 8, characterized in that: Including multi-scale spatial domain, frequency domain information fusion network and multi-scale depth prediction network; The multi-scale spatial domain and frequency domain information fusion network includes a feature extraction module, a spatial domain detail enhancement module, a frequency domain dynamic aggregation module and a dual-domain information fusion module; the feature extraction module is used to extract multi-scale features from multi-view images to obtain multi-scale features. ; The spatial detail enhancement module is used to perform deep convolution on multi-scale features to achieve spatial detail enhancement and output features. ; The frequency domain dynamic aggregation module is used to analyze multi-scale features Perform dynamic aggregation and enhancement in the frequency domain and output features ; The dual-domain information fusion module is based on feature ,feature and multi-scale features The feature fusion of is used to obtain multi-scale enhanced features; The multi-scale depth prediction network includes a depth initialization module, a dynamic range adaptive propagation module, a dynamic range uncertainty perception module, a feature volume adaptive aggregation module, and a cost volume regularization module; The depth initialization module is used to determine Probability volume of reference image at different scales and predicted depth ; The dynamic range adaptive propagation module is used to construct a Partial depth assumptions at scale; The dynamic range uncertainty awareness module is used to construct a Another part of the depth hypothesis at the scale of ; The feature volume adaptive aggregation module is used to perform differentiable homography transformation on the multi-scale enhanced features at scales k=1, 2, and 3 to obtain the transformed feature volume, and then adaptively aggregate the transformed feature volume to obtain the cost volume at the current scale; The cost body regularization module is used to process the cost body at the scale of k=1,2,3 to obtain the corresponding probability body and predicted depth.

10. The dynamic adaptive multi-view stereo reconstruction model based on dual-domain information fusion according to claim 9, characterized in that: The feature extraction module contains 8 2D convolutional layers, each of which is followed by batch normalization and activation functions. The convolution kernel size of the 3rd, 5th, and 7th convolutional layers is , the stride and padding are 2, and the convolution kernel size of the remaining convolutional layers is , stride and padding are 1.

Images