Multi-view three-dimensional reconstruction method based on frequency perception feature enhancement and cost aggregation

Abstract

The invention relates to an image three-dimensional reconstruction method, in particular to a multi-view three-dimensional reconstruction method based on frequency perception feature enhancement and cost aggregation. The objective of the invention is to overcome the defects of lack of frequency sensing capability and limited processing capability for problems of weak texture, noise, illumination variation, color distortion and the like in an existing learning-based multi-view three-dimensional reconstruction method. According to the method, multi-view three-dimensional reconstruction is realized through the steps of acquiring a multi-view image, calculating a multi-scale frequency sensing feature, calculating an initial cost body, embedding frequency information into the initial cost body, calculating depth estimation, performing back projection and the like in sequence; when multi-scale frequency sensing features are calculated, a double-branch frequency component enhancement module is arranged to process wavelet transform layer decomposition to obtain lossless approximate low-frequency components and high-frequency components of an input image, so that global consistency and local detail expression of depth estimation are improved.

Description

Technical Field

[0001] The present invention relates to a method for stereoscopic image reconstruction, and in particular to a multi-view stereoscopic reconstruction method based on frequency perception feature enhancement and cost aggregation. Background Art

[0002] Multi-view stereo reconstruction provides a contactless solution for 3D surface reconstruction and measurement and has been widely used in fields such as medical diagnosis, pose estimation, virtual reality / augmented reality, and cultural heritage preservation. In recent years, learning-based multi-view stereo reconstruction (MVS) methods use convolutional neural networks to extract multi-channel image features and aggregate information from different views into a cost volume that measures the confidence of each pixel under different depth hypotheses. Subsequently, a 3D convolutional neural network (3D CNN) is used to regularize the cost volume to eliminate the effects of occlusion and noise, completing multi-view stereo reconstruction. Compared with traditional methods, this method shows superior performance in challenging areas such as weak textures and specular reflections.

[0003] Existing learning-based cascaded MVS methods often construct multi-scale features using a feature pyramid, continuously refining depth hypothesis intervals and gaps in a coarse-to-fine manner, ultimately achieving a refined depth estimate. However, existing methods only exploit the input image's feature representations and the spatial correlations between pixels in the spatial domain, failing to explore the differences in the modeling capabilities of different frequency information for complex regions from a frequency perspective. This results in the generated cost volume lacking frequency awareness and limited ability to handle weak textures, noise, illumination variations, and color distortion. Furthermore, some methods typically employ conventional convolutional downsampling operations and perform skip connections between features at different scales when constructing multi-scale features, which can lead to loss of important spatial information and semantic ambiguity, compromising the preservation of key details such as boundaries, scale, and texture. Furthermore, a 3D U-Net regularization network is used to enforce cost volume consistency to reduce mismatches caused by occlusion and illumination variations. Furthermore, due to the limited local receptive field, depth information is smoothly propagated across adjacent regions, blurring boundary features. Therefore, it is crucial to impart frequency awareness to the model during multi-scale feature construction and cost aggregation, enabling it to sensitively capture rich frequency variations. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of existing learning-based multi-view stereo reconstruction methods, such as the lack of frequency perception and limited ability to handle problems such as weak texture, noise, illumination changes and color distortion, and to provide a multi-view stereo reconstruction method based on frequency perception feature enhancement and cost aggregation.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: A multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation is characterized in that it includes the following steps: S1, collect multi-view images of the target to be reconstructed, the multi-view images include N images, where N≥2, record the camera parameters corresponding to the image; take each image as a reference image, and take the rest as the reference image. images as their corresponding source images; S2, calculates multi-scale frequency-aware features; S2.1, extract the multi-scale features of each reference image and the corresponding source image, denoted as and ,in, is the sequence number of the source image, with values ​​ranging from 1,...,N-1, and k represents the scale; S2.2, by wavelet transform layer decomposition and dual-branch frequency component enhancement, the scale Frequency characteristics of the reference image when ; S2.3, Fusion Scale Frequency characteristics of the reference image when and the corresponding multi-scale features , and obtain the frequency perception characteristics of the corresponding scale ; S3, calculate the initial cost volume; Using homography transformation, the multi-scale features of the source image at the current scale are transformed Frequency perception features projected to the corresponding scale or multi-scale features of the reference image At the same perspective, perform correlation calculation to obtain the initial cost volume at the current scale ; S4, using the local dynamic window self-attention mechanism to embed the frequency information into the initial cost body, and obtain the frequency-aware cost body , based on frequency-aware cost and the initial cost body Obtaining a probability body , calculate the depth hypothesis corresponding to the position with the highest probability, and obtain the depth estimate of each pixel at the current scale; S5, back-projects the depth estimate at scale k=2 into three-dimensional space, constructs a dense three-dimensional point cloud representation of the target scene, and completes multi-view stereo reconstruction.

[0006] Furthermore, step S2.2 is specifically as follows: S2.2.1. Wavelet transform layer decomposition; Reference image Perform wavelet transform layer decomposition to obtain approximate low-frequency components and high-frequency components when the scale k=1, wherein the high-frequency components include high-frequency components in horizontal, vertical and diagonal directions; S2.2.2. Back projection; The reference image is transformed into The two-dimensional coordinates of each pixel in the image are converted into a ray direction in three-dimensional space, and then a scale-invariant high-dimensional spatial position encoding is generated for each component of the ray direction; S2.2.3. Feature mapping; Will Add the low-frequency features of the current scale, and then map the features to the multi-head attention space through linear transformation to obtain query, key and value tensors; The low-frequency feature is calculated based on the approximate low-frequency component; S2.2.4. Global self-attention calculation; Perform global self-attention calculation based on query, key and value tensors to obtain enhanced low-frequency fusion features; S2.2.5. Enhancement of high frequency components; The high-frequency components are processed through two layers of convolution operations with residual connections to obtain enhanced high-frequency features; S2.2.6. Feature stitching; The enhanced low-frequency fusion features and enhanced high-frequency features are spliced ​​along the channel dimension, and the frequency features at scale k=1 are obtained by point-by-point convolution. ; S2.2.7. Generation of high-resolution frequency features; Frequency characteristics Perform wavelet transform layer decomposition to obtain the approximate low-frequency component and high-frequency component when scale k=0. Follow the method of steps S2.2.2 to S2.2.6 to obtain the frequency characteristics when scale k=0. .

[0007] Furthermore, in step S2.2.1, the wavelet transform layer decomposition is a Haar wavelet transform layer decomposition; In step S2.2.2, the high-dimensional spatial position encoding is specifically implemented by the following formula:

[0008]

[0009] in, Respectively represent the coordinates of the ray direction in the X, Y, and Z directions in three-dimensional space, express The inverse projection matrix, Represents the intrinsic parameters of the camera corresponding to the reference image, R represents the rotation matrix of the camera corresponding to the reference image, and t represents the translation matrix of the camera corresponding to the reference image. represents the frequency factor, is the learnable scale factor, represents the direction of the ray in three-dimensional space, Represents high-dimensional spatial position encoding; In step S2.2.3, the low-frequency features Specifically: When scale k=1, ,in, Represents the approximate low-frequency component when k=1; When scale k=0, ,in, Represents the approximate low-frequency component when k=0.

[0010] Furthermore, in step S2.2.5, the enhanced high-frequency features are obtained specifically by the following method:

[0011] in, represents the enhanced high-frequency features when scale k=1, represents residual convolution, Indicates cascade operation, Respectively represent the high-frequency components in the horizontal, vertical and diagonal directions when the scale k=1; In step S2.2.6, frequency characteristics Specifically obtained through the following methods:

[0012] in, represents 1 × 1 point-by-point convolution, Represents the enhanced low-frequency fusion features when scale k=1.

[0013] Furthermore, step S3 is specifically as follows: S3.1, depth assumption for each pixel in N images at the current scale Perform uniform sampling based on frequency perception features , get the reference feature volume at the current scale ; Through differentiable homography transformation, the pixels of the reference image are transformed As a benchmark, the multi-scale features of each source image are obtained In depth assumption The corresponding pixel , based on each pixel of the source image , get the feature volume of the source image , Represents the feature volume of the i-th source image; S3.2, the reference feature body at the current scale and the feature volume corresponding to the source image The channels are divided into Group, perform correlation calculation on the sub-features of each group of the reference image and its corresponding source image, and obtain the correlation calculation results of all group sub-features of the i-th source image ; S3.3, obtaining the visibility weight of the source image relative to the reference image ,use and Calculate the initial cost volume.

[0014] Furthermore, in step S3.1, the depth hypothesis The sampling intervals are as follows: scale When , the sampling interval is the distance range from the target to be measured in the scene to the optical center of each camera; scale When , the sampling interval is [ ]; scale When , the sampling interval is [ ]; in, represents the depth estimation when scale k=0, represents the depth estimation when scale k=1, and Respectively represent scale Time-depth hypothesis set The interval between depth sampling, and Respectively represent scale The sampling interval scale factor when ; Number of samples for depth hypothesis exist ,2 when decreasing successively; The reference feature Calculated as follows: When scale k=0,1, the frequency perception features of the reference image at the current scale are Copy M times in the depth dimension to obtain the corresponding reference feature body ; When scale k=2, the multi-scale features of the reference image at the current scale are Copy M times in the depth dimension to obtain the corresponding reference feature body ; The pixel It is obtained by the following formula:

[0015] in, represents the inverse operation of the camera intrinsic parameters of the reference image, Indicates the indivual The intrinsic parameters of the camera, The rotation matrix of the source image's camera relative to the reference image's camera, Represents the translation matrix of the source image's camera relative to the reference image's camera.

[0016] Furthermore, step S3.2 is specifically as follows: S3.2.1. and The channels are divided into Group; for reference feature Perform depth-wise separable convolution and dilated convolution to obtain enhanced reference feature volumes , specifically:

[0017] in, represents point-wise convolution, is the dilated convolution, It is a depth-wise separable convolution; S3.2.2. Use Convolution will enhance the reference feature volume Mapped to a single-channel space-depth selectivity weight and normalized to obtain the reference image normalized weight ,in, Indicates the shape Tensor of , Represents the height and width of the image respectively; Normalize weights to the reference image With reference feature Perform element-wise multiplication and add residual connections to generate the modulated feature volume , specifically:

[0018] in, express and Element-wise multiplication of ; The modulated feature Perform spatial compression and generate affinity matrix through fully connected layers ,in, Indicates the shape Tensor, C represents the number of channels; S3.2.3. According to affinity matrix ,Will Regroup according to the group, among which, Indicates the shape The tensor of W is used to obtain the sub-features of each group of the source image. :

[0019] in, express In the The first one under the deep assumption channels, Indicates the mth depth hypothesis Channel and The degree of correlation between groups, c=1,...,C, m=1,...,M, g=1,...,G; Similarly, the sub-features of the g-th group of the reference image are calculated ; S3.2.4. Perform correlation calculation for each sub-feature of the reference image and the corresponding source image using the following formula, and record the similarity calculation result of all sub-features of the i-th source image as ;

[0020] in, represents the similarity calculation result of the g-th grouped sub-feature of the reference image and the i-th source image, express and Element-wise multiplication of .

[0021] Furthermore, step S3.3 is specifically as follows: Using shallow 3D CNN to learn pixel-level visibility weights of source images relative to reference images ,use and Calculate the initial cost volume , , Indicates the shape The specific calculation process is as follows:

[0022] in, express and Element-wise multiplication of .

[0023] Furthermore, step 4 is specifically as follows: S4.1. Frequency feature self-similarity learning; Set the size to The window of the frequency feature After matrix mapping and reshaping, we get and ; When scale k=2, frequency characteristics Depend on Obtained by bilinear upsampling; Then calculate the attention weight matrix between each element in each window based on frequency perception ;

[0024] in, and Represent the Query and Key matrices in the local dynamic window self-attention mechanism, Indicates the shape Tensor of , Represent the height and width of the image respectively, C represents the number of channels, represents the normalized exponential function; S4.2. Frequency-aware cost volume propagation; The initial cost volume at the current scale Reshape into Value matrix , Indicates the shape tensor of ; Then reshape the frequency-aware cost body by the following formula: :

[0025] in, It means reshaping; Frequency-aware cost With the initial cost body Cascaded and then processed through two 3D convolutional layers to generate the updated cost volume ; After 3D U-Net regularization network processing, the probability volume is obtained ,in, Indicates the shape tensor of ; Calculate the depth hypothesis corresponding to the position with the highest probability and obtain the depth estimate for each pixel.

[0026] Furthermore, in step S4.2, the depth hypothesis corresponding to the position with the highest probability is calculated using a winner-takes-all strategy, as shown below:

[0027] in, The coordinates are Depth estimation at are the horizontal and vertical coordinates of the two-dimensional coordinates respectively; Indicates the depth assumption when maximizing the function ; express exist The value at .

[0028] Beneficial effects of the present invention: 1. The present invention constructs a feature fusion module based on frequency information enhancement and global geometric structure embedding, and designs a dual-branch frequency component enhancement module to process the wavelet transform to obtain lossless approximate low-frequency components and high-frequency components of the input image, so as to improve the global consistency and local detail expression of depth estimation.

[0029] 2. This method projects the enhanced multi-scale features onto the same viewing angle, designs a dynamic grouping decision module to dynamically group the features and perform correlation calculations to obtain the initial pixel cost volume. The dynamic grouping decision module quantifies and models the semantic correlation between each feature channel and each group, and dynamically groups them based on this correlation. Compared to average grouping, this method significantly improves the semantic consistency and structural rationality of the groups, enhances the synergy of intra-group processing, and effectively enhances the accuracy and robustness of depth estimation while maintaining network efficiency. It exhibits superior performance, especially in scenes with weak textures, severe occlusion, varying illumination, color distortion, and drastic parallax changes.

[0030] 3. The frequency-aware cost aggregation module of the present invention uses the local dynamic window self-attention mechanism to embed the frequency information of the image into the cost body, enhance the frequency perception ability of the cost body, and then guide the model to adaptively aggregate the cost according to the attention weight matrix between the elements in the window, avoiding information redundancy while capturing richer contextual information.

[0031] 4. The method of the present invention is accurate, flexible and robust, and provides effective technical support for multi-view stereo reconstruction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the structure of the FA-MVS network model in an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of an embodiment of the present invention; Figure 3 Schematic diagram of the process of a feature fusion module based on frequency information enhancement and global geometric structure embedding in an embodiment of the present invention; Figure 4Schematic diagram of the process of the frequency-aware cost aggregation module in an embodiment of the present invention; Figure 5 This is a reconstruction effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation of the present invention is based on a frequency-aware multi-view stereo matching network model (FA-MVS network model), such as Figure 1 As shown, the FA-MVS network model includes a feature fusion module based on frequency information enhancement and global geometric structure embedding, a dynamic grouping decision module associated with channel-group, and a frequency-aware cost aggregation module. The process of this embodiment is as follows: Figure 2 As shown, the specific steps include: Step 1, image acquisition; The multi-view images of the target to be reconstructed are collected by the camera. The multi-view images include N images, where N≥2, and the camera parameters corresponding to each image are recorded. The camera parameters include camera pose and intrinsic parameters. In the present invention, when traversing all images, each image is taken as a reference image in turn, and the rest are taken as the reference image. The image is taken as the corresponding source image, and the rotation matrix of the source image camera relative to the reference image camera is recorded as , the translation matrix is ​​recorded as , the camera intrinsic parameters of the reference image and the source image are recorded as and ,in, is the sequence number of the source image, which takes the value of 1,...,N-1.

[0034] Step 2: Calculate multi-scale frequency perception features; The present invention sets up a feature fusion module based on frequency information enhancement and global geometric structure embedding, including a feature extraction module and a dual-branch frequency component enhancement module (DFE module), extracts multi-scale features, and then traverses all images, taking each image as a reference image in turn, and taking the rest as the reference image. The images are used as the corresponding source images, and the scales are obtained respectively. When the reference image Multi-scale frequency perception features The process of this step is as follows Figure 3 As shown, the specific steps include: 2.1. Feature extraction; The feature extraction module uses the existing three-layer feature pyramid module (FPN module) to extract the multi-scale features of each reference image and the corresponding source image, which are respectively denoted as and ,in, Indicates scale, Indicates the Zhang Yuan image at scale The following features, Specifically, at scale When , the multi-scale feature size is usually 1 / 4, 1 / 2 and 1 times the image size, and the corresponding number of channels are {32, 16, 8} respectively.

[0035] In other embodiments of the present invention, the feature pyramid module may use two or four layers, or variants of the feature pyramid, such as ASPP, ASFF, and FPT, to extract multi-scale features for all images. The number of channels can be set based on the scale; the higher the resolution at the corresponding scale, the fewer the channels.

[0036] 2.2. Perform two wavelet transform layer decompositions and two-branch frequency component enhancement on the reference image in sequence; specifically, the following steps are included: 1) Haar wavelet transform layer decomposition; Reference Images After the Haar wavelet transform layer, it is decomposed into low-resolution approximate low-frequency components and high-frequency components:

[0037] in, represents the approximate low-frequency component when k=1, Respectively represent the high-frequency components in the horizontal, vertical and diagonal directions when k=1, Represents the reference image Perform Haar wavelet transform layer decomposition.

[0038] Haar wavelet transform layer decomposition can separate frequency information losslessly. The spatial information in the image is losslessly mapped to approximate low-frequency and high-frequency components, i.e., four low-resolution frequency sub-bands are obtained, thereby better preserving fine-grained texture details and global contours.

[0039] The present invention enhances the global long-range dependency of the approximate low-frequency components by setting the Transformer structure through steps 2) to 4).

[0040] 2) Back projection; The reference image is transformed into The two-dimensional coordinates of each pixel in Convert to ray direction in 3D space , and then generates a scale-invariant high-dimensional spatial position encoding for each component of the ray direction , as shown in the following formula:

[0041]

[0042] in, Respectively represent the coordinates of the ray direction in the X, Y, and Z directions in three-dimensional space, represents the inverse projection matrix, Represents the intrinsic parameters of the camera corresponding to the reference image, R represents the rotation matrix of the camera corresponding to the reference image, and t represents the translation matrix of the camera corresponding to the reference image. represents the frequency factor, is a learnable scale factor used to ensure the consistency of encoding at different scales.

[0043] 3) Feature mapping; Encoding high-dimensional spatial positions With low frequency characteristics Add, and then map the features to the multi-head attention space through linear transformation to get the query ,key Sum tensor , ,in, represents the number of heads in the multi-head attention space, Indicates the number of channels corresponding to each head in the multi-head attention space, Indicates the shape Tensor of , Represents the height and width of the image respectively.

[0044] At this time, the low-frequency characteristics .

[0045] 4) Global self-attention calculation; Perform global self-attention calculation ( ), get the enhanced low-frequency fusion features when scale k=1 , as shown in the following formula:

[0046] in, represents the normalized exponential function as the activation function; express The transpose of .

[0047] In this process, global feature information is implicitly encoded through high-dimensional spatial position encoding and global geometric enhancement, and the mutual relations between different channels are emphasized.

[0048] 5) Enhancement of high frequency components; For high frequency components , introduces two layers of convolution operations with residual connections to enhance the expression of local details, while more efficiently capturing and associating contextual semantics to obtain enhanced high-frequency features , as shown in the following formula:

[0049] in, represents residual convolution, Indicates a cascade operation.

[0050] 6) Feature stitching; The enhanced low-frequency fusion features and enhanced high-frequency features are spliced ​​along the channel dimension and extracted by 1×1 point-by-point convolution to obtain the scale k Frequency characteristics when , to further remove redundant channel information, as shown below:

[0051] in, Represents 1 × 1 point-wise convolution.

[0052] 7) High-resolution frequency feature generation; According to the method of steps 1) to 6), according to the scale k Frequency characteristics when , get the frequency characteristics when scale k=0 Specifically: After the Haar wavelet transform layer, it is decomposed into low-resolution approximate low-frequency components and high-frequency components, as shown below:

[0053] in, represents the approximate low-frequency component when k=0, Respectively represent the high-frequency components in the horizontal, vertical and diagonal directions when k=0, Represents the frequency characteristics Perform Haar wavelet transform layer decomposition.

[0054] At the same time, Perform inverse projection and feature mapping, in feature mapping , through linear transformation Mapped to the multi-head attention space, the query is obtained ,key Sum tensor , .

[0055] Then, the frequency features at scale k=0 are obtained through global self-attention calculation, high-frequency component enhancement and feature splicing .

[0056] 2.3. Feature fusion; The frequency features of the reference image at scale k=0 and 1 are fused through convolution operation. and the corresponding multi-scale features , get the frequency perception characteristics when scale k=0,1 .

[0057] When scale k=0,1, the frequency components are embedded after the dual-branch frequency component enhancement in step 2.2, which not only enhances the sensitivity to subtle frequency changes but also maintains spatial geometric consistency, thus helping to achieve robust cost matching.

[0058] Step 3, calculate the initial cost volume; Through the dynamic grouping decision module of channel-group association, the multi-scale features of the source image at the current scale are transformed using homography transformation. Frequency perception features projected to the corresponding scale or multi-scale features of the reference image At the same viewing angle, correlation calculation is performed to obtain the initial cost volume of each pixel under different depth assumptions. The details are as follows: 3.1. Depth sampling; Depth hypothesis for each pixel in N images of the current scale Perform uniform sampling to obtain a depth hypothesis set , ,in, Indicates the number of samples of the depth hypothesis, m is the serial number of the depth hypothesis corresponding to each pixel. The value of the number of samples of the depth hypothesis can be set as needed, and the scale When there is no prior knowledge of the corresponding depth of the pixel, it is necessary to take more, scale There is already a scale The depth information obtained when is used as a priori, so less is taken and the scale is And so on.

[0059] scale When , the sampling interval is the distance range from the target to be measured in the scene to the optical center of each camera, which is measured by the rangefinder. Uniform sampling is performed in the inverse depth space of the sampling interval, and the number of samples is 32.

[0060] scale When the depth is estimated based on the scale k=0 Calculate the sampling interval [ ], and sample evenly within the sampling interval, with a sampling number of 8.

[0061] scale When the depth is estimated based on the scale k=1 Calculate the sampling interval [ ], and sample evenly within the sampling interval, with a sampling number of 4.

[0062] in, and Respectively represent scale Time-depth hypothesis set The interval between depth sampling, and Respectively represent scale The sampling interval scale factor when .

[0063] When scale k=0,1, the frequency perception features of the reference image are Copy M times in the depth dimension to obtain the reference feature volume at the current scale When scale k=2, the multi-scale features of the reference image are Copy M times in the depth dimension to obtain the reference feature volume at the current scale .

[0064] Through differentiable homography transformation, the pixels of the reference image are transformed As a benchmark, the multi-scale features of each source image are obtained In depth assumption The corresponding pixel , as shown in the following formula: ] in, represents the inverse operation of the camera intrinsic parameters of the reference image, Indicates the indivual The intrinsic parameters of the camera.

[0065] right Perform the above operation on all pixels at all positions in the image to obtain the feature volume of the source image. , Represents the feature volume of the i-th source image.

[0066] 3.2. Feature grouping; Using the dynamic grouping decision module (GDN module) and The channels are divided into Group, perform correlation calculation, the correlation calculation is as follows: Dynamic grouping decision module for reference feature body Perform depth-wise separable convolution and dilated convolution to enhance local context and obtain enhanced reference feature volume , specifically:

[0067] in, represents point-wise convolution, is the dilated convolution, is a depth-wise separable convolution.

[0068] use convolution( ) will enhance the reference feature Mapped to a single-channel space-depth selective weight and normalized by the Softmax function to obtain the reference image normalized weight :

[0069] in, Indicates the shape Tensor of .

[0070] Then, normalize the weights to the reference image With reference feature Perform element-wise multiplication and add residual connections to emphasize the strongly correlated parts in the depth space to generate the modulated feature volume , as shown below:

[0071] in, express and Element-wise multiplication of .

[0072] The modulated feature Perform spatial compression and generate affinity matrix through fully connected layers ,in, Indicates the shape Tensor; C represents the number of channels; M represents the number of samples of the depth hypothesis; The elements in , indicating that The first one under the deep assumption Channel and The degree of correlation between groups; c=1,...,C; m=1,...,M; g=1,...,G.

[0073] According to the degree of association between channels and groups, that is, affinity matrix ,Will Reorganize by group, Indicates the shape The tensor of W, obtains the sub-features of the g-th group of the source image :

[0074] in, express In the The first one under the deep assumption channels.

[0075] Similarly, the sub-features of the g-th group of the reference image are calculated .

[0076] The following formula is used to perform correlation calculation for each sub-feature of the reference image and the corresponding source image, and the correlation calculation results of all sub-features of the i-th source image are recorded as .

[0077]

[0078] in, represents the similarity calculation result of the g-th grouped sub-feature of the reference image and the i-th source image, express and Element-wise multiplication of .

[0079] The channel grouping strategy widely adopted in existing MVS methods is often an even split, where all feature channels are equally divided into multiple subgroups. This strategy ignores the semantic differences in the information contained in different channels: in MVS, different channels may focus on features such as edge contours, repetitive texture regions, geometric structure cues, or disparity trends. Even splitting and grouping may group channels with completely different perceptual tasks and semantic emphases, resulting in misleading feature interactions and intra-group information interference, reducing the accuracy of cost volume construction and, consequently, compromising the quality of multi-view stereo reconstruction.

[0080] To address these issues, the present invention applies a dynamic grouping decision module based on channel-group associations, quantitatively models the semantic correlation between each feature channel and each group, and dynamically groups the groups based on this correlation. Compared to the average partitioning strategy, this method significantly improves the semantic consistency and structural rationality of the groups, enhances the synergy of intra-group processing, and effectively enhances the accuracy and robustness of depth estimation while maintaining network efficiency. It exhibits particularly superior performance in scenes with weak textures, severe occlusion, or drastic parallax changes.

[0081] 3.3. Calculate the initial cost volume; Using shallow 3D CNN to learn pixel-level visibility weights of source images relative to reference images ,use and Calculate the initial cost volume under different depth assumptions , , Indicates the shape The specific calculation process is as follows:

[0082] in, express and Element-wise multiplication of .

[0083] Some areas are visible in the reference image but are occluded in the source image, so the visibility weight of the source image relative to the reference image is calculated. The accuracy of depth estimation can be guaranteed.

[0084] Step 4, depth estimation; The local dynamic window self-attention mechanism is used to embed the frequency information at the current scale into the initial cost body, so that the frequency changes can be captured more finely during the cost aggregation process, thereby enhancing the frequency perception ability of the model.

[0085] The frequency-aware cost aggregation module (FCA module) includes a frequency feature self-similarity learning module and a frequency-aware cost volume propagation module. Directly applying the traditional global attention calculation is redundant and will bring a large computational burden. To this end, the present invention designs a local dynamic window self-attention mechanism to capture important information associations at different frequencies. Specifically, the size is set to Window, frequency characteristics Include Group window.

[0086] 4.1. Frequency feature self-similarity learning; Set the size to The window of the current scale Perform matrix mapping and reshaping to obtain the corresponding scales and , Indicates the shape Tensor of , and Represent the query and key matrices in the local dynamic window self-attention mechanism. The matrix mapping includes 2D convolution Conv2D, batch normalization BN, and linear rectification function ReLU.

[0087] When scale k=2, frequency characteristics Depend on Obtained by bilinear upsampling.

[0088] The self-similarity of the frequency features in each group of windows is calculated by the following formula to obtain the attention weight matrix between the elements in each window based on frequency perception: , which contains important geometric relationships within the frequency characteristics:

[0089] 4.2. Frequency-aware cost volume propagation; Under different depth assumptions, the cost volume shares similar context information. Therefore, the initial cost volume at the current scale Reshape into Value matrix , Indicates the shape Then each position of the initial cost volume is guided by the frequency-aware attention weight and the window is aggregated. Local context information, reshape ( ) to obtain the frequency-aware cost body , as shown below:

[0090] The set window size is positively correlated with the resolution of the initial cost volume, ensuring that richer context information can be captured when processing the high-resolution initial cost volume.

[0091] Frequency-aware cost With the initial cost body Cascade, and then process through two 3D convolution layers (including 3D convolution Conv3D, batch normalization BN and linear rectification function ReLU) to generate the updated cost volume ,like Figure 4 shown.

[0092] After 3D U-Net regularization network processing, the probability volume is obtained ,in, Indicates the shape The basic unit of the 3D U-Net regularization network is 3D CNN.

[0093] The winner-takes-all strategy (WTA) is used to calculate the depth hypothesis corresponding to the position with the highest probability, that is, to obtain the depth estimate of each pixel, as shown below:

[0094] in, The coordinates are Depth estimation at are the horizontal and vertical coordinates of the two-dimensional coordinates respectively; Indicates the depth assumption when maximizing the function , this is public knowledge; express exist The value at is a three-dimensional probability body, yes The index of .

[0095] Step 5: multi-view stereo reconstruction; After the image is processed by the trained FA-MVS network model through the above steps, a multi-view depth estimation is generated. Back-projection to three-dimensional space, and verification and screening are performed through photometric consistency and geometric consistency constraints to construct a dense three-dimensional point cloud representation of the target scene and complete multi-view stereo reconstruction. Figure 5 , which is a reconstruction effect diagram obtained by performing multi-view stereo reconstruction using the present invention.

[0096] The training of FA-MVS network model uses existing database images. During the training process, the present invention introduces edge perception loss and The loss constrains the difference between the depth estimate and the known ground truth depth map, effectively guiding the network training process. Among them, the edge-aware loss focuses more on the consistency between the depth prediction and the true depth value at edges (such as object outlines and sudden changes). This makes the depth prediction of sudden changes in foreground and background, such as object outlines, more accurate, thereby optimizing the stereo reconstruction effect at edges. The details are as follows: Edge-aware loss : ; loss: ; Overall loss : .

[0097] in, represents the Sobel gradient operator, represents the weight coefficient, Represents the true depth value.

Claims

1. A multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation, characterized in that: The following steps are involved: S1, collect multi-view images of the target to be reconstructed, the multi-view images include N images, where N≥2, record the camera parameters corresponding to the image; take each image as a reference image, and take the rest as the reference image. images as their corresponding source images; S2, calculates multi-scale frequency-aware features; S2.1, extract the multi-scale features of each reference image and the corresponding source image, denoted as and ,in, is the sequence number of the source image, with values ​​ranging from 1,...,N-1, and k represents the scale; S2.2, by wavelet transform layer decomposition and dual-branch frequency component enhancement, the scale Frequency characteristics of the reference image when ; S2.3, Fusion Scale Frequency characteristics of the reference image when and the corresponding multi-scale features , and obtain the frequency perception characteristics of the corresponding scale ; S3, calculate the initial cost volume; Using homography transformation, the multi-scale features of the source image at the current scale are transformed Frequency perception features projected to the corresponding scale or multi-scale features of the reference image At the same perspective, perform correlation calculation to obtain the initial cost volume at the current scale ; S4, using the local dynamic window self-attention mechanism to embed the frequency information into the initial cost body, and obtain the frequency-aware cost body , based on frequency-aware cost and the initial cost body Obtaining a probability body , calculate the depth hypothesis corresponding to the position with the highest probability, and obtain the depth estimate of each pixel at the current scale; S5, back-projects the depth estimate at scale k=2 into three-dimensional space, constructs a dense three-dimensional point cloud representation of the target scene, and completes multi-view stereo reconstruction.

2. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 1, characterized in that: Step S2.2 is specifically as follows: S2.2.

1. Wavelet transform layer decomposition; Reference image Perform wavelet transform layer decomposition to obtain approximate low-frequency components and high-frequency components when the scale k=1, wherein the high-frequency components include high-frequency components in horizontal, vertical and diagonal directions; S2.2.

2. Back projection; The reference image is transformed into The two-dimensional coordinates of each pixel in the image are converted into a ray direction in three-dimensional space, and then a scale-invariant high-dimensional spatial position encoding is generated for each component of the ray direction; S2.2.

3. Feature mapping; Will Add the low-frequency features of the current scale, and then map the features to the multi-head attention space through linear transformation to obtain query, key and value tensors; The low-frequency feature is calculated based on the approximate low-frequency component; S2.2.

4. Global self-attention calculation; Perform global self-attention calculation based on query, key and value tensors to obtain enhanced low-frequency fusion features; S2.2.

5. Enhancement of high frequency components; The high-frequency components are processed through two layers of convolution operations with residual connections to obtain enhanced high-frequency features; S2.2.

6. Feature stitching; The enhanced low-frequency fusion features and enhanced high-frequency features are spliced ​​along the channel dimension, and the frequency features at scale k=1 are obtained by point-by-point convolution. ; S2.2.

7. Generation of high-resolution frequency features; Frequency characteristics Perform wavelet transform layer decomposition to obtain the approximate low-frequency component and high-frequency component when scale k=0. Follow the method of steps S2.2.2 to S2.2.6 to obtain the frequency characteristics when scale k=0. .

3. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 2, characterized in that: In step S2.2.1, the wavelet transform layer decomposition is a Haar wavelet transform layer decomposition; In step S2.2.2, the high-dimensional spatial position encoding is specifically implemented by the following formula: ； ； in, Respectively represent the coordinates of the ray direction in the X, Y, and Z directions in three-dimensional space, express The inverse projection matrix, Represents the intrinsic parameters of the camera corresponding to the reference image, R represents the rotation matrix of the camera corresponding to the reference image, and t represents the translation matrix of the camera corresponding to the reference image. represents the frequency factor, is the learnable scale factor, represents the direction of the ray in three-dimensional space, Represents high-dimensional spatial position encoding; In step S2.2.3, the low-frequency features Specifically: When scale k=1, ,in, Represents the approximate low-frequency component when k=1; When scale k=0, ,in, Represents the approximate low-frequency component when k=0.

4. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 3, characterized in that: In step S2.2.5, the enhanced high-frequency features are obtained by: ； in, represents the enhanced high-frequency features when scale k=1, represents residual convolution, Indicates cascade operation, Respectively represent the high-frequency components in the horizontal, vertical and diagonal directions when the scale k=1; In step S2.2.6, frequency characteristics Specifically obtained through the following methods: ； in, represents 1 × 1 point-by-point convolution, Represents the enhanced low-frequency fusion features when scale k=1.

5. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 4, characterized in that: Step S3 is specifically as follows: S3.1, depth assumption for each pixel in N images at the current scale Perform uniform sampling based on frequency perception features , get the reference feature volume at the current scale ; Through differentiable homography transformation, the pixels of the reference image are transformed As a benchmark, the multi-scale features of each source image are obtained In depth assumption The corresponding pixel , based on each pixel of the source image , get the feature volume of the source image , Represents the feature volume of the i-th source image; S3.2, the reference feature body at the current scale and the feature volume corresponding to the source image The channels are divided into Group, perform correlation calculation on the sub-features of each group of the reference image and its corresponding source image, and obtain the correlation calculation results of all group sub-features of the i-th source image ; S3.3, obtaining the visibility weight of the source image relative to the reference image ,use and Calculate the initial cost volume.

6. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 5, characterized in that: In step S3.1, the depth hypothesis The sampling intervals are as follows: scale When , the sampling interval is the distance range from the target to be measured in the scene to the optical center of each camera; scale When , the sampling interval is [ ]; scale When , the sampling interval is [ ]; in, represents the depth estimation when scale k=0, represents the depth estimation when scale k=1, and Represents scale Time-depth hypothesis set The interval between depth sampling, and Represents scale The sampling interval scale factor when ; Number of samples for depth hypothesis exist ,2 when decreasing successively; The reference feature Calculated as follows: When scale k=0,1, the frequency perception features of the reference image at the current scale are Copy M times in the depth dimension to obtain the corresponding reference feature body ; When scale k=2, the multi-scale features of the reference image at the current scale are Copy M times in the depth dimension to obtain the corresponding reference feature body ; The pixel It is obtained by the following formula: ； in, represents the inverse operation of the camera intrinsic parameters of the reference image, Indicates the indivual The intrinsic parameters of the camera, The rotation matrix of the source image's camera relative to the reference image's camera, Represents the translation matrix of the source image's camera relative to the reference image's camera.

7. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 6, characterized in that: Step S3.2 is specifically as follows: S3.2.

1. and The channels are divided into Group; for reference feature Perform depth-wise separable convolution and dilated convolution to obtain enhanced reference feature volumes , specifically: ； in, represents point-wise convolution, is the dilated convolution, It is a depth-wise separable convolution; S3.2.

2. Use Convolution will enhance the reference feature volume Mapped to a single-channel space-depth selectivity weight and normalized to obtain the reference image normalized weight ,in, Indicates the shape Tensor of , Represents the height and width of the image respectively; Normalize weights to the reference image With reference feature Perform element-wise multiplication and add residual connections to generate the modulated feature volume , specifically: ； in, express and Element-wise multiplication of ; The modulated feature Perform spatial compression and generate affinity matrix through fully connected layers ,in, Indicates the shape Tensor, C represents the number of channels; S3.2.

3. According to affinity matrix ,Will Regroup according to the group, among which, Indicates the shape The tensor of W is used to obtain the sub-features of each group of the source image. : ； in, express In the The first one under the deep assumption channels, Indicates the mth depth hypothesis Channel and The degree of correlation between groups, c=1,...,C, m=1,...,M, g=1,...,G; Similarly, the sub-features of the g-th group of the reference image are calculated ; S3.2.

4. Perform correlation calculation for each sub-feature of the reference image and the corresponding source image using the following formula, and record the similarity calculation result of all sub-features of the i-th source image as ; ； in, represents the similarity calculation result of the g-th grouped sub-feature of the reference image and the i-th source image, express and Element-wise multiplication of .

8. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 7, characterized in that: Step S3.3 is specifically as follows: Using shallow 3D CNN to learn pixel-level visibility weights of source images relative to reference images ,use and Calculate the initial cost volume , , Indicates the shape The specific calculation process is as follows: ； in, express and Element-wise multiplication of .

9. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to any one of claims 1 to 8, characterized in that: Step 4 is as follows: S4.

1. Frequency feature self-similarity learning; Set the size to The window of the frequency feature After matrix mapping and reshaping, we get and ; When scale k=2, frequency characteristics Depend on Obtained by bilinear upsampling; Then calculate the attention weight matrix between each element in each window based on frequency perception ; ； in, and Represent the Query and Key matrices in the local dynamic window self-attention mechanism, Indicates the shape Tensor of , Represent the height and width of the image respectively, C represents the number of channels, represents the normalized exponential function; S4.

2. Frequency-aware cost volume propagation; The initial cost volume at the current scale Reshape into Value matrix , Indicates the shape tensor of ; Then reshape the frequency-aware cost body by the following formula: : ； in, It means reshaping; Frequency-aware cost With the initial cost body Cascaded and then processed through two 3D convolutional layers to generate the updated cost volume ; After 3D U-Net regularization network processing, the probability volume is obtained ,in, Indicates the shape tensor of ; Calculate the depth hypothesis corresponding to the position with the highest probability and obtain the depth estimate for each pixel.

10. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 9, characterized in that: In step S4.2, the depth hypothesis corresponding to the position with the highest probability is calculated using a winner-takes-all strategy, as shown below: ； in, The coordinates are Depth estimation at are the horizontal and vertical coordinates of the two-dimensional coordinates respectively; Indicates the depth assumption when maximizing the function ; express exist The value at .

Images