Multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation

Abstract

The present application relates to image stereo reconstruction method, specifically to multi-view stereo reconstruction method based on frequency perception feature enhancement and cost aggregation. In order to solve the problem that the existing learning-based multi-view stereo reconstruction method lacks frequency perception ability and has limited processing ability for weak texture, noise, illumination change and color distortion, the present application realizes multi-view stereo reconstruction through the steps of acquiring multi-view images, calculating multi-scale frequency perception features, calculating initial cost volume, embedding frequency information into the initial cost volume, calculating depth estimation and back projection in turn. When calculating multi-scale frequency perception features, a double-branch frequency component enhancement module is set to process the lossless approximate low-frequency component and high-frequency component of the input image decomposed by wavelet transform layer, so as to improve the global consistency and local detail expression of depth estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to an image stereo reconstruction method, in particular to a multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation. BACKGROUND

[0002] Multi-view stereo reconstruction provides a non-contact solution for 3D surface reconstruction and measurement, and has been widely used in fields such as medical diagnosis, pose estimation, virtual reality / augmented reality, and cultural heritage protection. In recent years, learning-based multi-view stereo reconstruction (MVS) methods use convolutional neural networks to extract multi-channel image features and aggregate information from different views into a cost volume that measures the confidence of each pixel under different depth hypotheses. Subsequently, a 3D convolutional neural network (3D CNN) is used to regularize the cost volume, eliminate the effects of occlusion and noise, and complete multi-view stereo reconstruction. Compared with traditional methods, this method performs better in challenging areas such as weak texture and mirror reflection.

[0003] Existing learning-based cascaded architecture MVS methods usually construct multi-scale features using feature pyramids to gradually refine the depth hypothesis interval and interval in a coarse-to-fine manner, and finally obtain refined depth estimation. However, existing methods only mine the feature representation of input images and the spatial correlation of each pixel in the spatial domain, and do not explore the difference in modeling ability of different frequency information in complex regions from the frequency domain, resulting in a lack of frequency-aware ability in the generated cost volume and limited processing ability for weak texture, noise, illumination changes, and color distortion. In addition, some methods usually use regular convolution downsampling operations and perform jump connections between multi-scale features when constructing multi-scale features, which may cause loss of important spatial information and semantic ambiguity, affecting the preservation of key details such as boundaries, scales, and textures. At the same time, 3D U-Net regularization networks are used to enhance the consistency of the cost volume to reduce false matches caused by occlusion and illumination changes. Moreover, due to the limited local receptive field, depth information is smoothed and propagated in adjacent regions, causing boundary features to be blurred. Therefore, it is very important to endow the model with frequency-aware ability when constructing multi-scale features and performing cost aggregation, so that it can sensitively capture rich frequency variations. SUMMARY

[0004] The purpose of the present application is to solve the problem of lack of frequency-aware ability and limited processing ability for weak texture, noise, illumination changes, and color distortion in existing learning-based multi-view stereo reconstruction methods, and to provide a multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation.

[0005] To achieve the above-mentioned purpose, the technical solution provided by the present application is as follows:

[0006] A multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation is characterized by the following steps:

[0007] S1: Acquire multi-view images of the target to be reconstructed, including N images where N≥2; record the camera parameters corresponding to each image; use each image as a reference image, and then use the remaining images... Zhang image serves as its corresponding source image;

[0008] S2, calculates multi-scale frequency-sensing features;

[0009] S2.1, extract the multi-scale features of each reference image and its corresponding source image, denoted as follows: and ,in, The index of the source image is 1, ..., N-1, where k represents the scale.

[0010] S2.2, through wavelet transform layer decomposition and dual-branch frequency component enhancement, the scale is obtained. Frequency characteristics of the reference image at that time ;

[0011] S2.3, Fusion Scale Frequency characteristics of the reference image at that time and corresponding multi-scale features To obtain the frequency-sensing features at the corresponding scale ;

[0012] S3, calculate the initial cost body;

[0013] Using homography, multi-scale features of the source image at the current scale are transformed. Frequency-sensing features projected onto the corresponding scale or multi-scale features of the reference image From the same perspective, perform correlation calculations to obtain the initial cost volume at the current scale. ;

[0014] S4. Frequency information is embedded into the initial cost volume using a local dynamic window self-attention mechanism to obtain the frequency-aware cost volume. Frequency-aware cost body and initial cost body Obtaining probability body Calculate the depth hypothesis corresponding to the position with the highest probability, and obtain the depth estimate of each pixel at the current scale;

[0015] S5 backprojects the depth estimate at scale k=2 to three-dimensional space to construct a dense three-dimensional point cloud representation of the target scene, thus completing multi-view stereo reconstruction.

[0016] Further, step S2.2 is specifically:

[0017] S2.2.1. Wavelet transform layer decomposition;

[0018] The reference image is subjected to wavelet transform layer decomposition to obtain an approximate low-frequency component and high-frequency components at scale k = 1, the high-frequency components including high-frequency components in horizontal, vertical and diagonal directions;

[0019] S2.2.2. Inverse projection;

[0020] The two-dimensional coordinates of each pixel in the reference image are converted into a ray direction in three-dimensional space by an inverse projection matrix, and then a scale-invariant high-dimensional space position code is generated for each component of the ray direction;

[0021] S2.2.3. Feature mapping;

[0022] The is added to the low-frequency feature at the current scale, and then the feature is mapped to a multi-head attention space by a linear transformation to obtain query, key and value tensors;

[0023] The low-frequency feature is calculated based on the approximate low-frequency component;

[0024] S2.2.4. Global self-attention calculation;

[0025] Global self-attention calculation is performed based on the query, key and value tensors to obtain an enhanced low-frequency fusion feature;

[0026] S2.2.5. High-frequency component enhancement;

[0027] The high-frequency component is processed by two layers of convolution operations with residual connection to obtain an enhanced high-frequency feature;

[0028] S2.2.6. Feature splicing;

[0029] The enhanced low-frequency fusion feature and the enhanced high-frequency feature are spliced along the channel dimension, and a frequency feature at scale k = 1 is obtained by point-wise convolution extraction .

[0030] S2.2.7. High-resolution frequency feature generation;

[0031] The frequency feature is subjected to wavelet transform layer decomposition to obtain an approximate low-frequency component and high-frequency components at scale k = 0, and the frequency feature at scale k = 0 is obtained according to the method of steps S2.2.2 to S2.2.6. .

[0032] Further, in step S2.2.1, the wavelet transform layer is decomposed into a Haar wavelet transform layer;

[0033] In step S2.2.2, the high-dimensional space position encoding is specifically implemented by the following formula:

[0034]

[0035]

[0036] wherein, respectively represent the coordinates of the ray direction in the X, Y, Z directions in the three-dimensional space, represents the inverse projection matrix of represents the camera intrinsic parameters corresponding to the reference image, R represents the rotation matrix of the camera corresponding to the reference image, and t represents the translation matrix of the camera corresponding to the reference image, represents a frequency factor, is a learnable scale factor, represents the ray direction in the three-dimensional space, represents the high-dimensional space position encoding;

[0037] In step S2.2.3, the low-frequency feature is specifically:

[0038] When the scale k = 1, wherein, represents the approximate low-frequency component when k = 1;

[0039] When the scale k = 0, wherein, represents the approximate low-frequency component when k = 0.

[0040] Further, in step S2.2.5, the enhanced high-frequency feature is specifically obtained by the following way:

[0041]

[0042] wherein, represents the enhanced high-frequency feature when the scale k = 1, represents residual convolution, represents a concatenation operation, respectively represent the high-frequency components in the horizontal, vertical and diagonal directions when the scale k = 1;

[0043] In step S2.2.6, the frequency feature is specifically obtained by the following way:

[0044]

[0045] wherein, denotes a 1 × 1 point-wise convolution, denotes the enhanced low-frequency fused feature at scale k = 1.

[0046] Further, step S3 is specifically:

[0047] S3.1, obtaining a depth hypothesis of each pixel in the N images at the current scale performing uniform sampling based on the frequency-aware feature to obtain a reference feature volume at the current scale ;

[0048] obtaining a multi-scale feature of each source image through a differentiable homography transformation, taking the pixel of the reference image as a reference obtaining a feature volume of the source image based on each pixel of the source image corresponding to the depth hypothesis ; , denotes the feature volume of the i-th source image;

[0049] S3.2, dividing the channels of the reference feature volume at the current scale and the feature volume of the corresponding source image into groups, performing correlation calculation on the sub-features of each group of the reference image and the corresponding source image, to obtain the correlation calculation result of all group sub-features of the i-th source image ;

[0050] S3.3, obtaining the visibility weight of the source image relative to the reference image , and calculating the initial cost volume using and .

[0051] Further, in step S3.1, the sampling interval of the depth hypothesis is as follows:

[0052] when the scale , the sampling interval is the distance range from the target to be measured in the scene to the optical center of each camera;

[0053] when the scale , the sampling interval is ;

[0054] when the scale , the sampling interval is ;

[0055] wherein,​​ denotes the depth estimation at scale k = 0, denotes the depth estimation at scale k = 1, and denote the depth hypothesis set at scale the interval of depth sampling in and denote the sampling interval scale factor at scale

[0056] the number of samples of depth hypothesis decreases successively at ,2;

[0057] the reference feature volume is calculated by:

[0058] at scale k = 0, 1, the frequency-aware feature of the reference image at the current scale is copied M times in the depth dimension to obtain the corresponding reference feature volume ;

[0059] at scale k = 2, the multi-scale feature of the reference image at the current scale is copied M times in the depth dimension to obtain the corresponding reference feature volume ;

[0060] the pixel is obtained by:

[0061]

[0062] wherein, denotes the inverse operation of the camera intrinsic parameter of the reference image, denotes the camera intrinsic parameter of the th denotes the rotation matrix of the camera of the source image relative to the camera of the reference image, denotes the translation matrix of the camera of the source image relative to the camera of the reference image.

[0063] Further, step S3.2 is specifically:

[0064] S3.2.1. divide the channels of and into groups; perform depth separable convolution and dilated convolution on the reference feature volume to obtain the enhanced reference feature volume , specifically:

[0065] ​​​​​

[0066] where, denotes point-wise convolution, is dilated convolution, is depth-wise separable convolution;

[0067] S3.2.2. Using convolution, the enhanced reference feature volume is mapped to single-channel spatial-depth selective weights and normalized to obtain reference image normalized weights where, denotes a tensor of shape , and denote the height and width of the image, respectively;

[0068] The reference image normalized weights are element-wise multiplied with the reference feature volume and a residual connection is added to generate the modulated feature volume , which is specifically:

[0069]

[0070] where, denotes element-wise multiplication of and ;

[0071] The modulated feature volume is spatially compressed and an affinity matrix is generated through a fully connected layer, where, denotes a tensor of shape , and C denotes the number of channels;

[0072] S3.2.3. According to the affinity matrix , the is recombined according to the grouping, where, denotes a tensor of shape W, and the sub-feature of each group of the source image is obtained:

[0073]

[0074] where, denotes the th channel under the th depth hypothesis, denotes the association degree between the th channel and the th group under the th depth hypothesis, c = 1,..., C, m = 1,..., M, g = 1,..., G;​

[0075] Similarly, the sub-feature of the gth group of the reference image is calculated as ;

[0076] S3.2.4. The similarity of each group of the reference image and the corresponding source image is calculated by correlation, and the similarity of all group sub-features of the ith source image is recorded as ;

[0077]

[0078] wherein, represents the similarity of the gth group sub-feature of the reference image and the ith source image, represents and element-wise multiplication.

[0079] Further, step S3.3 is specifically:

[0080] Using a shallow 3D CNN to learn the visibility weight of the source image at the pixel level relative to the reference image , the initial cost volume is calculated by and , , represents a tensor with a shape of , and the specific calculation process is as follows:

[0081]

[0082] wherein, represents and element-wise multiplication.

[0083] Further, step 4 is specifically:

[0084] S4.1. Frequency feature self-similarity learning;

[0085] A window with a size of is set, and the frequency feature is mapped and reshaped by a matrix to obtain and ;

[0086] When the scale k=2, the frequency feature is obtained by bilinear up-sampling from ;

[0087] Then, the attention weight matrix between each element in each window based on frequency perception is calculated as ; ​

[0088]

[0089] wherein, and represent Query and Key matrices in the local dynamic window self-attention mechanism, respectively, denotes a tensor with shape ; denote the height and width of the image, respectively, and C denotes the number of channels, denotes a normalized exponential function;

[0090] S4.2. Cost volume propagation based on frequency perception;

[0091] Reshape the initial cost volume at the current scale to a Value matrix , denotes a tensor with shape ;

[0092] Reshape to get the frequency-aware cost volume by the following formula:

[0093]

[0094] wherein, denotes the reshaping;

[0095] Concatenate the frequency-aware cost volume with the initial cost volume , and then process it through two 3D convolution layers to generate the updated cost volume ; After processing by the 3D U-Net regularization network, the probability volume is obtained, wherein, denotes a tensor with shape ;

[0096] Calculate the depth hypothesis corresponding to the position of maximum probability to obtain the depth estimation of each pixel.

[0097] Further, in step S4.2, the depth hypothesis corresponding to the position of maximum probability is calculated by a winner-takes-all strategy, as shown below:

[0098]

[0099] wherein, denotes the depth estimation at coordinates ; and denote the horizontal and vertical coordinates of the two-dimensional coordinates, respectively; denotes the depth hypothesis when maximizing the function representing at the value.

[0100] The beneficial effects of the present application are:

[0101] 1. The present application constructs a feature fusion module based on frequency information enhancement and global geometric structure embedding, designs a double-branch frequency component enhancement module to process the lossless approximate low-frequency component and high-frequency component of the input image obtained by wavelet transform, so as to improve the global consistency and local detail expression of depth estimation.

[0102] 2. The present application projects the enhanced multi-scale features to the same perspective, designs a dynamic grouping decision module to dynamically group the features and perform correlation calculation to obtain the initial cost volume of the pixels. The dynamic grouping decision module quantitatively models the semantic correlation between each feature channel and each group, and dynamically groups according to the correlation result. Compared with average division grouping, the present method significantly improves the semantic consistency and structural rationality of grouping, and improves the cooperative effect of in-group processing, while ensuring the network efficiency, effectively enhances the depth estimation accuracy and robustness, especially in scenes with weak texture, serious occlusion, light change, color distortion and dramatic disparity change.

[0103] 3. The frequency-aware cost aggregation module of the present application uses a local dynamic window self-attention mechanism to embed the frequency information of the image into the cost volume, enhances the frequency-aware ability of the cost volume, and then guides the model to adaptively aggregate the cost according to the attention weight matrix between elements in the window, avoids information redundancy, and captures more rich context information.

[0104] 4. The method of the present application is accurate, flexible and robust, and provides effective technical support for multi-view stereo reconstruction tasks. BRIEF DESCRIPTION OF DRAWINGS

[0105] Figure 1 is a structural schematic diagram of the FA-MVS network model in the embodiment of the present application;

[0106] Figure 2 is a flowchart of the embodiment of the present application;

[0107] Figure 3 is a flowchart of the feature fusion module based on frequency information enhancement and global geometric structure embedding in the embodiment of the present application;

[0108] Figure 4 is a flowchart of the frequency-aware cost aggregation module in the embodiment of the present application;

[0109] Figure 5 is a reconstruction effect diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0110] The application is based on a multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation, which is based on a frequency-aware multi-view stereo matching network model (FA-MVS network model), as shown in the following formula: Figure 1 The FA-MVS network model includes a feature fusion module based on frequency information enhancement and global geometric structure embedding, a dynamic grouping decision module associated with channel-grouping, and a frequency-aware cost aggregation module. The flow of the embodiment is shown in the following formula: Figure 2 The specific steps include the following steps:

[0111] Step 1, image acquisition;

[0112] Multi-view images of the target to be reconstructed are collected by a camera, and the multi-view images include N images, where N≥2, and camera parameters corresponding to each image are recorded, including camera pose and intrinsic parameters. In this application, when all images are traversed, each image is sequentially taken as a reference image, and the remaining images are taken as corresponding source images. The rotation matrix of the camera of the source image relative to the camera of the reference image is denoted as , the translation matrix is denoted as , and the camera intrinsic parameters of the reference image and the source image are denoted as and , respectively, where is the serial number of the source image, and the value is 1,..., N-1.

[0113] Step 2, calculate multi-scale frequency-aware features;

[0114] The application sets a feature fusion module based on frequency information enhancement and global geometric structure embedding, including a feature extraction module and a double-branch frequency component enhancement module (DFE module). Multi-scale features are extracted, and then all images are traversed, each image is sequentially taken as a reference image, and the remaining images are taken as corresponding source images. When the scale is obtained, the multi-scale frequency-aware features of the reference image are obtained. The flow of this step is shown in the following formula: Figure 3 The specific steps include the following steps:

[0115] 2.1. Feature extraction;

[0116] The feature extraction module uses an existing 3-layer feature pyramid module (FPN module) to extract multi-scale features of each reference image and corresponding source image, denoted as and , respectively, where represents the scale, and represents the Features of the Zhang source image at scales . Specifically, at scales , the multi-scale feature sizes are usually 1 / 4, 1 / 2 and 1 times the image size in turn, and the corresponding channel numbers are {32, 16, 8} respectively.

[0117] In other embodiments of the application, the feature pyramid module can also use 2 layers, 4 layers, etc.; All multi-scale features of the image can also be extracted using variants of the feature pyramid, such as ASPP, ASFF, FPT, etc. The number of channels can be set according to the scale, and the higher the resolution of the corresponding scale, the fewer the number of channels.

[0118] 2.2. Two wavelet transform layer decompositions and double-branch frequency component enhancements are performed on the reference image in turn; specifically including the following steps:

[0119] 1) Haar wavelet transform layer decomposition;

[0120] The reference image is decomposed into low-resolution approximate low-frequency components and high-frequency components by Haar wavelet transform layer decomposition:

[0121]

[0122] wherein, represents the approximate low-frequency component when k=1, respectively represent the high-frequency components in the horizontal, vertical and diagonal directions when k=1, represents Haar wavelet transform layer decomposition on the reference image .

[0123] Haar wavelet transform layer decomposition can separate frequency information losslessly, which maps the spatial information in losslessly to the approximate low-frequency component and the high-frequency component, i.e. four low-resolution frequency subbands are obtained, so that the fine-grained texture details and global contours are better preserved.

[0124] The application sets the Transformer structure through steps 2)~4) to enhance the global long-range dependence of the approximate low-frequency component.

[0125] 2) Inverse projection;

[0126] The two-dimensional coordinates of each pixel in the reference image are converted into the ray direction in three-dimensional space by the inverse projection matrix, and then a scale-invariant high-dimensional space position encoding is generated for each component of the ray direction, as shown in the following formula: ​

[0127]

[0128]

[0129] wherein, respectively represent the coordinate of the ray direction in the X, Y, Z direction of the three-dimensional space, represents the inverse projection matrix, represents the camera intrinsic parameters corresponding to the reference image, R represents the rotation matrix of the camera corresponding to the reference image, and t represents the translation matrix of the camera corresponding to the reference image, represents the frequency factor, is a learnable scale factor used to ensure consistency of encoding at different scales.

[0130] 3) Feature mapping;

[0131] High-dimensional space position encoding is added to the low-frequency feature , and then the feature is mapped to the multi-head attention space through linear transformation to obtain the query , key and value tensors , wherein, represents the number of heads of the multi-head attention space, represents the number of channels corresponding to each head in the multi-head attention space, represents a tensor with a shape of , respectively represent the height and width of the image.

[0132] At this time, the low-frequency feature .

[0133] 4) Global self-attention calculation;

[0134] Global self-attention calculation is performed (Self-Attention) to obtain the enhanced low-frequency fusion feature at scale k = 1, which is specifically shown in the following formula:

[0135]

[0136] wherein, represents a normalized exponential function as an activation function; represents the transpose of .

[0137] In this process, the global feature information is implicitly encoded through high-dimensional space position encoding and global geometry enhancement, and the mutual relationship between different channels is emphasized.

[0138] 5) High-frequency component enhancement;

[0139] For high-frequency components , two layers of convolution operations with residual connection are introduced to enhance the expression ability of local details, and more efficiently capture and associate contextual semantics, obtaining enhanced high-frequency features , as shown in the following formula:

[0140]

[0141] wherein, represents residual convolution, represents cascading operation.

[0142] 6) Feature splicing;

[0143] The enhanced low-frequency fusion features and enhanced high-frequency features are spliced along the channel dimension, and the frequency features at scale k are obtained by 1×1 point convolution extraction , which realizes further removal of redundant channel information, as shown below:

[0144]

[0145] wherein, represents 1 × 1 point convolution.

[0146] 7) High-resolution frequency feature generation;

[0147] According to the method of steps 1) to 6), the frequency features at scale k are obtained from the frequency features at scale k , to obtain the frequency features at scale k=0 . Specifically: for , the Haar wavelet transform layer is decomposed into low-resolution approximate low-frequency components and high-frequency components, as shown below:

[0148]

[0149] wherein, represents the approximate low-frequency component at k=0, respectively represent the horizontal, vertical and diagonal direction high-frequency components at k=0, represents Haar wavelet transform layer decomposition on frequency features .

[0150] At the same time, inverse projection and feature mapping are performed on , and in the feature mapping , the is mapped to the multi-head attention space through linear transformation, to obtain query , key​ and value tensor , .

[0151] Then the frequency feature of scale k=0 is obtained by global self-attention calculation, high-frequency component enhancement and feature splicing .

[0152] 2.3. Feature fusion;

[0153] The frequency features of the reference image of scale k=0 and 1 are fused respectively by convolution operation and the corresponding multi-scale features , to obtain the frequency-aware features of scale k=0 and 1 .

[0154] At scale k=0 and 1, the frequency components are embedded by step 2.2 double-branch frequency component enhancement, which not only enhances the sensitivity to subtle frequency changes, but also maintains spatial geometric consistency, thereby helping to achieve robust cost matching.

[0155] Step 3, calculate the initial cost volume;

[0156] Through the dynamic grouping decision module of channel-group association, the multi-scale features of the source image at the current scale are projected to the same view of the frequency-aware features of the corresponding scale or the multi-scale features of the reference image , and the correlation calculation is performed to obtain the initial cost volume of each pixel under different depth hypotheses. Specifically as follows:

[0157] 3.1. Depth sampling;

[0158] Uniformly sample the depth hypotheses of each pixel in the N images at the current scale to obtain the depth hypothesis set , wherein, indicates the sampling number of depth hypotheses, and m is the serial number of the corresponding depth hypothesis of each pixel. The value of the sampling number of depth hypotheses can be set according to the needs. When scale , there is no any known priori for the corresponding depth of the pixel, so more should be taken, when scale , there is already depth information obtained at scale as priori, so less should be taken, and when scale , and so on.

[0159] Scale The sampling interval is the distance range from the target to the camera center, which is measured by a range finder. Uniform sampling is performed in the inverse depth space of the sampling interval, and the sampling number is 32.

[0160] Scale The depth estimation based on the scale k = 0 The sampling interval is calculated as follows: Uniform sampling is performed in the sampling interval, and the sampling number is 8.

[0161] Scale The depth estimation based on the scale k = 1 The sampling interval is calculated as follows: Uniform sampling is performed in the sampling interval, and the sampling number is 4.

[0162] Wherein, and represent the sampling interval of the depth hypothesis set when the scale is k = 0, and represent the sampling interval scale factor when the scale is k = 1. When the scale k = 0, 1, the frequency-aware feature of the reference image is copied M times in the depth dimension to obtain the reference feature body at the current scale .When the scale k = 2, the multi-scale feature of the reference image is copied M times in the depth dimension to obtain the reference feature body at the current scale

[0163] . Through the differentiable homographic transformation, the multi-scale feature of each source image is obtained based on the pixel of the reference image , and the corresponding pixel under the depth hypothesis is as follows:

[0164] , which is as follows:

[0165] ]

[0166] Wherein, represents the inverse operation of the camera intrinsic parameter of the reference image, represents the camera intrinsic parameter of the i-th source image. The above operation is performed on all pixels in the position to obtain the feature body of the source image ,

[0167] ​​​​​​​The i-th source image.

[0168] 3.2. Feature groupings;

[0169] The channels of and are grouped into groups by a dynamic grouping decision module (GDN module), and the correlation calculation is performed, which is specifically as follows:

[0170] The dynamic grouping decision module performs a depth separable convolution, a dilated convolution, and an enhanced local context on the reference feature body to obtain an enhanced reference feature body , which is specifically as follows:

[0171]

[0172] wherein, denotes a point-wise convolution, is a dilated convolution, and a depth separable convolution.

[0173] The enhanced reference feature body is mapped to a single-channel space-depth selective weight by using a convolution , and is normalized by a Softmax function to obtain a reference image normalized weight :

[0174] wherein,

[0175] denotes a tensor with a shape of . Then, the reference image normalized weight

[0176] is element-wise multiplied with the reference feature body and a residual connection is added to emphasize the strongly correlated part in the depth space, to generate a modulated feature body , which is specifically as follows:

[0177] wherein,

[0178] denotes an element-wise multiplication of and .

[0179] The modulated feature body is spatially compressed, and an affinity matrix is generated by a fully connected layer, wherein, denotes a tensor with a shape of .wherein, denotes the tensor of shape W x C x G; C denotes the number of channels; M denotes the number of depth hypotheses sampled; wherein, denotes the element in the affinity matrix , denotes the degree of association between the th channel and the th group under the th depth hypothesis; c = 1,..., C; m = 1,..., M; g = 1,..., G.

[0180] According to the degree of association between the channels and the groups, i.e., the affinity matrix , the th group of the source image is recombined, denotes the tensor of shape W x G, and the sub-feature of the th group of the source image is obtained as :

[0181]

[0182] wherein, denotes the th channel under the th depth hypothesis. Similarly, the sub-feature of the

[0183] th group of the reference image is calculated as .

[0184] The correlation of all the group sub-features of the th source image is calculated by performing correlation calculation on the group sub-features of the reference image and the corresponding source image, and is denoted as .

[0185]

[0186] wherein, denotes the similarity calculation result of the th group sub-feature of the reference image and the th source image, denotes the element-wise multiplication of and

[0187] The channel grouping strategy widely used in existing MVS methods is to equally divide the groups, i.e., to equally divide all the feature channels into multiple sub-groups. This strategy ignores the semantic difference of the information contained in different channels: in MVS, different channels may respectively focus on edge contour, texture repetitive area, geometric structure clue or disparity change trend, etc. Equal division of groups may divide channels with completely different perception tasks and semantic focuses into the same group, thereby causing misleading feature interaction and intra-group information interference, reducing the accuracy of cost volume construction, and further impairing the quality of multi-view stereo reconstruction.

[0188] To solve the above problems, the application applies a channel-packet association dynamic packet decision module to quantitatively model the semantic correlation between each feature channel and each packet, and dynamically groups according to the correlation result. Compared with the average division strategy, the application significantly improves the semantic consistency and structural rationality of the grouping, enhances the collaborative effect of the in-group processing, effectively enhances the accuracy and robustness of the depth estimation while ensuring the network efficiency, and especially performs better in scenes with weak texture, severe occlusion or dramatic disparity changes.

[0189] 3.3. Calculate the initial cost volume;

[0190] Use a shallow 3D CNN to learn the visibility weight of the source image relative to the reference image at the pixel level , and use and to calculate the initial cost volume under different depth hypotheses , , , which represents a tensor with a shape of , and the specific calculation process is as follows:

[0191]

[0192] , wherein represents the element-wise multiplication of and .

[0193] Some areas can be seen in the reference image, but are occluded in the source image, so the visibility weight of the source image relative to the reference image is calculated to ensure the accuracy of the depth estimation.

[0194] Step 4, depth estimation;

[0195] Use a local dynamic window self-attention mechanism to embed the frequency information at the current scale into the initial cost volume, so that the frequency changes can be captured more carefully in the cost aggregation process, and the frequency perception ability of the model is enhanced.

[0196] The frequency-aware cost aggregation module (FCA module) includes a frequency feature self-similarity learning module and a cost volume propagation module based on frequency awareness. It is worth noting that directly applying traditional global attention calculation on the frequency feature is redundant and will bring a lot of computational burden. Therefore, the application designs a local dynamic window self-attention mechanism to capture important information correlation at different frequencies. Specifically, a window with a size of is set, and the frequency feature contains groups of windows.

[0197] 4.1. Frequency feature self-similarity learning;

[0198] A window with size is set to map and reshape the at the current scale to get and , denotes a tensor with shape , and represent Query and Key matrix in the local dynamic window self-attention mechanism respectively. The matrix mapping includes 2D convolution Conv2D, batch normalization BN and linear rectifier function ReLU.

[0199] At scale k = 2, the frequency feature is obtained by bilinear up-sampling from .

[0200] The self-similarity of frequency feature in each window is calculated by which contains important geometric relationships within the frequency feature:

[0201]

[0202] 4.2. Cost volume propagation based on frequency perception;

[0203] Under different depth assumptions, the cost volume enjoys similar contextual information. Therefore, the initial cost volume at the current scale is reshaped to the Value matrix , denotes a tensor with shape . Then each position of the initial cost volume is guided by the frequency perception based attention weight and aggregates the local contextual information of the window to reshape to get the frequency perception cost volume , which is shown as follows:

[0204]

[0205] The window size is positively related to the resolution of the initial cost volume, which ensures that more rich contextual information can be captured when processing high-resolution initial cost volume.

[0206] The frequency perception cost volume is combined with the initial cost volume Cascade, then processed by two 3D convolution layers (including 3D convolution Conv3D, batch normalization BN and linear rectifier function ReLU) to generate an updated cost volume As shown in Figure 4 .

[0207] After being processed by the 3D U-Net regularization network, the probability volume is obtained , wherein represents a tensor with a shape of . The basic unit of the 3D U-Net regularization network is a 3D CNN.

[0208] The winner-takes-all strategy (WTA) is used to calculate the depth hypothesis corresponding to the position with the maximum probability, that is, to obtain the depth estimation of each pixel, which is specifically as follows:

[0209]

[0210] , wherein represents the depth estimation at coordinates ; x and y respectively represent the horizontal and vertical coordinates of the two-dimensional coordinates; represents the depth hypothesis when the function is maximized , which is known; represents the value of at , because is a three-dimensional probability volume, is the index of .

[0211] Step 5, multi-view stereo reconstruction;

[0212] After the image is processed by the trained FA-MVS network model through the above steps, multi-view depth estimation is generated. The depth estimation when the scale k=2 is projected to the three-dimensional space, and is checked and screened through the photometric consistency and geometric consistency constraints to construct the dense three-dimensional point cloud representation of the target scene, and the multi-view stereo reconstruction is completed. As shown in , it is a reconstruction effect diagram obtained by using the multi-view stereo reconstruction of the present application. Figure 5

[0213] The training of the FA-MVS network model uses existing database images. During the training process, the present application introduces edge-aware loss and ​​The difference between the loss constraint depth estimation and the known ground truth depth map is estimated, thereby effectively guiding the training process of the network. Among them, the edge-aware loss pays more attention to the consistency of the depth prediction and the depth truth at the edge (such as object contour, mutation area), so that the depth prediction of the object contour and other foreground and background mutation areas is more accurate, thereby optimizing the stereo reconstruction effect at the edge. As shown below:

[0214] Edge-aware loss : ;

[0215] Loss: ;

[0216] Total loss : .

[0217] Wherein, represents the Sobel gradient operator, represents the weight coefficient, represents the depth truth.

Claims

1. A multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation, characterized in that, The method comprises the following steps: S1, collecting multi-view images of a target to be reconstructed, the multi-view images comprising N images, where N≥2, and recording camera parameters corresponding to the images; taking each image as a reference image and taking the remaining N-1 images as corresponding source images thereof; images as corresponding source images thereof; S2, calculating multi-scale frequency-aware features; S2.1, extract multi-scale features of each reference image and corresponding source image, respectively denoted as and wherein, is the sequence number of the source image, taking values 1,...,N-1, and k represents the scale. S2.2, by wavelet transform layer decomposition and double-branch frequency component enhancement, obtain scale frequency features of the reference image at the time ; S2.3, fusion scale frequency features of the reference image at the time and corresponding multi-scale features resulting in frequency-aware features at the corresponding scale ; S3, calculating an initial cost volume; using a homography transform, the multi-scale features of the source image at the current scale projected to the corresponding scale of the frequency-aware features or the multi-scale features of the reference image at the same view angle, performing a correlation computation to obtain an initial cost volume at the current scale ; S4. Frequency information is embedded into the initial cost volume using a local dynamic window self-attention mechanism to obtain the frequency-aware cost volume. Frequency-aware cost body and initial cost body Obtaining probability body Calculate the depth hypothesis corresponding to the position with the highest probability, and obtain the depth estimate of each pixel at the current scale; S5, projecting the depth estimation at scale k=2 to the three-dimensional space to construct a dense three-dimensional point cloud representation of the target scene, and completing multi-view stereo reconstruction.

2. The method of claim 1, wherein, Step S2.2 is specifically: S2.2.

1. Wavelet transform layer decomposition; Reference image Wavelet transform layer decomposition is performed to obtain approximate low-frequency components and high-frequency components at scale k=1. The high-frequency components include high-frequency components in the horizontal, vertical and diagonal directions. S2.2.

2. Inverse projection; transforming the two-dimensional coordinates of each pixel in the reference image into a ray direction in three-dimensional space, and then generating a scale-invariant high-dimensional space position encoding for each component of the ray direction; S2.2.

3. Feature mapping; will be described below with reference to the accompanying drawings. In the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it is deemed that such a detailed description will unnecessarily obscure the subject matter of the present disclosure. The following terms are used herein. The low-frequency features of the current scale are added, and then the features are mapped to the multi-head attention space through a linear transformation to obtain query, The low-frequency feature is calculated based on the approximate low-frequency component; S2.2.

4. Global self-attention calculation; The global self-attention calculation is performed based on the query, key and value tensors to obtain an enhanced low-frequency fusion feature; S2.2.

5. High-frequency component enhancement; The high-frequency component is processed by two layers of convolution operations with residual connection to obtain an enhanced high-frequency feature; S2.2.

6. Feature splicing; The enhanced low-frequency fusion features and the enhanced high-frequency features are spliced along the channel dimension, and frequency features at scale k=1 are extracted through point-by-point convolution ; S2.2.

7. High-resolution frequency feature generation; Frequency feature The wavelet transform layer decomposition is performed to obtain the approximation low-frequency component and the high-frequency component at scale k=0, and the frequency feature at scale k=0 is obtained according to the method of steps S2.2.2 to S2.2.6 .

3. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 2, characterized in that: In step S2.2.1, the wavelet transform layer decomposition is Haar wavelet transform layer decomposition; In step S2.2.2, the high-dimensional space position encoding is specifically implemented by the following formula: ； ； wherein, respectively represent the coordinate of the ray direction in the X, Y, Z direction in the three-dimensional space, represents the inverse projection matrix of represents the camera intrinsic parameters corresponding to the reference image, R represents the rotation matrix of the camera corresponding to the reference image, and t represents the translation matrix of the camera corresponding to the reference image, represents a frequency factor, is a learnable scale factor, represents the ray direction in the three-dimensional space, represents a high-dimensional space position encoding; In step S2.2.3, the low-frequency feature Specifically: for scale k = 1, wherein, denotes the approximated low frequency component for k = 1; At scale k = 0, where, denotes the approximated low frequency component at k = 0.

4. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 3, characterized in that: In step S2.2.5, the enhanced high-frequency feature is obtained by the following method: ； wherein, represents the enhanced high frequency feature at scale k = 1, represents residual convolution, represents concatenation operation, respectively represent horizontal, vertical and diagonal direction high frequency components at scale k = 1; In step S2.2.6, the frequency characteristic This is achieved in particular by the following: ； wherein, denotes a 1 × 1 point-wise convolution, denotes the enhanced low-frequency fused feature at scale k = 1.

5. The method of claim 4, wherein, Step S3 is specifically: S3.1, for each pixel in the N images at the current scale, a depth hypothesis Uniform sampling is performed based on the frequency-aware features , obtaining reference feature volumes at the current scale ; by a differentiable homography, to the pixels of the reference image are obtained for each source image Under the depth hypothesis corresponding pixels , based on each pixel of the source image , a feature volume of the source image is obtained , denotes the feature volume of the i-th source image; S3.2, the reference feature volume at the current scale and the feature body of the corresponding source image The channels are divided into For each group of reference images and their corresponding source images, correlation calculations are performed on the sub-features of each group to obtain the correlation calculation results of all group sub-features of the i-th source image. ; S3.3, obtaining the visibility weight of the source image relative to the reference image , using and to calculate an initial cost volume.

6. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 5, characterized in that: In step S3.1, the depth hypothesis The sampling interval for the depth hypothesis is as follows: Scale When the sampling interval is the distance range from the target to be measured in the scene to the optical center of each camera. Scale When the sampling interval is [0, 1]. ] scale When, the sampling interval is [ ]; wherein, denotes the depth estimate at scale k = 0, denotes the depth estimate at scale k = 1, and denote the set of depth hypotheses at scale the interval of depth samples in and denote the interval of samples at scale ;​ Number of samples for the depth hypothesis exist The numbers decrease sequentially from 2 to 3. the reference feature by calculating: the frequency-aware feature of the reference image at the current scale k = 0, 1 copying M times in the depth dimension to obtain the corresponding reference feature body ; At scale k = 2, the multi-scale feature of the reference image at the current scale is obtained In the depth dimension, M times are copied to obtain the corresponding reference feature body ; The pixel By the following formula: ； wherein denotes the inverse operation of the camera intrinsic parameters of the reference image, denotes the camera intrinsic parameters of the denotes the rotation matrix of the camera of the source image relative to the camera of the reference image, denotes the translation matrix of the camera of the source image relative to the camera of the reference image.​​ 7. The method of claim 6, wherein the cost aggregation is performed based on the frequency-aware features. Step S3.2 is specifically: S3.2.

1. The channel of and is divided into groups; the reference feature body is subjected to a depth separable convolution and a dilated convolution to obtain an enhanced reference feature body , specifically: ； wherein, denotes point-wise convolution, is dilated convolution, is depthwise separable convolution; S3.2.

2. Use The enhanced reference feature volume is convolved with the single-channel spatial-depth-selective weights to obtain the reference image normalized weights wherein, represents a tensor of shape , and represents the height and width of the image, respectively; Normalizing weights of reference images With reference to the feature body Performing element-by-element multiplication and adding a residual connection to generate a modulated feature body Specifically: ； wherein represents and element-wise multiplication; The modulated feature body Spatial compression is performed and an affinity matrix is generated through a fully connected layer wherein, denotes a tensor of shape C denotes the number of channels; S3.2.

3. According to the affinity matrix , the grouping is recombined according to the affinity matrix, wherein the shape of the tensor is W, to obtain the sub-feature of each group of the source image : ； wherein, represents at the th depth hypothesis, th channel, represents the degree of association between the th channel and the th group at the mth depth hypothesis, c = 1,..., C, m = 1,..., M, g = 1,..., G; Similarly, the sub-feature of the gth group of the reference image is calculated as ; S3.2.

4. The similarity of all the grouped sub-features of the i-th source image is calculated as ; ； wherein, denotes the similarity computation result of the gth group sub-feature of the reference image and the ith source image, denotes and an element-wise multiplication.

8. The method of claim 7, wherein, Step S3.3 is specifically: Learning pixel-level visibility weights of source images relative to reference images using shallow 3D CNNs , by using and to compute an initial cost volume , , denotes a tensor of shape , computed as follows: ； wherein represents and element-wise multiplication.

9. The method of any one of claims 1-8, wherein, Step 4 is specifically: S4.

1. Frequency feature self-similarity learning; A window of size is set and the frequency features are mapped through a matrix and reshaped to get and ; At scale k = 2, the frequency characteristics By Obtained by bilinear up-sampling; The attention weight matrix between each element in each window based on frequency perception is recalculated ; ； wherein, and represent the Query and Key matrices in the local dynamic window self-attention mechanism, respectively, denotes a tensor of shape , denote the height and width of the image, respectively, and C denotes the number of channels, denotes the normalized exponential function; S4.

2. Frequency-aware cost volume propagation; transforming the initial cost volume at the current scale reshaped into a Value matrix , denotes a tensor of shape ; The frequency-aware cost function is then reshaped by the following equation : ； wherein denotes remodeling; The frequency-aware cost volume is concatenated with the initial cost volume and then processed by two 3D convolutional layers to generate an updated cost volume ; The probability volume is obtained by processing the 3D U-Net regularization network , where denotes a tensor of shape ; The depth hypothesis corresponding to the position with the maximum probability is calculated to obtain the depth estimation of each pixel.

10. The multi-view stereo reconstruction method based on frequency-aware feature enhancement and cost aggregation according to claim 9, characterized in that: In step S4.2, the depth hypothesis corresponding to the position with the maximum probability is calculated by the winner-takes-all strategy, which is specifically as follows: ； wherein, denotes a depth estimate at coordinates denote the horizontal and vertical coordinates, respectively, of a two-dimensional coordinate; denotes a depth hypothesis when maximizing the function ; denotes the value at .​

Images