Multi-view stereo 3d reconstruction method based on cross-scale correction

Abstract

The application relates to a multi-view stereo three-dimensional reconstruction method based on cross-scale correction, and belongs to the field of computer vision.The method comprises the following steps: acquiring a plurality of input images which are divided into reference images and source images, extracting feature maps of different scales through an FPN, and embedding a cross-scale feature fusion module in the FPN; constructing a multi-view three-dimensional cost volume according to camera parameters of the reference images and the source images and a hypothetical depth plane; supplementing the three-dimensional cost volume by using a global semantic information supplementing module, performing regularization processing on the optimized cost volume, and generating a depth probability distribution volume; calculating a depth map according to the depth probability distribution volume, and reconstructing a point cloud through the depth map.The application can effectively solve the problems of missing spatial relationship and blurred details and edges in the cross-scale feature fusion process in the current FPN, and the problem that it is difficult to obtain correct depth due to a large depth range and a large number of depth planes in the rough stage in most current MVS methods.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and relates to a multi-view stereo three-dimensional reconstruction method based on cross-scale correction. BACKGROUND

[0002] Multi-view stereo image reconstruction is an important research direction in the field of computer vision, aiming to reconstruct a dense three-dimensional point cloud from a series of images taken from multiple angles. This technology has shown great application value in many fields, such as in the field of intelligent driving, helping autonomous driving systems to perceive surrounding objects in real time, significantly improving driving safety; in medical applications, assisting doctors in better positioning of diseases, greatly improving the accuracy of diagnosis and treatment; in virtual reality and augmented reality fields, significantly improving the user's immersion and interaction experience. In summary, multi-view stereo image reconstruction technology reconstructs the three-dimensional structure of the real world with high precision, promoting the intelligentization and digitization process of transportation, medical treatment, virtual reality and other fields, and has important social value and practical significance.

[0003] Traditional multi-view stereo image three-dimensional reconstruction methods include object space voxel method, seed point growth method and image space multi-view depth map reconstruction method. The object space voxel method has large calculation amount and high memory consumption, the seed point growth method is sensitive to initial points and is easily affected by noise. The method based on depth map converts the point cloud reconstruction into a depth estimation problem, which not only can directly generate dense point cloud to avoid complex topological modeling, but also can more completely reconstruct surface details. However, in the case of dealing with complex scenes, large changes in viewing angle or serious noise interference, traditional multi-view stereo image three-dimensional reconstruction methods are prone to artifacts and mismatching problems. The three-dimensional reconstruction algorithm based on deep learning can better handle these problems through powerful feature learning ability. MVSNet first proposes an end-to-end multi-view stereo image reconstruction network, the key of this method is to first extract deep visual image features, then construct a 3D cost volume on the reference image through differentiable homography transformation, apply 3D convolution to regularize the cost volume and then regress the depth. Since MVSNet constructs a 3D cost volume and regularizes it, the memory and calculation cost increases cubically with the increase of volume resolution, so it is significantly limited when generating high-resolution results. In order to solve this problem, subsequent research proposes a method based on cascaded framework.

[0004] The core idea of the cascaded framework is to construct a cost volume based on a feature pyramid and estimate depth by stage-wise optimization. In the multi-stage processing, the framework gradually narrows down the depth search range of the current stage by using the depth prediction of the previous stage. At the same time, by gradually increasing the resolution of the cost volume and adaptively adjusting the depth interval, the high-resolution depth map is finally reconstructed in a coarse-to-fine manner. There are two problems in the current cascaded depth prediction model. First, the network usually uses a feature pyramid network (FPN) for feature extraction. It is difficult to ensure the semantic integrity of high-level features and the detail fidelity of low-level features in FPN, which will lead to uneven expression ability of multi-scale features and problems of spatial relationship loss and detail ambiguity in the cross-scale feature fusion process. Second, the depth range is large and the depth plane is multiple in the rough stage, which is difficult to obtain the correct depth, leading to subsequent error amplification and reduced robustness. Therefore, how to efficiently solve the above problems has become a core challenge of multi-view stereo three-dimensional reconstruction. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a multi-view stereo three-dimensional reconstruction method based on cross-scale correction. The method effectively solves the problems of spatial relationship loss and detail ambiguity caused by cross-scale feature fusion in FPN, and balances the transmission of high-level semantics and low-level detail information. The method also alleviates the difficulty of classification or regression depth caused by low-texture regions and the large depth range and multiple depth planes in the rough stage, significantly improving the robustness and accuracy of the algorithm.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] A multi-view stereo three-dimensional reconstruction method based on cross-scale correction, the method comprising the following steps:

[0008] S1, obtaining a plurality of input images divided into reference images and source images, and extracting feature maps of different scales;

[0009] S2, fusing multi-scale feature maps through a cross-scale feature fusion module, which at least includes feature upsampling, feature fusion, feature alignment and attention fusion operations;

[0010] S3, according to the camera parameters of the reference image and the source image, mapping the source image features to the reference image coordinate system by homography transformation and plane scanning method, and constructing a three-dimensional cost volume;

[0011] S4, using a global information supplement module to aggregate the three-dimensional cost volume, which at least includes cost volume expansion, global feature extraction, feature filtering and cost volume fusion processing;

[0012] S5, performing regularization processing on the optimized cost volume to generate a depth probability distribution volume; calculating a first depth map according to the depth probability distribution volume;

[0013] S6, taking a depth plane around the first depth map, performing the steps of S3 and S5 to obtain a second depth map; taking a depth plane around the second depth map, performing the steps of S3 and S5 again to obtain a final depth map, and finally generating a point cloud from the depth map.

[0014] Further, in step S1, input N RGB pictures, one of which is a reference picture and the remaining N-1 are source pictures; the N RGB pictures are processed by a feature extraction network sharing parameters to generate three scale feature maps F1∈R C ×H×W 、 and wherein the feature extraction network uses a convolutional neural network as a feature extractor, first, an original scale feature F1 is extracted from an original RGB image with a size of 3×H×W; then the original scale feature is applied to a convolutional layer with a specified step size for down-sampling to obtain a feature map F 1 / 2 ; then the low scale feature map F 1 / 2 is applied to a convolutional layer with a preset step size for down-sampling to obtain a feature map F 1 / 4 .

[0015] Further, in step S2, at least the following steps are included:

[0016] S21, performing bilinear interpolation up-sampling on the low scale feature F to obtain F adjusting the low resolution of the low scale feature to the same high resolution as the high scale feature F ; then performing convolution operations on the low resolution and high resolution features respectively to adjust the channel number and feature representation of the features;

[0017] S22, concatenating the processed low resolution feature and high resolution feature in the channel dimension, and then generating a set of offset and attention weights required for feature fusion through convolution;

[0018] S23, generating an offset grid coordinate for the low resolution feature according to the generated offset, and performing spatial resampling on the low resolution feature through the grid;

[0019] S24, weighting and fusing the aligned low resolution feature and high resolution feature according to the attention weight generated by the feature fusion unit and using 1+tanh activation, and finally outputting the fused feature map. Cross-scale fusion modules are used in (F 1 / 4 , F 1 / 2 ) and (F 1 / 2 , F1).

[0020] Further, in step S22, the offset is used for aligning the low-resolution features, and the attention weight is used for controlling the fusion ratio of the low-resolution and high-resolution features, which is calculated as follows:

[0021]

[0022] where f k×k,x→y represents the convolution of the channel number from x to y with the kernel size of k x k, and concat(·) represents the concatenation in the channel dimension.

[0023] Further, in step S23, the operation process of feature alignment according to the offset is as follows:

[0024] Let the spatial coordinates of each position on the feature map be {(1, 1), (1, 2),..., (H, W)}, and the learned 2D offset map be Δ ∈ R 2×H×W , then the feature output after resampling is:

[0025]

[0026] where U h，w represents the feature value calculated by bilinear interpolation, F h′w′ represents the feature value at the (h', w') position, (h, w) is the offset coordinate, and (h', w') is the integer grid position closest to the target point (h, w); and are the offsets learned in the vertical and horizontal directions, respectively.

[0027] Further, in step S24, the process of obtaining the fusion feature according to the attention weight and the feature after the alignment is as follows:

[0028]

[0029] where O is the fusion feature, is pixel-by-pixel multiplication, F l is the low-scale feature after processing, and F is the high-scale feature after processing. h is the low-scale feature after processing, and F is the high-scale feature after processing, and α and β are the weights of the low-scale feature and the high-scale feature after activation, respectively, where the low-scale feature attention weight is w l , and the high-scale feature attention weight is w h , and the corresponding activation weights are obtained by activating them through the 1+tanh activation function.

[0030] Further, in step S3, the construction of the cost volume includes the following steps:

[0031] S31, obtain the internal and external parameters of the input image: the internal parameter K includes the focal length fx , f y , optical center c x , c y , the extrinsic parameters include rotation matrix R, translation vector t;

[0032] S32, homography transformation: for each assumed depth plane d, the homography transformation from the source image plane to the reference image camera coordinate system is:

[0033]

[0034] where H i (d) refers to the homography transformation between the feature map of the i-th view and the reference feature map at depth d. In addition, K i , R i , t i respectively refer to the camera intrinsic, rotation and translation matrix of the i-th view. When i = 1, it represents the reference reference image. n represents the principal axis of the reference camera.

[0035] S33, plane scanning method for feature mapping: define a depth range, and select a series of discrete depth values in the range, for each depth value, use the corresponding homography transformation to map the feature map of the source image to the parallel plane of the reference image, to obtain a series of mapped feature bodies at different depth planes.

[0036] S34, feature matching and cost calculation: for each pixel in the reference image, calculate the matching cost between it and the features mapped from the source image by variance, feature bodies, namely:

[0037]

[0038] where V1 represents the feature body of the reference image, and the rest are feature bodies of any source image.

[0039] Further, in step S4, it at least includes the following steps:

[0040] S41, five-dimensional cost body Flatten to two-dimensional features in the channel and depth dimensions CV 2d Obtain CV 2d with global semantic information through multi-layer convolution and filtering operations;

[0041] S42, embed CV 2d into CV 3d : extend CV 2d in the depth dimension to make it consistent with the depth number of CV 3d , then generate weight attention (As ):

[0042] ConvNext(F) = df 7×7，n→2n (f 1×1，2n→2n (f 1×1，2n→n (F)))+F

[0043] GSE(F) = f 1×1,m→n (ConvNext(F) ×3 )

[0044] CV 2d-expand =expand(GSE(CV) 2d ))

[0045] A s =σ(CV) 3d +CV 2d-expand )

[0046] In the formula, df k×k,x→y (·) represents a depthwise separable convolution with a kernel size of k×k and number of channels from x to y, expand(·) represents expansion in the depth direction, and σ(·) represents the sigmoid activation function;

[0047] S43, CV through attention weights 2d-expand Perform filtering operation on CV 2d-expand Perform filtering:

[0048]

[0049] S44, CV 3d and CV 2d-expand The data is stitched together along the channel direction and then mapped through a convolution to finally generate the optimized cost volume.

[0050] Furthermore, in step S5, the optimized cost volume is regularized using 3DU-Net. This is achieved by encoding it using multiple 3D convolutional layers and then decoding it using 3D deconvolution, filtering out noise contamination and reducing errors caused by object occlusion. The probability P(d) of M depth planes d is calculated using Softmax, and the predicted depth D1 for each pixel is calculated as follows:

[0051]

[0052] Where d m Let D1 be the probability value of the m-th depth plane, and D1 is the first depth map.

[0053] Further, on the basis of the D1 depth map, the range of the assumed plane is dynamically adjusted, higher resolution features are input on the basis of the range, a new cost volume is reconstructed, the steps S4 and S5 are repeated to obtain a second depth map D2, and the above operations are repeated again on the basis of D2 to obtain a final depth map;

[0054] The probability volume generated by the cost volume of the above process is supervised, and the total loss function thereof is:

[0055]

[0056] In the formula, P (d) (p) represents the probability value of the depth d, represents the value closest to the depth assumed plane, {p v} represents the pixel subset with the effective ground truth;

[0057] After obtaining the final depth map, the depth value of the depth map and the camera internal parameter are used to convert each pixel coordinate in the image plane into a three-dimensional point in the camera coordinate system, the position of the three-dimensional point is verified and optimized in combination with the multi-view information, the redundant or error points are eliminated, and all effective three-dimensional points are integrated to form a dense point cloud structure, and the three-dimensional reconstruction is completed.

[0058] The present application has the advantages that:

[0059] The present application effectively solves the problems of spatial relationship missing and detail blurring in the cross-scale feature fusion process in the current FPN by optimizing the cross-scale feature fusion. The spatial continuity and detail expression ability of the feature map are enhanced, and the transmission effect of high-level semantic information and low-level detail information is balanced. Especially in the high-resolution feature generation process, the blurring effect is effectively suppressed, and the detail recovery ability is ensured, thereby providing higher quality input features for subsequent tasks.

[0060] The present application effectively solves the problem that it is difficult to obtain correct depth due to large depth range and multiple depth planes in the rough stage by introducing a global semantic information supplement module. The module effectively alleviates the classification and regression difficulties caused by low-texture areas and large-range depth, thereby significantly improving the overall depth estimation performance and enhancing the robustness and accuracy of the algorithm.

[0061] Other advantages, objects, and features of the present application will be apparent to those skilled in the art from the following specification and will be learned from practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which:

[0063] Figure 1 A schematic diagram of the overall flow of the multi-view stereo 3D reconstruction method based on cross-scale correction of the present application;

[0064] Figure 2 A schematic diagram of the structure of the cross-scale feature fusion module OFAM of the present application;

[0065] Figure 3 A schematic diagram of the structure of the global information supplement module GSIM of the present application.

[0066] Figure 4 A schematic diagram of the qualitative comparison of the embodiments of the present application and other existing methods in the DTU verification set scan10 and scan48;

[0067] Figure 5 The reconstruction results of the embodiments of the present application on the Tanks and Temples data set. DETAILED DESCRIPTION

[0068] The embodiments of the present application are described below through specific and concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied through other different specific embodiments, and each detail in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0069] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable for those skilled in the art that some known structures and their descriptions in the drawings may be omitted.

[0070] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0071] Please refer to Figures 1-5 , a multi-view stereo 3D reconstruction method based on cross-scale correction.

[0072] Embodiments

[0073] The present embodiment gives a detailed implementation process of a multi-view stereo 3D reconstruction method based on cross-scale correction, Figure 1 is the overall flowchart of the multi-view stereo 3D reconstruction method based on cross-scale correction of the present application, which at least includes the following steps:

[0074] S1, a plurality of input images divided into reference images and source images are obtained, and feature maps of different scales are extracted;

[0075] S2, the multi-scale feature maps are fused through a cross-scale feature fusion module, which at least includes feature upsampling, feature fusion, feature alignment and attention fusion operations;

[0076] S3, according to the camera parameters of the reference image and the source image, the source image features are mapped to the reference image coordinate system through homography transformation and plane scanning method, and a three-dimensional cost volume is constructed;

[0077] S4, the three-dimensional cost volume is aggregated by a global information supplement module, which at least includes cost volume unfolding, global feature extraction, feature filtering and cost volume fusion processing;

[0078] S5, the optimized cost volume is subjected to regularization processing to generate a depth probability distribution volume; a first depth map is calculated according to the depth probability distribution volume.

[0079] S6, a depth plane is taken around the first depth map, and the steps S3 and S5 are performed to obtain a second depth map. A depth plane is taken around the second depth map, and the steps S3 and S5 are performed to obtain a final depth map, and finally a point cloud is generated through the depth map.

[0080] In step S1 of this embodiment, the input includes N RGB images, one of which is a reference image, and the remaining N-1 images are source images. The N RGB images are processed by a feature extraction network with shared parameters to generate feature maps at three scales: original image size (1), 1 / 2, and 1 / 4 scale. The feature extraction network uses a convolutional neural network as the feature extractor. First, it extracts the original scale feature F1∈R from the original RGB image with dimensions H×W×C. C×H×W Then, the original scale features are downsampled using a convolutional layer with a stride of 2 and followed by two more convolutional layers to obtain... Then, the 1 / 2 scale feature map is downsampled again using a convolutional layer with a stride of 2 and followed by two more convolutional layers to obtain...

[0081] In step S2 of this embodiment, a cross-scale feature fusion module (OAFM, Offset-Aware Fusion Module) is used to fuse feature map information of different scales, and its structure is as follows: Figure 2 As shown, it includes at least a feature alignment unit, a feature fusion unit, and an attention fusion unit, and specifically includes the following steps:

[0082] S21, Low-scale features Bilinear interpolation upsampling was performed to obtain Adjust the low resolution of low-scale features to match that of high-scale features. The same high resolution is achieved; subsequently, convolution operations are performed on the low-resolution and high-resolution features respectively, and the feature F is adjusted. h Number of channels C h arrive Number of channels C l This makes it suitable for subsequent fusion operations.

[0083] S22. The processed low-resolution and high-resolution features are concatenated along the channel dimension and input into the feature fusion unit. The feature fusion unit generates a set of offsets and attention weights required for feature fusion through multiple convolutions. The offsets are used to align the low-resolution features, and the attention weights are used to control the fusion ratio of low-resolution and high-resolution features. The calculation method is as follows:

[0084]

[0085] S23, in the feature alignment unit, an offset grid coordinate is generated for the low-resolution feature according to the offset generated by the feature fusion unit. Through the grid, the low-resolution feature is resampled in space (alignment operation) to ensure its spatial position consistency with the high-resolution feature, thereby eliminating the spatial difference between the multi-scale features. Specifically, assuming that the spatial coordinates of each position on the feature map are {(1, 1), (1, 2),..., (H, W)}, and the learned 2D offset map is Δ ∈ R 2 ×H×W , then the resampled feature output is:

[0086]

[0087] In the formula, U h,w represents the feature value calculated by bilinear interpolation, F h′w′ represents the feature value at the (h', w') position, (h, w) is the offset coordinate, and (h', w') is the integer grid position closest to the target point (h, w). respectively, are the offsets learned in the vertical direction (row direction) and the horizontal direction (column direction).

[0088] S24, in the attention fusion unit, the attention weight generated by the feature fusion unit is used to weight and fuse the aligned low-resolution feature and the high-resolution feature:

[0089]

[0090] In the formula, O is the fused feature, is pixel-by-pixel multiplication, F l ' is the processed low-scale feature, F' h is the processed high-scale feature, and α and β are the weights of the activated low-scale feature and high-scale feature respectively, wherein the low-scale feature attention weight is w l , and the high-scale feature attention weight is w h , which are activated by the 1+tanh activation function to obtain the corresponding activation weights.

[0091] The attention weight is used to dynamically balance the importance of the two, which can not only retain the global information in the low-resolution feature, but also highlight the detailed information of the high-resolution feature, and finally output the fused feature map.

[0092] In step S3 of the embodiment, the construction of the cost volume includes the following steps:

[0093] S31, obtaining the internal and external parameters of the input image; the internal parameter K includes the focal length f x , fy , optical center c x , c y , the extrinsic parameters include rotation matrix R and translation vector t. These parameters are used to determine the relative position and pose between two cameras.

[0094] S32, compute homography transformation: for each hypothesized depth plane d, compute the homography transformation from the source image plane to this depth plane in the reference image camera coordinate system. This transformation maps points on the source image to a plane in the reference image camera coordinate system.

[0095]

[0096] where H i (d) refers to the homography transformation between the feature map of the i-th view and the reference feature map at depth d. In addition, K i , R i , t i respectively refer to the camera intrinsic, rotation and translation matrix of the i-th view. When i = 1, it represents the reference reference image. n represents the principal axis of the reference camera.

[0097] S33, plane scanning method for feature mapping: define a depth range and select a series of discrete depth values within the range, for each depth value, use the corresponding homography transformation to map the feature map of the source image to the parallel plane of the reference image, to obtain a series of mapping feature bodies at different depth planes.

[0098] S34, feature matching and cost calculation: for each pixel in the reference image, calculate the matching cost between it and the features mapped from the source image by variance. V1, … V N ∈R C×D×H×W feature bodies, where V1 represents the feature body of the reference image, and the rest are feature bodies of any source image.

[0099]

[0100] In step S4 of the embodiment, the input cost volume and context features are fused and optimized by a global information supplement module (GSIM, Global Semantic Injection Module) to improve the quality of geometric features and the accuracy of depth estimation. The structure diagram of the global information supplement module is as follows Figure 3As shown, it includes: cost volume unfolding: unfolding the generated cost volume along the channel (C) and depth (D) dimensions, global semantic feature extraction: extracting global semantic features through a ConvNeXt network, semantic feature filtering: expanding in the depth (D) dimension, fusing with the cost volume, and filtering the global semantic features after Sigmoid activation to enhance the expression of effective information. Cost volume fusion: further fusion of the cost volume after global feature filtering to ensure the integrity and geometric consistency of the fused cost volume. Specifically, it includes at least the following steps:

[0101] S41, input cost volume is a five-dimensional tensor containing depth dimension information. First, it is adjusted to flatten its depth dimension into a two-dimensional feature This facilitates subsequent convolution processing. Then, CV 2d Through multiple layers of convolution and filtering operations, CV 2d with global semantic information is obtained.

[0102] S42, fuse CV 2d with CV 3d . Expand CV 2d in the depth (D) dimension to match the depth (D) of the cost volume. Then, generate weight attention (A s ), filter CV 2d through the attention mechanism. The attention weight determines which features should be strengthened or weakened, thereby achieving more effective feature fusion.

[0103] ConvNext(F)=df 7×7，n→n (f 1×1，n→2n (f 1×1，2n→n (F)))+F

[0104] GSE(F)=f 1×1，n→2n (ConvNext(F) ×3 )

[0105] cv 2d-expand =expand(GSE(cv 2d ))

[0106] A s =σ(cv 3d +cv 2d-expand )

[0107] where df k×k,x→y (·) represents a depth separable convolution with a convolution kernel size of k x k, channel number from x to y, expand(·) represents expansion from the depth direction, and σ(·) represents a sigmoid activation function;

[0108] S43, adjust the CV 2d application attention weights, so that different parts of the CV 2d are dynamically adjusted according to the context information. This step ensures that the details of the geometric features can be consistent with the context information, avoiding the loss or error of depth information. By A s the CV 2d-expand is dynamically adjusted.

[0109]

[0110] S44, the fused features are further strengthened by an additional aggregation operation (3D convolution processing), and the spatial and geometric continuity of the features is further strengthened, and finally the optimized cost volume features are generated. These features can provide higher quality input for subsequent tasks (such as depth estimation or 3D reconstruction). The 3D U-Net is used to optimize the cost volume, mainly to filter the matching error points caused by occlusion, light, weak texture and other problems. At the same time, further aggregate the matching correct information, improve the quality and robustness of the cost volume.

[0111] In step S5 of the embodiment, the optimized cost volume is subjected to 3D U-Net regularization processing, which is encoded by using multiple 3D convolution layers, and then decoded by using 3D deconvolution, to filter the noise pollution of the cost volume and reduce the error caused by object occlusion. The probability P(d) of M depth planes d is calculated by Softmax, and the predicted depth D1 of each pixel is calculated as:

[0112]

[0113] where d m is the probability value of the mth depth plane;

[0114] Further, in step S6: on the basis of the obtained depth map, input higher resolution features. According to the depth estimation result of the last stage, dynamically adjust the range of the hypothesis plane (i.e. reduce the depth search range), and reconstruct a new cost volume based on the range, repeat steps S4 and S5. Obtain the depth map, repeat twice.

[0115] The probability volume generated by the 3-layer cost volume is supervised, and the total loss function is:

[0116]

[0117] In the formula, P (d)(p) represents the probability value of depth d, and D1 is a depth map.

[0118] The total loss function of the probability body generated by the 3-layer cost volume is:

[0119]

[0120] In the formula, P (d) (p) represents the probability value of depth d, represents the value closest to the depth hypothesis plane, and {p v} represents a subset of pixels with valid ground truth.

[0121] After obtaining the final depth map, the depth value of the depth map and the camera internal parameter are used to convert each pixel coordinate in the image plane into a three-dimensional point in the camera coordinate system. The positions of the three-dimensional points are verified and optimized in combination with multi-view information, and redundant or incorrect points are eliminated. All valid three-dimensional points are integrated to form a dense point cloud structure, and three-dimensional reconstruction is completed.

[0122] By optimizing the upsampling method, the problems of missing spatial relationship and blurred details in the cross-scale feature fusion process in the current FPN are effectively solved. The spatial continuity and detail expression ability of the feature map are enhanced, and the transmission effect of high-level semantic information and low-level detail information is balanced. Especially in the high-resolution feature generation process, the blurring effect is effectively suppressed, and the detail recovery ability is ensured, thereby providing higher quality input features for subsequent tasks.

[0123] The present application effectively solves the problem of large depth range and multiple depth planes in the rough stage, which makes it difficult to obtain correct depth. The module effectively alleviates the classification and regression difficulties caused by low-texture areas and large-range depth, thereby significantly improving the overall depth estimation performance and enhancing the robustness and accuracy of the algorithm.

[0124] The present embodiment gives the performance results of the multi-view stereo three-dimensional reconstruction method based on cross-scale correction of the present application and other existing methods on the DTU data set. The lower the result is, the better. The results are as shown in Figure 4 and Table 1:

[0125] Table 1

[0126] Method Acc. (mm) Comp. (mm) Overall. (mm) Gipuma 0.283 0.873 0.578 COLMAP 0.400 0.664 0.532 Patchmatchnet 0.427 0.277 0.352 Effi-MVSNet 0.321 0.313 0.317 CasMVSNet 0.325 0.385 0.355 MVSNet 0.396 0.527 0.462 Our 0.360 0.264 0.312

[0127] Quantitative results of the DTU dataset as shown in Table 1, the lower the value, the better the result. Compared with some other deep learning methods (such as standard MVSNet) and traditional multi-view stereo methods, the present application is superior to all the above methods in integrity and overall score. In addition, as shown in the reconstruction results of the Tanks and Temples dataset using the method of the present application, it effectively solves the problems of spatial relationship loss and detail blur in the cross-scale feature fusion process in the current FPN. The spatial continuity and detail expression ability of the feature map are enhanced, and the transmission effect of high-level semantic information and low-level detail information is balanced. Especially in the high-resolution feature generation process, the blur effect is effectively suppressed, and the detail recovery ability is ensured. Figure 5

[0128] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.​

Claims

1. A multi-view stereo 3D reconstruction method based on cross-scale correction, characterized in that: The method includes the following steps: S1. Obtain several input images, divided into reference images and source images, and extract feature maps at different scales. , and ; S2. Multi-scale feature maps are fused through a cross-scale feature fusion module, which includes at least feature upsampling, feature fusion, feature alignment and attention fusion operations; S3. Based on the camera parameters of the reference image and the source image, the features of the source image are mapped to the coordinate system of the reference image through homography transformation and planar scanning method to construct a three-dimensional cost volume; S4. The three-dimensional cost volume is aggregated using a global information supplementation module, which includes at least cost volume expansion, global feature extraction, feature filtering and cost volume fusion processing. S5. Regularize the optimized cost volume to generate a depth probability distribution volume; calculate the first depth map based on the depth probability distribution volume. S6. Take a depth plane around the first depth map, and perform steps S3 and S5 to obtain the second depth map; take a depth plane around the second depth map, and perform steps S3 and S5 again to obtain the final depth map; finally, generate a point cloud from the depth map. Step S2 includes at least the following steps: S21, Low-scale features Bilinear interpolation upsampling was performed to obtain Adjusting the low resolution of low-scale features to match that of high-scale features The same high resolution is achieved; then convolution operations are performed on the low-resolution and high-resolution features respectively, adjusting the number of channels and feature representation. S22. The processed low-resolution features and high-resolution features are concatenated along the channel dimension, and then a set of offsets and attention weights required for feature fusion are generated through convolution. S23. Generate an offset grid coordinate for the low-resolution feature based on the generated offset, and resample the low-resolution feature in space using the grid. S24. Based on the attention weights generated by the feature fusion unit, and using... Activation involves weighted fusion of aligned low-resolution and high-resolution features, ultimately outputting a fused feature map. and Employ cross-scale fusion modules; Step S4 includes at least the following steps: S41, The three-dimensional cost body Flatten the channel and depth dimensions into two-dimensional features. ; By employing multiple convolutional and filtering operations, global semantic information is obtained. ; S42, will Embedded into China: To Expand in the depth dimension to make it compatible with... The depths are consistent, and then weighted attention is generated ( ). ): In the formula, Indicates the kernel size as The number of channels from arrive Depth-separable convolution, Indicates the kernel size as The number of channels from arrive convolution, Indicates expansion in the depth direction. express Activation function; S43, Using attention weights Perform filtering operation, for Perform filtering: S44, will and The data is stitched together along the channel direction and then mapped through a convolution to finally generate the optimized cost volume.

2. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 1, characterized in that: In step S1, the input consists of N RGB images, one of which is a reference image and the remaining N-1 images are source images. The N RGB images are processed by a feature extraction network with shared parameters to generate feature maps at three scales. , and , The feature extraction network uses a convolutional neural network as the feature extractor. First, it extracts features from a feature size of... Original scale features are extracted from the original RGB image. ; Then, the original scale features are downsampled using a convolutional layer with a specified stride to obtain the feature map. ; Then the low-scale feature map Apply a convolutional layer with a preset stride to downsample again to obtain the feature map. .

3. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 2, characterized in that: In step S22, the offset is used to align low-resolution features, and the attention weight is used to control the fusion ratio of low-resolution and high-resolution features. The calculation method is as follows: in, This indicates that the data is spliced ​​according to the channel dimension.

4. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 1, characterized in that: In step S23, the feature alignment operation based on the offset is as follows: Let the spatial coordinates of each location on the feature map be... The learned 2D offset map is The resampled feature output is as follows: In the formula, This represents the eigenvalues ​​calculated using bilinear interpolation. Indicates in Eigenvalues ​​at location, These are the offset coordinates. To represent the target point The nearest integer grid position; These are the offsets learned in the vertical and horizontal directions, respectively.

5. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 1, characterized in that: In step S24, the process of obtaining the fused features based on the attention weights and subsequent features is as follows: In the formula, O represents the fusion feature. For pixel-by-pixel multiplication, for Processed low-scale features for Processed high-scale features, and These are the low-scale feature weights and high-scale feature weights after activation, respectively, where the low-scale feature attention weight is... The attention weights for high-scale features are: The corresponding activation weights are obtained by activating them respectively using the 1+tanh activation function.

6. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 1, characterized in that: In step S3, the construction of the cost body includes the following steps: S31. Obtain the intrinsic and extrinsic parameters of the input image: Intrinsic parameters Including focal length , light center The extrinsic parameters include the rotation matrix. Translation vector ; S32. Homography Transformation: For each assumed depth plane The homography transformation from the source image plane to the reference image camera coordinate system is as follows: In the formula It refers to the first Feature map and depth of each view Homography transformation between reference feature maps at the location; , , They refer to the first The camera intrinsics, rotation and translation matrices for each view, when When n represents the reference image, n represents the principal axis of the reference camera; S33. Feature mapping using planar scanning method: Define a depth range and select a series of discrete depth values ​​within the range. For each depth value, use the corresponding homography transformation to map the feature map of the source image onto the parallel plane of the reference image to obtain a series of mapped feature bodies on different depth planes. S34. Feature Matching and Cost Calculation: For each pixel in the reference image, calculate the matching cost between it and the features mapped from the source image using variance. Each feature body, namely: in One represents the feature volume of the reference image, and the rest are feature volumes of any source image.

7. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 1, characterized in that: In step S5, the optimized cost volume is regularized using 3DU-Net. This involves encoding it using multiple 3D convolutional layers and then decoding it using 3D deconvolution to filter out noise contamination and reduce errors caused by object occlusion. Softmax is then used for computation. The probability of a depth plane d Predicted depth for each pixel The calculation is as follows: in For the first The probability value of each depth plane. This is the first depth map.

8. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 7, characterized in that: exist Based on the depth map, the extent of the hypothetical plane is dynamically adjusted. Using this extent as a basis, higher-resolution features are input to reconstruct a new cost volume. Steps S4 and S5 are repeated to obtain the second depth map. ; Again Based on this, repeat the above operations to obtain the final depth map; The total loss function for supervising the probability volume generated from the cost body in the above process is: In the formula, This represents the probability value for depth d. This represents the value closest to the depth hypothesis plane. Represents a subset of pixels that have valid face facts; After obtaining the final depth map, the depth values ​​and camera intrinsic parameters of the depth map are used to convert the coordinates of each pixel in the image plane into a 3D point in the camera coordinate system. Combined with multi-view information, the position of the 3D points is verified and optimized, redundant or erroneous points are eliminated, and all valid 3D points are integrated to form a dense point cloud structure, thus completing the 3D reconstruction.

Images