Multi-view three-dimensional reconstruction method based on cross-scale correction

Abstract

The invention relates to a multi-view three-dimensional reconstruction method based on cross-scale correction, and belongs to the field of computer vision. The method comprises the following steps: acquiring a plurality of input images which are divided into reference images and source images, extracting feature maps of different scales through an FPN, and embedding a cross-scale feature fusion module into the FPN; constructing a multi-view three-dimensional cost body according to the camera parameters of the reference image and the source image and the assumed depth plane; a global semantic information supplementing module is adopted to supplement the three-dimensional cost body, regularization processing is carried out on the optimized cost body, and a depth probability distribution body is generated; and calculating a depth map according to the depth probability distribution body, and reconstructing a point cloud through the depth map. According to the method, the problems of spatial relationship missing and fuzzy details and edges in the cross-scale feature fusion process in the current FPN can be effectively solved, and the problem that correct depth is difficult to obtain due to large depth range and many depth planes in the rough stage in most current MVS methods can be effectively solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a multi-view stereoscopic three-dimensional reconstruction method based on cross-scale correction. Background Art

[0002] Multi-view stereo image reconstruction is an important research direction in the field of computer vision, which aims to reconstruct dense three-dimensional point clouds from a series of images taken from multiple perspectives. This technology has shown great application value in many fields. For example, in the field of intelligent driving, it helps the autonomous driving system to perceive surrounding objects in real time, significantly improving driving safety; in medical applications, it helps doctors better locate diseases and greatly improves the accuracy of diagnosis and treatment; in the field of virtual reality and augmented reality, it significantly improves the user's immersion and interactive experience. In summary, multi-view stereo image reconstruction technology has promoted the intelligent and digital process in the fields of transportation, medical care, virtual reality, etc. by reconstructing the three-dimensional structure of the real world with high precision, and has important social value and practical significance.

[0003] Traditional multi-view stereo 3D reconstruction methods include object space voxel method, seed point growth method and image space multi-view depth map reconstruction method. The object space voxel method is computationally intensive and memory intensive, while the seed point growth method is sensitive to the initial point and is easily affected by noise. The depth map-based method transforms point cloud reconstruction into a depth estimation problem, which can not only directly generate dense point clouds and avoid complex topological modeling, but also reconstruct surface details more completely. However, when dealing with complex scenes, large changes in perspective or severe noise interference, traditional multi-view stereo 3D reconstruction methods are prone to artifacts and mismatching problems. The 3D reconstruction algorithm based on deep learning can better handle these problems through its powerful feature learning ability. MVSNet first proposed an end-to-end multi-view stereo reconstruction network. The key to this method is to first extract deep visual image features, then construct a 3D cost volume on the reference image through a differentiable homography, apply 3D convolution to regularize the cost volume and then regress the depth. Since MVSNet constructs a 3D cost volume and regularizes it, its memory and computational costs grow cubically with the increase of volume resolution, so it is significantly limited in generating high-resolution results. To solve this problem, subsequent research proposed a method based on a cascade framework.

[0004] The core idea of ​​the cascade framework is to construct a cost volume based on a feature pyramid and optimize the depth estimate in stages. In the multi-stage processing process, the framework gradually narrows the depth search range of the current stage using the depth prediction of the previous stage. At the same time, by gradually improving the resolution of the cost volume and adaptively adjusting the depth interval, a high-resolution depth map is finally accurately reconstructed in a coarse-to-fine manner. There are two problems with the current cascade depth prediction model. One is that the network usually uses a feature pyramid network (FPN) for feature extraction. It is often difficult to simultaneously ensure the semantic integrity of high-level features and the detail fidelity of low-level features in FPN, which leads to an imbalance in the expression ability of multi-scale features, as well as the lack of spatial relationships and blurred details in the process of cross-scale feature fusion. Second, the coarse stage has a large depth range and many depth planes, making it difficult to obtain the correct depth, resulting in subsequent error amplification and reduced robustness. Therefore, how to efficiently solve the above problems has become a core challenge for 3D reconstruction of multi-view stereo images. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a multi-view stereo 3D reconstruction method based on cross-scale correction. It effectively solves the problem of missing spatial relationships and blurred details caused by cross-scale feature fusion in FPN, and balances the transmission of high-level semantics and low-level detail information. The method also alleviates the difficulty of classification or regression depth caused by the large depth range and multiple depth planes in low-texture areas and rough stages, and significantly improves the robustness and accuracy of the algorithm.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-view stereo 3D reconstruction method based on cross-scale correction, the method comprising the following steps:

[0008] S1. Obtain several input images divided into reference images and source images, and extract feature maps of different scales;

[0009] S2, fusing the multi-scale feature maps through a cross-scale feature fusion module, which at least includes feature upsampling, feature fusion, feature alignment and attention fusion operations;

[0010] S3, according to the camera parameters of the reference image and the source image, the source image features are mapped to the reference image coordinate system through homography transformation and plane scanning method to construct a three-dimensional cost volume;

[0011] S4, using the global information supplement module three-dimensional cost volume to aggregate, which at least includes cost volume expansion, global feature extraction, feature filtering and cost volume fusion processing;

[0012] S5, performing regularization processing on the optimized cost volume to generate a depth probability distribution volume; and calculating a first depth map according to the depth probability distribution volume;

[0013] S6. Take a depth plane around the first depth map, perform steps S3 and S5, and obtain a second depth map; take a depth plane around the second depth map, perform steps S3 and S5 again, obtain a final depth map, and finally generate a point cloud through the depth map.

[0014] Further, in step S1, the input includes N RGB images, one of which is a reference image and the remaining N-1 are source images; the N RGB images are processed by a feature extraction network with shared parameters to generate feature maps F1∈R at three scales C ×H×W , and The feature extraction network uses a convolutional neural network as a feature extractor. First, the original scale feature F1 is extracted from the original RGB image of size 3×H×W; then the original scale feature is downsampled by a convolutional layer with a specified step size to obtain the feature map F 1 / 2 ; Then the low-scale feature map F 1 / 2 Apply a convolutional layer with a preset step size again for downsampling to obtain the feature map F 1 / 4 .

[0015] Further, in step S2, at least the following steps are included:

[0016] S21. Low-scale features Perform bilinear interpolation upsampling to obtain Adjust the low resolution of low-scale features to be consistent with high-scale features The same high resolution; then convolution operations are performed on the low-resolution and high-resolution features respectively to adjust the number of feature channels and feature representation;

[0017] S22, concatenate the processed low-resolution features and high-resolution features in the channel dimension, and then generate a set of offsets and attention weights required for feature fusion through convolution;

[0018] S23, generating an offset grid coordinate for the low-resolution feature according to the generated offset, and performing spatial resampling on the low-resolution feature through the grid;

[0019] S24, according to the attention weights generated by the feature fusion unit and using 1+tanh activation, the aligned low-resolution features and high-resolution features are weighted fused, and finally the fused feature map is output. 1 / 4 , F 1 / 2 ) and (F 1 / 2 , F1) uses a cross-scale fusion module.

[0020] Further, in step S22, the offset is used to align the low-resolution features, and the attention weight is used to control the fusion ratio of the low-resolution and high-resolution features, which is calculated as follows:

[0021]

[0022] where f k×k,x→y It represents the convolution from x to y with a kernel size of k×k and the number of channels. concat(·) represents concatenation according to the dimension of the channel.

[0023] Further, in step S23, the operation process of performing feature alignment according to the offset is:

[0024] Assume that the spatial coordinates of each position on the feature map are {(1, 1), (1, 2), ..., (H, W)} and the learned 2D offset map is Δ∈R 2×H×W , then the resampled feature output is:

[0025]

[0026] Where U h，w represents the eigenvalue calculated by bilinear interpolation, F h′w′ represents the eigenvalue at the position (h′, w′), (h, w) is the offset coordinate, and (h′, w′) is the integer grid position closest to the target point (h, w); are the learned offsets in the vertical and horizontal directions, respectively.

[0027] Further, in step S24, the process of obtaining fused features according to the attention weights and the subsequent features is as follows:

[0028]

[0029] In the formula, O is the fusion feature, is the pixel-by-pixel multiplication, F l 'for The processed low-scale features, F′ h for The processed high-scale features, α and β are the weights of the activated low-scale features and high-scale features, respectively, where the low-scale feature attention weight is w l , the high-scale feature attention weight is w h , and activate them respectively through the 1+tanh activation function to obtain the corresponding activation weights.

[0030] Furthermore, in step S3, the construction of the cost body includes the following steps:

[0031] S31, obtain the internal and external parameters of the input image: the internal parameter K includes the focal length fx , f y 、Optical center c x , c y , the external parameters include the rotation matrix R and the translation vector t;

[0032] S32. Homography transformation: For each assumed depth plane d, the homography transformation from the source image plane to the reference image camera coordinate system is:

[0033]

[0034] Where H i (d) refers to the homography between the feature map of the i-th view and the reference feature map at depth d. In addition, K i , R i ,t i They are the camera intrinsic parameters, rotation and translation matrices of the i-th view. When i=1, it refers to the reference image. n represents the principal axis of the reference camera.

[0035] S33, plane scanning method for feature mapping: define a depth range and select a series of discrete depth values ​​within the range. For each depth value, use the corresponding homography transformation to map the feature map of the source image to a parallel plane of the reference image to obtain a series of mapping feature bodies on different depth planes.

[0036] S34, feature matching and cost calculation: For each pixel in the reference image, the matching cost between it and the feature mapped from the source image is calculated by variance. feature bodies, namely:

[0037]

[0038] Among them, V1 represents the feature volume of the reference image, and the rest are feature volumes of any source images.

[0039] Further, in step S4, it at least includes the following steps:

[0040] S41, the five-dimensional cost body Flatten the channel and depth dimensions into two-dimensional features CV 2d Through multi-layer convolution and filtering operations, CV with global semantic information is obtained 2d ;

[0041] S42, CV 2d Embedded in CV 3d Chinese: CV 2d Expand in the depth dimension to make it similar to CV 3d The depth of the same number, then generate weighted attention (As ):

[0042] ConvNext(F) = df 7×7，n→2n (f 1×1，2n→2n (f 1×1，2n→n (F)))+F

[0043] GSE(F)=f 1×1,m→n (ConvNext(F) ×3 )

[0044] CV 2d-expand = expand(GSE(CV 2d ))

[0045] A s =σ(CV 3d +CV 2d-expand )

[0046] Where, df k×k,x→y (·) represents the depthwise separable convolution with kernel size of k×k and number of channels from x to y, expand(·) represents expansion in the depth direction, and σ(·) represents the sigmoid activation function;

[0047] S43. CV by attention weight 2d-expand Perform filtering operation on CV 2d-expand To filter:

[0048]

[0049] S44, CV 3d and CV 2d-expand The splicing is performed along the channel direction, and the channel mapping is performed through a convolution to finally generate the optimized cost volume.

[0050] Further, in step S5, the optimized cost volume is subjected to 3DU-Net regularization processing, and it is encoded by using multiple layers of 3D convolutional layers, and then decoded by 3D deconvolution to filter the noise pollution of the cost volume and reduce the error caused by object occlusion; the probability P(d) of M depth planes d is calculated by Softmax, and the predicted depth D1 of each pixel is calculated as:

[0051]

[0052] where d m is the probability value of the mth depth plane, and D1 is the first depth map.

[0053] Further, based on the depth map D1, the range of the hypothesized plane is dynamically adjusted, and based on the range, features with higher resolution are input to reconstruct a new cost volume, and steps S4 and S5 are repeated to obtain a second depth map D2; based on D2 again, the above operations are repeated to obtain a final depth map;

[0054] The probability body generated by the cost body of the above process is supervised, and its total loss function is:

[0055]

[0056] Where P (d) (p) represents the probability value of depth d, represents the value closest to the depth hypothesis plane, {p v} represents the subset of pixels with valid ground facts;

[0057] After obtaining the final depth map, the depth value of the depth map and the camera intrinsic parameters are used to convert the coordinates of each pixel in the image plane into a three-dimensional point in the camera coordinate system. Combined with multi-view information, the position of the three-dimensional points is verified and optimized, redundant or erroneous points are eliminated, and all valid three-dimensional points are integrated to form a dense point cloud structure to complete the three-dimensional reconstruction.

[0058] The beneficial effects of the present invention are:

[0059] The present invention effectively solves the problems of missing spatial relationships and blurred details in the cross-scale feature fusion process in the current FPN by optimizing cross-scale feature fusion. The spatial continuity and detail expression ability of the feature map are enhanced, while balancing the transmission effect of high-level semantic information and low-level detail information. In particular, in the process of high-resolution feature generation, the blurring effect is effectively suppressed, ensuring the ability to restore details, thereby providing higher quality input features for subsequent tasks.

[0060] The present invention effectively solves the problem of difficulty in obtaining the correct depth due to the large depth range and multiple depth planes in the rough stage by introducing a global semantic information supplement module. The module effectively alleviates the classification and regression difficulties caused by low-texture areas and large-range depths, thereby significantly improving the overall depth estimation performance and enhancing the robustness and accuracy of the algorithm.

[0061] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0063] Figure 1 It is a schematic diagram of the overall process of the multi-view stereo 3D reconstruction method based on cross-scale correction of the present invention;

[0064] Figure 2 It is a structural schematic diagram of the cross-scale feature fusion module OFAM of the present invention;

[0065] Figure 3 It is a structural schematic diagram of the global information supplement module GSIM of the present invention.

[0066] Figure 4 It is a schematic diagram of qualitative comparison between the embodiment of the present invention and other existing methods in the DTU verification set scan10 and scan48;

[0067] Figure 5 This is the reconstruction result of the embodiment of the present invention on the Tanks and Temples dataset. DETAILED DESCRIPTION

[0068] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0069] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0070] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0071] See also Figure 1 to Figure 5 , which is a multi-view stereo 3D reconstruction method based on cross-scale correction.

[0072] Example

[0073] This embodiment provides a detailed implementation process of a multi-view stereo 3D reconstruction method based on cross-scale correction. Figure 1 The overall process diagram of the multi-view stereoscopic 3D reconstruction method based on cross-scale correction of the present invention is as follows:

[0074] S1. Obtain several input images divided into reference images and source images, and extract feature maps of different scales;

[0075] S2, fusing the multi-scale feature maps through a cross-scale feature fusion module, which at least includes feature upsampling, feature fusion, feature alignment and attention fusion operations;

[0076] S3, according to the camera parameters of the reference image and the source image, the source image features are mapped to the reference image coordinate system through homography transformation and plane scanning method to construct a three-dimensional cost volume;

[0077] S4, using the global information supplement module three-dimensional cost volume to aggregate, which at least includes cost volume expansion, global feature extraction, feature filtering and cost volume fusion processing;

[0078] S5. Regularize the optimized cost volume to generate a depth probability distribution volume; and calculate a first depth map according to the depth probability distribution volume.

[0079] S6, taking a depth plane around the first depth map, performing steps S3 and S5, and obtaining a second depth map. Taking a depth plane around the second depth map, performing steps S3 and S5, and obtaining a final depth map, and finally generating a point cloud through the depth map.

[0080] In step S1 of this embodiment, the input includes N RGB images, one of which is a reference image and the remaining N-1 are source images. The N RGB images are processed by a feature extraction network with shared parameters to generate feature maps of three scales, which are the original image size (1), 1 / 2, and 1 / 4. The feature extraction network uses a convolutional neural network as a feature extractor. First, the original scale feature F1∈R is extracted from the original RGB image of size H×W×C. C×H×W ; Then apply a convolutional layer with a step size of 2 to downsample the original scale feature and 2 convolutional layers to obtain The 1 / 2 scale feature map is then downsampled by a convolutional layer with a stride of 2 and two convolutional layers to obtain

[0081] In step S2 of this embodiment, an Offset-Aware Fusion Module (OAFM) ​​is used to fuse feature map information of different scales. The structure of OAFM is as follows: Figure 2 As shown, it at least includes a feature alignment unit, a feature fusion unit and an attention fusion unit, and specifically includes the following steps:

[0082] S21. Low-scale features Perform bilinear interpolation upsampling to obtain Adjust the low resolution of low-scale features to be consistent with high-scale features The same high resolution; then convolution operations are performed on the low-resolution and high-resolution features respectively to adjust the feature F h The number of channels C h arrive Number of channels C l , making it suitable for subsequent fusion operations.

[0083] S22, concatenate the processed low-resolution features and high-resolution features in the channel dimension and input them into the feature fusion unit. The feature fusion unit generates a set of offsets and attention weights required for feature fusion through multi-layer convolution. The offsets are used to align the low-resolution features, and the attention weights are used to control the fusion ratio of the low-resolution and high-resolution features. The calculation method is:

[0084]

[0085] S23. In the feature alignment unit, an offset grid coordinate is generated for the low-resolution feature according to the offset generated by the feature fusion unit. The low-resolution feature is spatially resampled (aligned) through the grid to ensure its spatial position consistency with the high-resolution feature, thereby eliminating the spatial difference between multi-scale features. Specifically, assuming that the spatial coordinates of each position on the feature map are {(1, 1), (1, 2), ..., (H, W)}, the learned 2D offset map is Δ∈R 2 ×H×W . Then the resampled feature output is:

[0086]

[0087] Where U h,w represents the eigenvalue calculated by bilinear interpolation, F h′w′ represents the eigenvalue at the position (h′, w′), (h, w) is the offset coordinate, and (h′, w′) is the integer grid position closest to the target point (h, w). They are the offsets learned in the vertical direction (row direction) and the horizontal direction (column direction), respectively.

[0088] S24. In the attention fusion unit, the aligned low-resolution features and high-resolution features are weighted fused according to the attention weights generated by the feature fusion unit:

[0089]

[0090] In the formula, O is the fusion feature, is the pixel-by-pixel multiplication, F l 'for The processed low-scale features, F′ h for The processed high-scale features, α and β are the weights of the activated low-scale features and high-scale features, respectively, where the low-scale feature attention weight is w l , the high-scale feature attention weight is w h , and activate them respectively through the 1+tanh activation function to obtain the corresponding activation weights.

[0091] The attention weight is used to dynamically balance the importance of the two, which can not only retain the global information in the low-resolution features, but also highlight the detailed information of the high-resolution features, and finally output the fused feature map.

[0092] In step S3 of this embodiment, the construction of the cost body includes the following steps:

[0093] S31, obtain the internal and external parameters of the input image; the internal parameter K includes the focal length f x , fy 、Optical center c x , c y , the external parameters include the rotation matrix R and the translation vector t. These parameters are used to determine the relative position and attitude between the two cameras.

[0094] S32, calculate homography transformation: for each assumed depth plane d, calculate the homography transformation from the source image plane to the depth plane in the reference image camera coordinate system. This transformation maps the points on the source image to a plane in the reference image camera coordinate system.

[0095]

[0096] Where H i (d) refers to the homography between the feature map of the i-th view and the reference feature map at depth d. In addition, K i , R i ,t i They are the camera intrinsic parameters, rotation and translation matrices of the i-th view. When i=1, it refers to the reference image. n represents the principal axis of the reference camera.

[0097] S33, plane scanning method for feature mapping: define a depth range and select a series of discrete depth values ​​within the range. For each depth value, use the corresponding homography transformation to map the feature map of the source image to a parallel plane of the reference image to obtain a series of mapping feature bodies on different depth planes.

[0098] S34, feature matching and cost calculation: For each pixel in the reference image, the matching cost between it and the feature mapped from the source image is calculated by variance. V1,…V N ∈R C×D×H×W feature bodies, where V1 represents the feature body of the reference image, and the rest are feature bodies of any source image.

[0099]

[0100] In step S4 of this embodiment, a global information supplementation module (GSIM) is used to fuse and optimize the input cost volume and context features to improve the quality of geometric features and the accuracy of depth estimation. The structural diagram of the global information supplementation module is shown in FIG. Figure 3As shown, it includes: cost body expansion: expanding the generated cost body along the channel (C) and depth (D) dimensions, global semantic feature extraction: extracting global semantic features through the ConvNeXt network, semantic feature filtering: expanding in the depth (D) dimension, fusing with the cost body, filtering the global semantic features after Sigmoid activation to enhance the expression of effective information. Cost body fusion: further fusing the cost body after global feature filtering to ensure the integrity and geometric consistency of the fused cost body. Specifically, it includes at least the following steps:

[0101] S41. Input cost body is a five-dimensional tensor containing information about the depth dimension. First, adjust it to flatten its depth dimension into a two-dimensional feature This facilitates subsequent convolution processing. Then, CV 2d Through multi-layer convolution and filtering operations, CV with global semantic information is obtained 2d .

[0102] S42, CV 2d With CV 3d Fusion. 2d Expand in the depth (D) dimension, which is consistent with the depth (D) of the cost body. Next, generate the weighted attention (A s ), through the attention mechanism for CV 2d Filtering. The attention weights determine which features should be strengthened or weakened, thereby achieving more effective feature fusion.

[0103] ConvNext(F) = df 7×7，n→n (f 1×1，n→2n (f 1×1，2n→n (F)))+F

[0104] GSE(F)=f 1×1，n→2n (ConvNext(F) ×3 )

[0105] cv 2d-expand = expand(GSE(cv 2d ))

[0106] A s =σ(cv 3d +cv 2d-expand )

[0107] Where, df k×k,x→y (·) represents the depthwise separable convolution with kernel size of k×k and number of channels from x to y, expand(·) represents expansion in the depth direction, and σ(·) represents the sigmoid activation function;

[0108] S43, CV 2d Applying attention weights makes CV 2d Different parts of are dynamically adjusted according to the context information. This step ensures that the details of the geometric features are consistent with the context information and avoids loss or error of depth information. s About CV 2d-expand Make dynamic adjustments.

[0109]

[0110] S44. The fused features are further enhanced in terms of spatial and geometric continuity through an additional aggregation operation (3D convolution processing), and finally generate optimized cost volume features. These features can provide higher quality input for subsequent tasks (such as depth estimation or 3D reconstruction). 3DU-Net is used to optimize the cost volume, mainly filtering out matching error points caused by occlusion, illumination, weak texture, etc. At the same time, the correct matching information is further aggregated to improve the quality and robustness of the cost volume.

[0111] In step S5 of this embodiment, the optimized cost volume is subjected to 3DU-Net regularization processing, encoded by using multiple layers of 3D convolutional layers, and then decoded by 3D deconvolution to filter out noise pollution in the cost volume and reduce errors caused by object occlusion. The probability P(d) of M depth planes d is calculated by Softmax, and the predicted depth D1 of each pixel is calculated as:

[0112]

[0113] where d m is the probability value of the mth depth plane;

[0114] Further, in step S6: Based on the obtained depth map, input higher resolution features. According to the depth estimation result of the previous stage, dynamically adjust the range of the hypothesized plane (i.e., narrow the depth search range), reconstruct a new cost volume based on the range, and repeat steps S4 and S5. Obtain the depth map, repeat twice.

[0115] For the probability body generated by the 3-layer cost body, the total loss function is supervised:

[0116]

[0117] Where P (d)(p) represents the probability value of depth d based on the depth map D1. Dynamically adjust the range of the hypothesized plane (i.e., narrow the depth search range), input higher resolution features based on this range, and reconstruct a new cost volume. Repeat steps S4 and S5 to obtain the depth map D2. Based on D2, repeat the above operations to obtain the final depth map.

[0118] For the probability body generated by the 3-layer cost body, the total loss function is supervised:

[0119]

[0120] Where P (d) (p) represents the probability value of depth d, represents the value closest to the depth hypothesis plane, {p v} represents the subset of pixels with valid ground facts.

[0121] After obtaining the final depth map, the depth values ​​of the depth map and the camera intrinsic parameters are used to convert each pixel coordinate in the image plane into a 3D point in the camera coordinate system. Combined with multi-view information, the position of the 3D points is verified and optimized to eliminate redundant or erroneous points. All valid 3D points are integrated to form a dense point cloud structure to complete the 3D reconstruction.

[0122] By optimizing the upsampling method, the problem of missing spatial relationships and blurred details in the cross-scale feature fusion process in the current FPN is effectively solved. The spatial continuity and detail expression ability of the feature map are enhanced, while the transmission effect of high-level semantic information and low-level detail information is balanced. In particular, in the process of high-resolution feature generation, the blurring effect is effectively suppressed and the detail recovery ability is ensured, thereby providing higher quality input features for subsequent tasks.

[0123] The present invention effectively solves the problem of difficulty in obtaining the correct depth due to the large depth range and multiple depth planes in the rough stage by introducing a global semantic information supplement module. The module effectively alleviates the classification and regression difficulties caused by low-texture areas and large-range depths, thereby significantly improving the overall depth estimation performance and enhancing the robustness and accuracy of the algorithm.

[0124] This example shows the performance of the multi-view stereo 3D reconstruction method based on cross-scale correction of the present invention and other existing methods on the DTU dataset, and the lower the result, the better. Figure 4 As shown in Table 1:

[0125] Table 1

[0126] Method Acc.(mm) Comp.(mm) Overall.(mm) Gipuma 0.283 0.873 0.578 COLMAP 0.400 0.664 0.532 Patchmatchnet 0.427 0.277 0.352 Effi-MVSNet 0.321 0.313 0.317 CasMVSNet 0.325 0.385 0.355 MVSNet 0.396 0.527 0.462 Our 0.360 0.264 0.312

[0127] As shown in Table 1, the quantitative results of the DTU dataset, the lower the value, the better the result. Compared with some other deep learning methods (such as standard MVSNet) and traditional multi-view stereo methods, the present invention outperforms all the above methods in completeness and overall score. Figure 5 The reconstruction results of the method of the present invention on the Tanks and Temples dataset are shown, which effectively solves the problems of missing spatial relationships and blurred details in the cross-scale feature fusion process in the current FPN. The spatial continuity and detail expression ability of the feature map are enhanced, while balancing the transmission effect of high-level semantic information and low-level detail information. In particular, in the process of high-resolution feature generation, the blurring effect is effectively suppressed to ensure the ability to restore details.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-view stereo 3D reconstruction method based on cross-scale correction, characterized by: The method comprises the following steps: S1. Obtain several input images divided into reference images and source images, and extract feature maps of different scales; S2, fusing the multi-scale feature maps through a cross-scale feature fusion module, which at least includes feature upsampling, feature fusion, feature alignment and attention fusion operations; S3, according to the camera parameters of the reference image and the source image, the source image features are mapped to the reference image coordinate system through homography transformation and plane scanning method to construct a three-dimensional cost volume; S4, using the global information supplement module three-dimensional cost volume to aggregate, which at least includes cost volume expansion, global feature extraction, feature filtering and cost volume fusion processing; S5, performing regularization processing on the optimized cost volume to generate a depth probability distribution volume; and calculating a first depth map according to the depth probability distribution volume; S6. Take a depth plane around the first depth map, perform steps S3 and S5, and obtain a second depth map; take a depth plane around the second depth map, perform steps S3 and S5 again, obtain a final depth map, and finally generate a point cloud through the depth map.

2. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 1, characterized in that: In step S1, the input contains N RGB images, one of which is a reference image and the remaining N-1 are source images; the N RGB images are processed by a feature extraction network with shared parameters to generate feature maps F1∈R at three scales. C×H×W , and The feature extraction network uses a convolutional neural network as a feature extractor. First, the original scale feature F1 is extracted from the original RGB image of size 3×H×W; then the original scale feature is downsampled by a convolutional layer with a specified step size to obtain the feature map F 1 / 2 ; Then the low-scale feature map F 1 / 2 Apply a convolutional layer with a preset step size again for downsampling to obtain the feature map F 1 / 4 .

3. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 2, characterized in that: In step S2, at least the following steps are included: S21. Low-scale features Perform bilinear interpolation upsampling to obtain Adjust the low resolution of low-scale features to be consistent with high-scale features The same high resolution; then convolution operations are performed on the low-resolution and high-resolution features respectively to adjust the number of feature channels and feature representation; S22, concatenate the processed low-resolution features and high-resolution features in the channel dimension, and then generate a set of offsets and attention weights required for feature fusion through convolution; S23, generating an offset grid coordinate for the low-resolution feature according to the generated offset, and performing spatial resampling on the low-resolution feature through the grid; S24, according to the attention weight generated by the feature fusion unit and using 1+tanh activation, the aligned low-resolution features and high-resolution features are weighted fused, and finally the fused feature map is output, respectively in (F 1 / 4 ,F 1 / 2 ) and (F 1 / 2 ,F1) Use cross-scale fusion module.

4. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 3, characterized in that: In step S22, the offset is used to align the low-resolution features, and the attention weight is used to control the fusion ratio of the low-resolution and high-resolution features, which is calculated as follows: where f k×k,x→y It represents the convolution from x to y with a kernel size of k×k and the number of channels. concat(·) represents concatenation according to the dimension of the channel.

5. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 3, characterized in that: In step S23, the operation process of performing feature alignment according to the offset is as follows: Assume that the spatial coordinates of each position on the feature map are {(1,1),(1,2),...,(H,W)} and the learned 2D offset map is Δ∈R 2×H×W , then the resampled feature output is: Where U h,w represents the eigenvalue calculated by bilinear interpolation, F h′w′ represents the eigenvalue at the position (h′, w′), (h, w) is the offset coordinate, and (h′, w′) is the integer grid position closest to the target point (h, w); are the learned offsets in the vertical and horizontal directions, respectively.

6. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 3, characterized in that: In step S24, the process of obtaining fused features based on the attention weights and subsequent features is as follows: In the formula, O is the fusion feature, is the pixel-by-pixel multiplication, F l 'for The processed low-scale features, F h 'for The processed high-scale features, α and β are the weights of the activated low-scale features and high-scale features, respectively, where the low-scale feature attention weight is w l , the high-scale feature attention weight is w h , and activate them respectively through the 1+tanh activation function to obtain the corresponding activation weights.

7. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 3, characterized in that: In step S3, the construction of the cost body includes the following steps: S31, obtain the internal and external parameters of the input image: the internal parameter K includes the focal length f x ,f y 、Optical center c x ,c y , the external parameters include the rotation matrix R and the translation vector t; S32. Homography transformation: For each assumed depth plane d, the homography transformation from the source image plane to the reference image camera coordinate system is: Where H i (d) refers to the homography between the feature map of the i-th view and the reference feature map at depth d; K i , R i ,t i They refer to the camera intrinsic parameters, rotation and translation matrices of the i-th view respectively. When i=1, it refers to the reference image, and n represents the principal axis of the reference camera; S33, plane scanning method for feature mapping: define a depth range, and select a series of discrete depth values ​​within the range. For each depth value, use the corresponding homography transformation to map the feature map of the source image to a parallel plane of the reference image, and obtain a series of mapping feature bodies on different depth planes; S34, feature matching and cost calculation: For each pixel in the reference image, the matching cost between it and the feature mapped from the source image is calculated by variance. feature bodies, namely: Among them, V1 represents the feature volume of the reference image, and the rest are feature volumes of any source images.

8. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 7, characterized in that: In step S4, it at least includes the following steps: S41, the five-dimensional cost body Flatten the channel and depth dimensions into two-dimensional features CV 2d Through multi-layer convolution and filtering operations, CV with global semantic information is obtained 2d ; S42, CV 2d Embedded in CV 3d Chinese: CV 2d Expand in the depth dimension to make it similar to CV 3d The depth of the same number, then generate weighted attention (A s ): ConvNext(F)=df 7×7,n→2n (f 1×1,2n→2n (f 1×1,2n→n (F)))+F GSE(F)=f 1×1,m→n (ConvNext(F) ×3 ) CV 2d-expand =expand(GSE(CV 2d )) AM s σ(CV 3d +CV 2d-expand ) Where, df k×k,x→y (·) represents the depthwise separable convolution with kernel size of k×k and number of channels from x to y, expand(·) represents expansion in the depth direction, and σ(·) represents the sigmoid activation function; S43. CV by attention weight 2d-expand Perform filtering operation on CV 2d-expand To filter: S44, CV 3d and CV 2d-expand The splicing is performed along the channel direction, and the channel mapping is performed through a convolution to finally generate the optimized cost volume.

9. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 8, characterized in that: In step S5, the optimized cost volume is subjected to 3DU-Net regularization processing, encoded by using multiple layers of 3D convolutional layers, and then decoded by 3D deconvolution to filter out the noise pollution of the cost volume and reduce the error caused by object occlusion; the probability P(d) of M depth planes d is calculated by Softmax, and the predicted depth D1 of each pixel is calculated as: where d m is the probability value of the mth depth plane, and D1 is the first depth map.

10. The multi-view stereo 3D reconstruction method based on cross-scale correction according to claim 9, characterized in that: Based on the depth map D1, dynamically adjust the range of the hypothesized plane, based on the range, input higher resolution features, reconstruct a new cost volume, repeat steps S4 and S5 to obtain the second depth map D2; based on D2 again, repeat the above operations to obtain the final depth map; The probability body generated by the cost body of the above process is supervised, and its total loss function is: Where P (d) (p) represents the probability value of depth d, represents the value closest to the depth hypothesis plane, {p v } represents the subset of pixels with valid ground facts; After obtaining the final depth map, the depth value of the depth map and the camera intrinsic parameters are used to convert the coordinates of each pixel in the image plane into a three-dimensional point in the camera coordinate system. Combined with multi-view information, the position of the three-dimensional points is verified and optimized, redundant or erroneous points are eliminated, and all valid three-dimensional points are integrated to form a dense point cloud structure to complete the three-dimensional reconstruction.

Images