A multi-view reconstruction method and system based on geometric perception and attention fusion

Abstract

The application discloses a multi-view reconstruction method and system based on geometric perception and attention fusion, and the reconstruction method designs a multi-scale feature enhancement network and a geometric perception feature fusion network; the multi-scale feature enhancement network extracts a multi-scale feature map and inputs the multi-scale feature map into a multi-stage depth estimation module to generate an initial depth map; the initial depth map is input into the geometric perception feature fusion network to generate a geometric perception enhanced feature map; the geometric perception enhanced feature map is input into the multi-stage depth estimation module to perform iterative optimization and output a refined depth map; and a joint loss function of a pixel-level cross-entropy loss function and a depth distribution similarity loss function is used to train a model constructed by the above network. The application effectively improves the expression capability of image features and the utilization efficiency of geometric information, focuses on solving the problem that the existing multi-view stereo reconstruction method has difficulty in depth estimation in a low-texture area and a structure repeated area, and significantly improves the accuracy and integrity of three-dimensional reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-view reconstruction method and system based on geometric perception and attention fusion. BACKGROUND

[0002] In recent years, with the wide development of technologies such as autonomous driving, augmented reality (AR), virtual reality (VR), robot navigation, and digital city, multi-view stereo reconstruction (MVS) as a core problem in three-dimensional reconstruction has attracted high attention from researchers in the field of computer vision. The core goal is to estimate the depth information of each pixel in the scene by analyzing the matching relationship and stereo correspondence relationship between multi-view images, so as to recover the complete three-dimensional geometric structure. Although existing technologies such as CasMVSNet have made significant progress in this field, the integrity and accuracy of the final point cloud still depend on the accuracy of the depth map, and there is still room for improvement.

[0003] As a typical deep learning method, the multi-view stereo reconstruction network (MVSNet) inputs reference images and source images, extracts multi-scale features using a feature pyramid network, constructs a three-dimensional cost volume using differentiable homography transformation, and finally regresses a high-precision depth map through a 3D convolutional neural network. Subsequent improvements such as CasMVSNet and UCSNet propose a cascaded cost volume based on a feature pyramid, and gradually estimate the depth map in a coarse-to-fine manner. These methods based on cost volume pyramid reduce the depth search range of the current stage by using the depth prediction results of the previous stage, optimize memory and computing time, and can provide better reconstruction results in high-resolution image processing. However, these methods still have certain difficulties when facing some complex challenges, especially when dealing with low-texture and structure-repetitive regions, they often ignore the details of these regions, leading to the loss of fine structures.

[0004] Especially in the depth map estimation process, the feature extraction and matching ability of existing methods cannot fully capture the geometric information of low-texture and structure-repetitive regions, resulting in insufficient accuracy in these regions. In addition, with the continuous improvement of image resolution, the increase of depth search space and the increase of computing complexity, existing technologies face efficiency bottlenecks while pursuing high accuracy. How to improve the feature extraction and geometric modeling ability while ensuring accuracy, especially in the aspect of detail recovery in low-texture regions or structure-repetitive regions, is still an important challenge in the field of multi-view stereo reconstruction.

[0005] In summary, how to effectively capture the key features of low-texture and structure-repeated areas in complex environments, and combine geometric information to improve the detail recovery capability of the depth map, has become a key problem for current multi-view stereo reconstruction methods to break through. The present application aims to enhance the depth recovery capability of these detail areas and improve the overall accuracy and integrity of three-dimensional reconstruction by introducing innovative geometric perception and attention mechanisms. SUMMARY

[0006] To solve the above technical problems, the present application provides a multi-view stereo reconstruction method and system based on geometric perception and attention fusion. The method effectively improves the expression ability of image features and the utilization efficiency of geometric information by designing a multi-scale feature enhancement network, introducing a geometric perception feature fusion network, and combining an improved loss function. It focuses on solving the problem of difficulty in depth estimation in low-texture areas and structure-repeated areas in existing multi-view stereo reconstruction methods, and significantly improves the accuracy and integrity of three-dimensional reconstruction.

[0007] The technical solution of the present application is as follows: a multi-view reconstruction method based on geometric perception and attention fusion, comprising the following steps:

[0008] Step S1: Prepare a multi-view stereo dataset, and divide the image pairs in the dataset into reference images and source images;

[0009] Step S2: input the reference images and source images in step S1 into a multi-scale feature enhancement network to extract multi-scale feature maps, and input the multi-scale feature maps into a multi-stage depth estimation module to generate a preliminary depth map;

[0010] Step S3: perform resolution processing on the reference images in step S1, and perform resolution and normalization processing on the preliminary depth map in step S2 to obtain multi-scale geometric features;

[0011] Step S4: input the multi-scale geometric features of the reference images and the initial depth map processed in step S3 into a geometric perception feature fusion network to generate a geometric perception enhanced feature map;

[0012] Step S5: input the geometric perception enhanced feature map into the multi-stage depth estimation module for iterative optimization, and output a refined depth map;

[0013] Step S6: construct a three-dimensional reconstruction model based on the multi-scale feature enhancement network, the geometric perception feature fusion network, and the multi-stage depth estimation module; and train the three-dimensional reconstruction model using a joint loss function of a pixel-level cross-entropy loss function and a depth distribution similarity loss function.

[0014] Further, the multi-view stereo dataset prepared in step S1 is specifically:

[0015] The multi-view stereo dataset is an indoor dataset, which is a public dataset for multi-view stereo reconstruction and 3D modeling, and contains 124 different objects or scenes, each scene is shot from 49 or 64 different perspectives, each perspective has 7 different lighting conditions, the image resolution in the dataset is 1600*1200 pixels, the dataset selects one image as a reference image and the rest as source images; the dataset provides camera intrinsic parameters, camera extrinsic parameters, high-precision laser point cloud and real depth map data for mapping pixel coordinates to the camera coordinate system.

[0016] Further, the reference image and the source image in step S1 are input to the multi-scale feature enhancement network to extract multi-scale feature maps, and the multi-scale feature maps are input to the multi-stage depth estimation module to generate a preliminary depth map; specifically:

[0017] In step S21, the multi-scale feature enhancement network is divided into a feature selection module and a feature fusion module, and the reference image and the source image in step S1 are input to the feature selection module of the multi-scale feature enhancement network;

[0018] In step S22, the reference image and the source image are operated by a preliminary convolution module composed of two consecutive convolution units to generate a first layer of initial feature maps with a spatial resolution of H*W;

[0019] In step S23, the feature selection module adopts a top-down down-sampling convolution module, and the first layer of initial feature maps are sequentially input to three down-sampling convolution modules, and the spatial resolution is gradually reduced to H / 2*W / 2, H / 4*W / 4 and H / 8*W / 8, H / 8*W / 8 is the highest layer of spatial resolution feature, and a second layer of initial feature maps, a third layer of initial feature maps and a fourth layer of initial feature maps are obtained;

[0020] In step S24, the first layer of initial feature maps, the second layer of initial feature maps, the third layer of initial feature maps and the fourth layer of initial feature maps are input to a lightweight multi-dimensional fusion attention module for feature enhancement in the channel-height, channel-width and spatial dimension directions to obtain a first layer of enhanced initial feature maps, a second layer of enhanced initial feature maps, a third layer of enhanced initial feature maps and a fourth layer of enhanced initial feature maps;

[0021] In step S25, the fourth layer of enhanced initial feature maps is input to a multi-scale context aggregation module of the feature fusion module to obtain a deep layer of initial feature maps, and the feature fusion module adopts a bottom-up up-sampling module to gradually up-sample the deep layer of initial feature maps, and the up-sampling module gradually fuses with the first layer of enhanced initial feature maps, the second layer of enhanced initial feature maps, the third layer of enhanced initial feature maps and the fourth layer of enhanced initial feature maps through a channel alignment module, and outputs four multi-scale feature maps with different resolutions, and the spatial resolution levels are ;

[0022] Step S26, input the four different resolution multi-scale feature maps into the multi-stage depth estimation module to generate a preliminary depth map.

[0023] Further, step S3: performing resolution processing on the reference image in step S1 and performing resolution and normalization processing on the preliminary depth map in step S2 to obtain multi-scale geometric features; specifically:

[0024] Step S31, performing resolution processing on the reference image and the preliminary depth map to obtain reference image features I and preliminary depth map features D, and performing normalization processing on the preliminary depth map features to obtain a normalized depth map, denoted as:

[0025] ;

[0026] where d is the normalized depth map, D is the preliminary depth map feature, 、 is the maximum and minimum value of the preliminary depth map feature;

[0027] Step S32, the reference image features I and the normalized depth map d generate multi-scale geometric features with different spatial resolutions according to std, uv, z or xyz four modes, where std, uv, z, xyz four modes represent feature expression from no geometric constraint to strong geometric constraint in turn, providing multi-level geometric information from two-dimensional coordinates to three-dimensional space for the geometric perception convolution module;

[0028] In std mode, no geometric feature is attached;

[0029] In uv mode, a two-dimensional coordinate grid covering each pixel of the image is generated based on the size of the input reference image features I, and the two-dimensional coordinate grid is linearly normalized to the [0, 1] interval; the normalized two-dimensional coordinate grid is down-sampled by 2x2 average pooling to form a 2-channel feature;

[0030] In z mode, the normalized depth map d is down-sampled by sparse pooling to form a 1-channel depth feature;

[0031] In xyz mode, according to the camera intrinsic parameters corresponding to the images in the data set, the normalized two-dimensional coordinates (u, v) of each pixel of the image are back-projected into three-dimensional coordinates (x, y, z) in the camera coordinate system with the 1-channel depth feature in z mode, and a 3-channel feature is output;

[0032] Step S34, the multi-scale geometric features of the four modes are generated in multi-scale representation according to the spatial resolution levels 1, 1 / 2, 1 / 4, 1 / 8 to form corresponding multi-scale geometric information, and the multi-scale geometric features are input into the geometric perception convolution module as multi-scale geometric feature information.

[0033] Further, step S4, the multi-scale geometric features of the reference image and the initial depth map processed in step S3 are input into a geometric perception feature fusion network to generate a geometric perception enhanced feature map; the geometric perception feature fusion network is divided into a first branch decoding module, a second branch decoding module and a final fusion module; specifically:

[0034] Step S41, the reference image and the initial depth map processed in step S3 are spliced and input into a preliminary convolution module of the first branch decoding module to generate a first fusion feature map, and the first fusion feature map is input into a five-level geometric perception convolution module, and the multi-scale geometric feature information of the reference image and the initial depth map is fused in each level of the geometric perception convolution module to obtain a second fusion feature map, a third fusion feature map, a fourth fusion feature map, a fifth fusion feature map and a sixth fusion feature map respectively;

[0035] Step S42, the fusion feature map is gradually up-sampled by a deconvolution module in the first branch decoding module, and the fusion feature map output by the preliminary convolution module and the geometric perception convolution module is connected in residual during the gradual up-sampling process of the deconvolution module; a second initial depth map with the same resolution as the input reference image is generated;

[0036] Step S421, after the first-level up-sampling of the deconvolution module, the fifth fusion feature map of the fourth-level geometric perception convolution module is connected in residual to obtain a spatial resolution of H / 8xW / 8;

[0037] Step S422, after the second-level up-sampling of the deconvolution module, the third fusion feature map of the second-level geometric perception convolution module is connected in residual to obtain a spatial resolution of H / 4xW / 4;

[0038] Step S423, after the third-level up-sampling of the deconvolution module, the second fusion feature map of the first-level geometric perception convolution module is connected in residual to obtain a spatial resolution of H / 4xW / 4;

[0039] Step S424, after the fourth-level up-sampling of the deconvolution module, the first fusion feature map of the preliminary convolution module is connected in residual to obtain a spatial resolution of H / 2xW / 2;

[0040] Step S425, after the fifth-level up-sampling of the deconvolution module, a spatial resolution of HxW is obtained; a second initial depth map is generated;

[0041] Step S43, the normalized depth map d channel and the second initial depth map channel are spliced, input to the preliminary convolution module of the second branch decoding module to generate the seventh fusion feature map, and the seventh fusion feature map is input to the five-level geometric perception convolution module of the second branch decoding module, and each level of fusion fuses the corresponding multi-scale geometric feature information of the reference image and the initial depth map at the same time;

[0042] Step S44, the seventh fusion feature map is input to the first geometric perception convolution module and the second geometric perception convolution module to obtain the eighth fusion feature map, the eighth fusion feature map is channel spliced with the feature output by the residual connection of step S424 to obtain the first joint feature; the first joint feature is input to the third geometric perception convolution module and the fourth geometric perception convolution module to obtain the ninth fusion feature map, and the ninth fusion feature map is channel spliced with the feature output by the residual connection of step S421 to obtain the second joint feature; the second joint feature is input to the fifth geometric perception convolution module to generate the third initial depth map; step S45, the final fusion module is a five-level deconvolution module and a five-time fusion process, the five-level deconvolution module includes a first-level deconvolution module, a second deconvolution module, a third deconvolution module, a fourth deconvolution module and a fifth deconvolution module, the third deconvolution module processes the first-stage geometric perception enhanced feature map output in the first stage, the fourth deconvolution module processes the second-stage geometric perception enhanced feature map output in the second stage, and the fifth deconvolution module processes the third-stage geometric perception enhanced feature map output in the third stage;

[0043] Step S46, the first fusion process, the sixth fusion feature map of the first branch decoding module and the feature of the third initial depth map of the second branch decoding module are element-wise added, input to the first-level deconvolution module of the final fusion module to generate the first-level fusion feature;

[0044] Step S47, the second fusion process, the fourth-level geometric perception module of the second branch decoding module is element-wise added with the first-level fusion feature, input to the second-level deconvolution module of the final fusion module to generate the second-level fusion feature;

[0045] Step S48, the third fusion process, the second-level geometric perception module of the second branch decoding module is element-wise added with the second-level fusion feature, input to the third-level deconvolution module of the final fusion module to generate the third-level fusion feature, if it belongs to the first stage, the third-level fusion feature is input to the first-level RGB deconvolution module, and after output, the residual connection is performed with the multi-scale feature enhancement network output The resolution multi-scale feature map to obtain the first-stage geometric perception enhanced feature map;

[0046] Step S49, the fourth fusion process, the first level geometric perception module of the second branch decoding module is added with the third level fusion feature element by element, input to the fourth level deconvolution module of the final fusion module, to generate the fourth level fusion feature, if it belongs to the second stage, the fourth level fusion feature is input to the second level RGB deconvolution module, after output and the residual connection of the 1 / 1 resolution multi-scale feature map output by the multi-scale feature enhancement network, the second stage geometric perception enhanced feature map is obtained;

[0047] Step S50, the fifth fusion process, the preliminary convolution module of the second branch decoding module is added with the fourth level fusion feature element by element, input to the fifth level deconvolution module of the final fusion module, to generate the fifth level fusion feature, if it belongs to the third stage, the fifth level fusion feature is input to the third level RGB deconvolution module, after output and the residual connection of the 1 / 1 resolution multi-scale feature map output by the multi-scale feature enhancement network, the third stage geometric perception enhanced feature map is obtained;

[0048] The geometric perception enhanced feature map is expressed by the formula:

[0049]

[0050] Wherein, Y is the geometric perception enhanced feature map, is the feature in the multi-scale feature enhancement network, is the fusion feature of the first branch decoding module and the second branch decoding module, is the fusion feature of the first branch decoding module and the second branch decoding module and the feature in the multi-scale feature enhancement network in the final fusion module is element by element fusion, and represent the first branch decoding module and the second branch decoding module respectively, I is the reference image feature, and d is the normalized depth map, is the reference image guided depth map.

[0051] Further, step S5: input the geometric perception enhanced feature map to the multi-stage depth estimation module for iterative optimization to output the refined depth map; specifically:

[0052] Step S51, input the geometric perception enhanced feature map, the geometric perception enhanced feature map contains the reference image feature and the source image feature, through the camera internal parameter and the camera external parameter provided in the data set, the source image feature is projected to the reference view angle according to the depth assumption of each pixel, to realize feature alignment;

[0053] Step S52, calculate the correlation of the reference image feature and the aligned source image feature according to group division;

[0054] ​​Step S53, the correlation of all source image features is weighted and averaged by weight to construct a preliminary cost volume, the preliminary cost volume is regularized by deep learning, and three-dimensional space consistency modeling is performed;

[0055] Step S54, the regularized preliminary cost volume is normalized by an exponential function to form a probability distribution of each pixel of the image under a depth hypothesis set;

[0056] Step S55, the final depth value of each pixel of the image is obtained according to the probability distribution, and the final depth values of all pixels of the image are combined to obtain a complete depth map;

[0057] Step S56, according to the final depth value of the current stage, a finer depth hypothesis set used in the next stage is generated by inverse depth resampling in the inverse depth domain of each pixel of the image;

[0058] Step S57, after subsequent multi-stage processing, a final depth map is finally output.

[0059] Further, a multi-view reconstruction system based on geometric perception and attention fusion constructs a three-dimensional reconstruction model based on a multi-scale feature enhancement network, a geometric perception feature fusion network and a multi-stage depth estimation module;

[0060] A three-dimensional reconstruction model is constructed based on a multi-scale feature enhancement network, a geometric perception feature fusion network and a multi-stage depth estimation module;

[0061] The multi-scale feature enhancement network includes a feature selection module and a feature fusion module;

[0062] The feature selection module includes a preliminary convolution module, a down-sampling convolution module and a lightweight multi-dimensional fusion attention module;

[0063] The feature fusion module is composed of a channel alignment module, an up-sampling module and a multi-scale context aggregation module;

[0064] The geometric perception feature fusion network mainly includes a first branch encoding module, a second branch encoding module and a final fusion module;

[0065] The first branch encoding module includes an initial convolution module and a geometric perception convolution module, and an inverse convolution module;

[0066] The second branch encoding module includes an initial convolution module and a geometric perception convolution module;

[0067] The final fusion module receives the output of the first branch encoding module and the output of the second branch encoding module, and scales the first branch encoding module and the second branch encoding module for fusion;

[0068] The multi-stage depth estimation module is composed of feature alignment, correlation calculation, regularization, normalization, depth regression and inverse depth resampling.

[0069] Compared with the prior art, the present application has the following beneficial effects:

[0070] The present application designs a multi-scale feature enhancement network, adopts an innovative structure combining bottom-up feature extraction and top-down feature fusion, and effectively integrates multi-scale features by fusing deep high-level semantic features and shallow high-resolution features through upsampling at each processing stage. A lightweight multi-dimensional fusion attention module is introduced at each layer of the bottom-up feature extraction to fully exploit the information interaction of space, channel and direction and enhance the attention ability of the network to target or boundary regions. In the top-down feature fusion stage, a multi-scale context aggregation module is applied to the highest layer features for feature enhancement to strengthen the consistency and stability of semantic features and thus improve the feature fusion effect.

[0071] The present application designs a geometric perception feature fusion network integrating geometric perception and attention mechanism, aiming to fuse reference images, depth maps and geometric guide information to generate enhanced feature maps with stronger spatial perception and structural expression ability. The geometric perception feature fusion network adopts a double-branch structure design to guide and multi-scale fuse semantic and geometric information, thereby synergistically improving the depth estimation accuracy. A geometric perception convolution module is introduced at each encoding stage to guide the structure and enhance the saliency of features at each scale, effectively improving the spatial recognition ability and structural consistency of the feature maps. The attention-enhanced feature maps output by the multi-scale feature enhancement network are stacked in the final result through residual connection, thereby balancing detail restoration and overall consistency.

[0072] The present application adopts a pixel-level cross-entropy loss function and a depth distribution similarity loss function. The pixel-level cross-entropy loss function is used to supervise the prediction of each pixel class by the model, enhancing the classification ability of detail areas and boundary targets and effectively improving the accuracy of overall semantic segmentation or depth estimation. The depth distribution similarity loss function is used to measure the similarity of the predicted depth map and the real depth map in overall distribution. This loss function can enhance the network's modeling ability for structural consistency and geometric continuity, thereby improving the stability of overall depth perception while maintaining local accuracy. The joint optimization of the two types of loss functions enables the model to achieve a good balance between spatial perception and semantic understanding, and finally output a prediction result with higher accuracy and structural integrity. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 It is a schematic diagram of the three-dimensional reconstruction model framework of the present application.

[0074] Figure 2A schematic diagram of the multi-scale feature enhancement network framework of the present application.

[0075] Figure 3 A schematic diagram of the geometric perception fusion network framework of the present application.

[0076] Figure 4 A schematic diagram of the multi-stage depth estimation module framework of the present application. DETAILED DESCRIPTION

[0077] As shown in the figure, a schematic diagram of the three-dimensional reconstruction model framework of the present application, a multi-view reconstruction method based on geometric perception and attention fusion, comprising the following steps: Figure 1

[0078] Step S1: Prepare a multi-view stereo dataset, and divide the image pairs in the dataset into reference images and source images;

[0079] Step S2: input the reference images and source images in step S1 into the multi-scale feature enhancement network to extract multi-scale feature maps, and input the multi-scale feature maps into the multi-stage depth estimation module to generate a preliminary depth map;

[0080] Step S3: perform resolution processing on the reference images in step S1, and perform resolution and normalization processing on the preliminary depth map in step S2 to obtain multi-scale geometric features;

[0081] Step S4: input the multi-scale geometric features of the reference images and the initial depth map processed in step S3 into the geometric perception feature fusion network to generate a geometric perception enhanced feature map;

[0082] Step S5: input the geometric perception enhanced feature map into the multi-stage depth estimation module for iterative optimization, and output a refined depth map;

[0083] Step S6: construct a three-dimensional reconstruction model based on the multi-scale feature enhancement network, the geometric perception feature fusion network, and the multi-stage depth estimation module; and train the three-dimensional reconstruction model using a joint loss function of a pixel-level cross-entropy loss function and a depth distribution similarity loss function.

[0084] ​Further, in step S1, the present application uses a multi-view stereo dataset, which is an indoor dataset composed of 124 different scenes, each recorded from 49 views, with 7 different lighting conditions and 7 levels of brightness; it contains ground-truth point clouds collected under well-controlled laboratory conditions and can be used to evaluate the accuracy and completeness of three-dimensional reconstruction methods. The multi-view stereo dataset is a public dataset for multi-view stereo reconstruction and 3D modeling published by the Computer Vision Laboratory of the Technical University of Denmark (DTU). The image resolution in the dataset is 1600x1200 pixels, and each image contains detailed depth information, while the internal and external parameters of the camera are provided to map pixel coordinates to the camera coordinate system and provide geometric constraints in multi-view reconstruction. It is suitable for research in the fields of computer vision, three-dimensional reconstruction, etc. In addition, each image is provided with a source image and a reference image to facilitate the application and testing of multi-view reconstruction algorithms; the image pairs in the dataset are divided into source images and reference images: the source image is used as the target image for depth estimation, and the reference image is used to provide cross-view geometric constraints and texture supplement information.

[0085] Further, in step S2: in the multi-scale feature enhancement network; the feature maps of the source image and the reference image at multiple scales are extracted through the multi-scale feature enhancement network. Unlike traditional multi-scale feature extraction structures, the multi-scale feature enhancement network introduces a multi-stage feature fusion mechanism, which can effectively alleviate the problem of feature weakening and unstable semantic expression in low-texture areas or structure-repeated areas due to downsampling.

[0086] In order to further improve the discriminability of feature representation, a lightweight multi-dimensional fusion attention module is introduced in each feature fusion stage. The lightweight multi-dimensional fusion attention module combines a light convolutional layer based on a triple attention mechanism to further enhance the features. Finally, the original input and the enhanced features are fused through residual connection to enhance the expression ability of the feature map, especially in different spatial and channel dimensions. A multi-scale context aggregation module is introduced in the high-level output of the network to further optimize the representation ability of high-level semantic information and enhance the network's understanding ability of global structure and semantics. The image features extracted in the first stage are input into the multi-stage depth estimation network to obtain the first-stage depth estimation result.

[0087] Further, step S4: in the geometry-aware feature fusion network, the initial depth result and the reference image are jointly feature-extracted by using the geometry-aware feature fusion network. The geometry-aware feature fusion network realizes deep coupling of features and geometric information through embedded geometry-aware convolution modules. The fused features will be up-sampled through deconvolution operation to restore spatial details, and further jointly constructed with the output of the multi-scale feature enhancement network to form a feature expression with geometric consistency for guiding the subsequent depth estimation and refinement process. After forming the feature expression with geometric consistency, these features will be used to guide the subsequent depth estimation and refinement process, thereby further improving the accuracy of depth prediction.

[0088] Further, step S5: in the multi-stage depth estimation module, the features output by the geometry-aware feature fusion network are input into the multi-stage depth estimation network for calculating the final depth map. The multi-stage depth estimation module gradually refines and optimizes the accuracy of the depth map through progressive feature updating and depth prediction. Each stage utilizes the depth estimation result of the previous stage to further adjust and improve the quality of the depth map, ensuring that the final output depth map reaches higher precision in spatial and geometric consistency.

[0089] The higher precision features are input into the multi-stage depth estimation module, which can gradually eliminate errors and inaccurate estimates in each step through a multi-stage iterative process, and finally obtain an accurate and detailed depth map. These steps ensure that global and local information in the depth estimation process is fully fused and optimized, thereby improving the overall quality of depth prediction. Specific embodiments

[0090] As shown in Figure 2 , in the designed multi-scale feature enhancement network, the first layer initial feature map, the second layer initial feature map, the third layer initial feature map and the fourth layer initial feature map are input into the lightweight multi-dimensional fusion attention module to perform feature enhancement in the channel-height, channel-width and spatial dimension directions to obtain the first layer enhanced initial feature map, the second layer enhanced initial feature map, the third layer enhanced initial feature map and the fourth layer enhanced initial feature map. The lightweight multi-dimensional fusion attention module is represented as:

[0091] ;

[0092] wherein, the feature map after the lightweight multi-dimensional fusion attention module, is the feature map input after the preliminary convolution module or the down-sampling convolution module, R represents the set of real numbers, B represents the batch quantity of input feature maps, C represents the channel number of input feature maps; and respectively represent the height and width of the input feature map; represents a light convolution, S represents a direction-aware spatial attention generation module, represents a channel pooling, , respectively represent a permutation operation between channels and height, width; , is the inverse operation of the permutation between channels and height, width; represents an element-wise multiplication;

[0093] The fourth layer enhanced initial feature map is input into the multi-scale context aggregation module to obtain a deep layer initial feature map. The feature fusion module adopts a bottom-up up-sampling module to gradually up-sample the deep layer initial feature map. The step-by-step up-sampling module is fused with the first layer enhanced initial feature map, the second layer enhanced initial feature map, the third layer enhanced initial feature map and the fourth layer enhanced initial feature map through the channel alignment module, and four multi-scale feature maps with different resolutions are output, and the spatial resolutions are ; The formula is expressed as:

[0094] ;

[0095] ;

[0096] wherein, is the i-th multi-scale feature map, is the i+1-th multi-scale feature map, is a multi-scale semantic feature, x represents the reference image and the source image, is the i-th generated initial feature map, is the i-th light multi-dimensional fusion attention module, is a multi-scale context aggregation module, is an up-sampling module, is a channel alignment module, is the i+1-th channel alignment module.

[0097] As shown in Figure 3 , in the design of the geometry-aware feature fusion network, the geometry-aware convolution module is built-in with a geometry convolution attention module Geo and a channel attention SE, which is used to enhance the response of the input feature to the key geometric information, so as to improve the feature fusion and depth perception ability. The geometry-aware convolution module is improved, and the specific process is as follows:

[0098] The input feature x (reference image and source image) and the first geometric information g1 are fused by a geometric convolution attention module Geo, and the fused feature is subjected to convolution Conv, normalization BN and activation function ReLU;

[0099] The input is fused with the corresponding second geometric information g2 in the geometric convolution attention module Geo again, and then subjected to convolution, normalization and ReLU activation, and then enters the squeeze-and-excitation attention to enhance the feature;

[0100] The squeeze-and-excitation attention module enhances the feature, and outputs the final feature, which is represented as:

[0101] ;

[0102] wherein, represents a feature map subjected to a series of operations, represents an activation function, represents channel attention, represents a normalization operation, represents a convolution operation, represents geometric convolution attention, g1 and g2 are the first geometric information and the second geometric information;

[0103] In step S32, two feature modes corresponding to the current resolution are selected from the multi-scale geometric information as the convolution input, which are referred to as the first geometric information g1 and the second geometric information g2; in the low-resolution level, the z mode and the uv mode features can be selected as the first geometric information g1 and the second geometric information g2, and in the high-resolution level, the z mode and the xyz mode features can be selected as the first geometric information g1 and the second geometric information g2.

[0104] The geometric convolution attention module inputs the input feature x and the geometric information feature corresponding to the resolution into the channel attention CA and the spatial attention SA for weighting, and inputs the input feature x and the geometric information g through concat splicing to output the feature, which is represented as:

[0105] ;

[0106] wherein, final represents the output feature, concat represents splicing, represents channel attention, and SA represents spatial attention.

[0107] As shown in Figure 4 , the geometric perception enhanced feature map is input into the multi-stage depth estimation module for iterative optimization, and a refined depth map is output.

[0108] The multi-stage depth estimation module: a set of depth hypotheses is initialized for the first stage , n is the number of depth hypotheses, the depth hypothesis set of the first stage is determined by the minimum depth and the maximum depth of the high-precision laser point cloud in the data set, and the sampling interval is generated by using equal interval or inverse proportional sampling in the sampling interval to generate the depth hypothesis set, and the depth hypothesis set of the subsequent stage is provided by the last stage depth, and specifically:

[0109] Step S51, input the geometric perception enhanced feature map, the geometric perception enhanced feature map contains reference image features and source image features, and the source image features are projected to the reference view angle according to the depth hypothesis of each pixel by the camera internal and camera external parameters provided in the data set, so as to realize feature alignment; as shown in the formula:

[0110]

[0111]

[0112] wherein p is the pixel coordinate (spatial position) in the reference image feature map, represents the pixel coordinate p in the reference image feature map corresponding to the i-th source image pixel coordinate under the j-th depth hypothesis is the i-th source image internal parameter, , is the rotation matrix and translation matrix (external parameter) between the reference image and the i-th source image; is the depth hypothesis,

[0113] represents the pixel coordinate p in the reference image feature map corresponding to the i-th source image pixel coordinate under the j-th depth hypothesis obtains the aligned feature from the source image feature map ;

[0114] The reference image feature map is , the source image feature map is , i=1,2,...,N−1, N is the total number of the reference image feature map plus all source image feature maps, and the reference image internal parameter is ;

[0115] Step S52, calculate the correlation of the reference image features and the aligned source image features according to groups;

[0116] The number of channels of the geometric perception enhanced feature map is divided into G groups, and the number of channels of each group is Cg=C / G. The correlation of each group of reference features and aligned source features is calculated, as shown in the formula:​​​​​

[0117] ;

[0118] where, is the g-th group of correlation values of pixel coordinate p in the reference image feature map under depth hypothesis ; is the g-th group of features of pixel coordinate p in the reference image feature map, is the inner product operation, ;

[0119] The correlation of all groups is averaged and summed to obtain the correlation of pixel coordinate p in the reference feature map with the i-th source image feature map under depth hypothesis (integrating all channels), as shown in the formula:

[0120] ;

[0121] The overall correlation is exponentially normalized, as shown in the formula:

[0122] ;

[0123] where, is the matching probability of pixel coordinate p in the reference image feature map with the i-th source image feature map under depth hypothesis , k refers to traversing the depth hypothesis set ;

[0124] The maximum probability of reference image feature image pixel coordinate p with the i-th source image under depth hypothesis is taken as the weight , as shown in the formula:

[0125] = max ;

[0126] The correlation of all source images is weighted and averaged, as shown in the formula:

[0127] ;

[0128] where, is the correlation of pixel coordinate p in the reference image feature map under depth hypothesis ; the preliminary cost volume ) is obtained by batch calculation of the reference image feature map pixel coordinate p;

[0129] ​​Step S53, the correlation of all source image features is weighted and averaged by weight to construct a preliminary cost volume, the preliminary cost volume is regularized by deep learning, and three-dimensional space consistency modeling is performed;

[0130] The preliminary cost volume is input into a three-dimensional convolutional neural network (RegNet) for regularization processing to obtain a smoother and more accurate cost volume , as shown in the formula:

[0131] ;

[0132] Among them, represents the regularized cost volume of all pixel coordinates p in the reference image feature map under the depth hypothesis ;

[0133] Step S54, the preliminary cost volume after regularization is normalized by an exponential function to form the probability distribution of each pixel of the image under the depth hypothesis set;

[0134] The regularized cost volume is normalized along the depth dimension by operation to obtain the probability volume of pixel coordinates p in the reference image feature map under different depth hypotheses , also called probability distribution; the probability volume is depth-regressed to obtain the final depth of each pixel, as shown in the formula:

[0135] ;

[0136] Among them, represents the final regression depth value of all pixel coordinates p in the reference image feature map, and all pixel depth values are combined to form a complete normalized depth map d;

[0137] Step S55, the final depth value of each pixel of the image is obtained according to the probability distribution regression, and the final depth value of all pixels of the image is combined to obtain a complete depth map;

[0138] The regularized cost volume is normalized along the depth dimension by operation to obtain the probability volume of pixel coordinates p in the reference image feature map under different depth hypotheses , also called probability distribution. Then the probability volume is depth-regressed to obtain the final depth of each pixel, as shown in the formula:

[0139] ;

[0140] Among them, represents the final regression depth value in all pixel coordinates p in the reference image feature map, and all pixel depth values are combined into a complete depth map d.

[0141] Step S56 generates a more refined depth hypothesis set used in the next stage by inverse depth resampling in the inverse depth domain of each pixel of the image according to the final depth value in the current stage;

[0142] Step S57 outputs a final depth map after subsequent multi-stage processing.

[0143] In order to enable the next stage to concentrate sampling to a more accurate depth range, it is necessary to combine all pixel coordinates p in the reference image feature map output by the current stage according to the final regression depth value Redefine the new depth sampling interval, so first calculate the inverse depth of each pixel predicted depth, then resample in the inverse depth domain with smaller intervals; finally, reverse back to the positive depth; as shown in the formula;

[0144] ;

[0145] ;

[0146] wherein, is the upper and lower bound of the inverse depth domain resampled by all pixel coordinates p in the reference image feature map, is a scaling coefficient, is the depth sampling interval of the current stage; uniformly sample c depth values in this new range to form a refined depth hypothesis set for the next stage .

[0147] The three-dimensional reconstruction model is trained by using a joint loss function of a pixel-level cross-entropy loss function and a depth distribution similarity loss function, and the model introduces the pixel-level cross-entropy loss function and the depth distribution similarity loss function:

[0148] Loss function: In the task of multi-view stereo reconstruction, the design of the loss function plays a crucial role in the performance of the depth estimation network. Traditional multi-view stereo networks do not directly predict depth values, but predict a depth probability volume, i.e., predict the probability distribution of each pixel on a series of candidate depth values, and finally regress the depth by weighted average. In order to effectively guide the network to learn the spatial geometric structure, two types of complementary loss functions are introduced in the training stage: pixel-level cross-entropy loss and depth distribution similarity loss. The former focuses on improving the discriminative ability of the predicted depth of each pixel, and the latter focuses on the consistency of the statistical structure of the predicted depth distribution and the true depth.

[0149] The pixel cross-entropy loss is used to train the network to predict the depth probability distribution of each pixel, so that the probability in the depth dimension is as concentrated as possible at the true depth value. This strategy can better guide the network to learn the depth distribution characteristics and improve the matching accuracy and stability. Specifically, the true depth is matched with the predicted depth value to obtain the minimum difference index of each pixel, and then the cross-entropy formula is used to calculate the loss of each pixel. The depth distribution similarity loss, the goal of this loss function is to make the distribution of the predicted depth map as close as possible to the distribution of the true depth map.

[0150] To this end, the KL divergence is used to measure the difference between the predicted depth distribution and the true depth distribution. As shown in the formula:

[0151] ;

[0152] Where n is the number of depth hypothesis sets, z is the index of traversing the depth hypothesis set, represents the true probability distribution of each pixel in the depth hypothesis set, and the probability is generated by mapping the true depth map in the data set to the depth hypothesis set, represents the probability distribution of each pixel in the depth hypothesis provided by the multi-stage depth estimation module, is the pixel-level cross-entropy loss value.

[0153] ;

[0154] Where KL represents the KL divergence of the predicted distribution and the true distribution for each pixel, represents the depth distribution similarity loss value.

[0155] The comprehensive loss function combines the pixel-level cross-entropy loss and the depth distribution similarity loss, and gradually optimizes the depth estimation of each stage through weighted averaging. As shown in the formula:

[0156] ;

[0157] Where , is the weight coefficient, used to balance the influence of the pixel-level loss and the depth distribution similarity loss, total loss function, m represents different stages.

[0158] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-view reconstruction method based on geometric perception and attention fusion, characterized in that, Includes the following steps: Step S1: Prepare a multi-view stereo dataset and divide the image pairs in the dataset into reference images and source images; Step S2: Input the reference image and source image from step S1 into the multi-scale feature enhancement network to extract multi-scale feature maps, and input the multi-scale feature maps into the multi-stage depth estimation module to generate a preliminary depth map; Step S3: Perform resolution processing on the reference image in step S1, and perform resolution and normalization processing on the preliminary depth map in step S2 to obtain multi-scale geometric features. Step S4: Input the multi-scale geometric features of the reference image and the initial depth map processed in step S3 into the geometric perception feature fusion network to generate a geometric perception enhanced feature map; Step S5: Input the geometry-aware enhanced feature map into the multi-stage depth estimation module, perform iterative optimization, and output a refined depth map; Step S6: Construct a 3D reconstruction model based on a multi-scale feature enhancement network, a geometric perception feature fusion network, and a multi-stage depth estimation module; train the 3D reconstruction model using a joint loss function of pixel-level cross-entropy loss function and depth distribution similarity loss function; Step S3 specifically includes: Step S31: The reference image and the preliminary depth map are processed for resolution to obtain reference image feature I and preliminary depth map feature D. The preliminary depth map feature is normalized to obtain normalized depth map d. Step S32: Based on the reference image feature I and the normalized depth map d, multi-scale geometric features with different spatial resolutions are generated according to four modes: std, uv, z, or xyz. The four modes std, uv, z, and xyz represent feature representation methods from no geometric constraints to strong geometric constraints, respectively, providing the geometry-aware convolution module with multi-level geometric information from two-dimensional coordinates to three-dimensional space. In std mode, no geometric features are attached; In UV mode, a two-dimensional coordinate grid covering each pixel of the image is generated based on the size of the input reference image feature I, and the two-dimensional coordinate grid is linearly normalized to the [0,1] interval; the normalized two-dimensional coordinate grid is downsampled step by step through 2×2 average pooling to form two-channel features; In z mode, the normalized depth map d is sparsely pooled and downsampled to form a channel depth feature; In xyz mode, based on the camera intrinsic parameters corresponding to the images in the dataset, the normalized two-dimensional coordinates (u,v) of each pixel in the image are back-projected with the one-channel depth feature in z mode to the three-dimensional coordinates (x,y,z) in the camera coordinate system, and the three-channel features are output. In step S33, the multi-scale geometric features of the four modes are used to generate multi-scale representations according to spatial resolution levels 1, 1 / 2, 1 / 4, and 1 / 8, forming corresponding multi-scale geometric information. The multi-scale geometric features are then input into the geometry-aware convolutional module as multi-scale geometric feature information.

2. The multi-view reconstruction method based on geometric perception and attention fusion according to claim 1, characterized in that: The multi-view stereo dataset prepared in step S1 is as follows: The Multi-View Stereo Dataset is an indoor dataset that is a public dataset for multi-view stereo reconstruction and 3D modeling. It contains 124 different objects or scenes, each scene is taken from 49 or 64 different viewpoints, and each viewpoint has 7 different lighting conditions. The images in the dataset have a resolution of 1600×1200 pixels. The dataset selects one image as the reference image and the rest as source images. The dataset provides camera intrinsics, camera extrinsics, high-precision laser point clouds, and real depth map data for mapping pixel coordinates to the camera coordinate system.

3. The multi-view reconstruction method based on geometric perception and attention fusion according to claim 2, characterized in that: The reference image and source image from step S1 are input into a multi-scale feature enhancement network to extract multi-scale feature maps. These multi-scale feature maps are then input into a multi-stage depth estimation module to generate a preliminary depth map. Specifically: Step S21: The multi-scale feature enhancement network is divided into a feature selection module and a feature fusion module. The reference image and source image in step S1 are input into the feature selection module of the multi-scale feature enhancement network. Step S22: The reference image and the source image are processed by a preliminary convolution module consisting of two consecutive convolution units to generate an initial feature map of the first layer with a spatial resolution of H×W. Step S23: The feature selection module adopts a top-down downsampling convolution module. The first layer initial feature map is sequentially input into three downsampling convolution modules, and the spatial resolution is gradually reduced to H / 2×W / 2, H / 4×W / 4 and H / 8×W / 8. H / 8×W / 8 is the feature with the highest spatial resolution, and the second, third and fourth layer initial feature maps are obtained. Step S24: The first layer initial feature map, the second layer initial feature map, the third layer initial feature map, and the fourth layer initial feature map are input into the lightweight multi-dimensional fusion attention module, and feature enhancement is performed in the channel-height, channel-width, and spatial dimension directions to obtain the first layer enhanced initial feature map, the second layer enhanced initial feature map, the third layer enhanced initial feature map, and the fourth layer enhanced initial feature map. Step S25: The fourth-layer enhanced initial feature map is input into the multi-scale context aggregation module of the feature fusion module to obtain a deep initial feature map. The feature fusion module uses a bottom-up upsampling module to progressively upsample the deep initial feature map. The progressive upsampling module fuses the deep initial feature map with the first-layer enhanced initial feature map, the second-layer enhanced initial feature map, the third-layer enhanced initial feature map, and the fourth-layer enhanced initial feature map through a channel alignment module, outputting four multi-scale feature maps with different resolutions. The spatial resolution levels are respectively those of the reference image and the source image. ; Step S26: Input four multi-scale feature maps with different resolutions into the multi-stage depth estimation module to generate a preliminary depth map.

4. The multi-view reconstruction method based on geometric perception and attention fusion according to claim 1, characterized in that: The preliminary depth map features are normalized to obtain the normalized depth map d, which is represented as: ； Where d is the normalized depth map, and D is the preliminary depth map feature. , These represent the maximum and minimum values ​​of the initial depth map features.

5. The multi-view reconstruction method based on geometric perception and attention fusion according to claim 4, characterized in that: Step S4: Input the multi-scale geometric features of the reference image and the initial depth map processed in step S3 into the geometric perception feature fusion network to generate a geometric perception enhanced feature map; the geometric perception feature fusion network is divided into a first branch decoding module, a second branch decoding module and a final fusion module; Specifically: Step S41: The reference image and the initial depth map processed in step S3 are stitched together and input into the preliminary convolution module of the first branch decoding module to generate the first fusion feature map. The first fusion feature map is input into the five-level geometric perception convolution module. Each level of the geometric perception convolution module simultaneously fuses the multi-scale geometric feature information of the reference image and the initial depth map to obtain the second, third, fourth, fifth and sixth fusion feature maps, respectively. In step S42, the first branch decoding module performs progressive upsampling of the fused feature map through the deconvolution module. During the progressive upsampling process, the deconvolution module performs residual connection with the fused feature map output by the initial convolution module and the geometric perception convolution module; and generates a second initial depth map with the same resolution as the input reference image. Step S421: After the deconvolution module performs the first-level upsampling, it is connected with the residual of the fifth fusion feature map of the fourth-level geometric perception convolution module to obtain a spatial resolution of H / 8xW / 8. Step S422: After the deconvolution module performs the second-level upsampling, it is connected with the third fusion feature map residual of the second-level geometric perception convolution module to obtain a spatial resolution of H / 4xW / 4. Step S423: After the deconvolution module performs the third-level upsampling, it is connected with the second fusion feature map residual of the first-level geometric perception convolution module to obtain a spatial resolution of H / 4xW / 4. Step S424: After the deconvolution module performs the fourth level of upsampling, it is connected with the first fused feature map residual of the initial convolution module to obtain a spatial resolution of H / 2XW / 2. Step S425: After the deconvolution module performs the fifth level of upsampling, a spatial resolution of H×W is obtained; a second initial depth map is generated. Step S43: The normalized depth map d channel and the second initial depth map channel are concatenated and input into the preliminary convolution module of the second branch decoding module to generate the seventh fusion feature map. The seventh fusion feature map is input into the five-level geometric perception convolution module of the second branch decoding module. Each level of fusion simultaneously fuses the corresponding multi-scale geometric feature information of the reference image and the initial depth map. Step S44: The seventh fused feature map is input into the first and second geometric perception convolutional modules to obtain the eighth fused feature map. The eighth fused feature map is concatenated with the features output from the residual connection in step S424 to obtain the first joint feature. The first joint feature is input into the third and fourth geometric perception convolutional modules to obtain the ninth fused feature map. The ninth fused feature map is concatenated with the features output from the residual connection in step S421 to obtain the second joint feature. The second joint feature is input into the fifth geometric perception convolutional module to generate the third initial depth map. Step S45: The final fusion module consists of a five-level deconvolution module and five fusion processes. The five-level deconvolution module is divided into a first-level deconvolution module, a second-level deconvolution module, a third-level deconvolution module, a fourth-level deconvolution module, and a fifth-level deconvolution module. The third deconvolution module processes the first stage to output the first-level geometric perception enhanced feature map. The fourth deconvolution module processes the second stage to output the second-level geometric perception enhanced feature map. The fifth deconvolution module processes the third stage to output the third-level geometric perception enhanced feature map. Step S45, the first fusion process, the features of the sixth fusion feature map of the first branch decoding module and the third initial depth map of the second branch decoding module are added element by element and input into the first-level deconvolution module of the final fusion module to generate the first-level fusion feature; Step S46, the second fusion process: the fourth-level geometric perception module of the second branch decoding module adds the first-level fusion features element by element and inputs them into the second-level deconvolution module of the final fusion module to generate the second-level fusion features; Step S47, the third fusion process: the second-level geometric perception module of the second branch decoding module adds the second-level fusion features element-wise to the second-level fusion features, and inputs them into the third-level deconvolution module of the final fusion module to generate third-level fusion features. If it belongs to the first stage, the third-level fusion features are input into the first-level RGB deconvolution module, and the output is then combined with the output of the multi-scale feature enhancement network. Residual connections are made between multi-scale feature maps of different resolutions to obtain the first-stage geometric perception enhancement feature map. Step S48, the fourth fusion process: The first-level geometric perception module of the second branch decoding module adds the third-level fusion features element-wise, and inputs them into the fourth-level deconvolution module of the final fusion module to generate the fourth-level fusion features. If it belongs to the second stage, the fourth-level fusion features are input into the second-level RGB deconvolution module, and the output is then combined with the output of the multi-scale feature enhancement network. Residual connections are performed on the multi-scale feature maps to obtain the second-stage geometric perception enhancement feature maps. Step S49, the fifth fusion process: the initial convolution module of the second branch decoding module adds the fourth-level fusion feature element by element and inputs it into the fifth-level deconvolution module of the final fusion module to generate the fifth-level fusion feature. If it belongs to the third stage, the fifth-level fusion feature is input into the third-level RGB deconvolution module. After output, it is residually connected with the 1 / 1 resolution multi-scale feature map output by the multi-scale feature enhancement network to obtain the third-stage geometric perception enhancement feature map. Geometric perception-enhanced feature maps, expressed by the formula: ； Where Y is the geometry-aware enhanced feature map. These are features in multi-scale feature enhancement networks. It is a fusion feature of the first branch decoding module and the second branch decoding module. The final fusion module enhances the network's features by combining the fusion features of the first branch decoding module and the second branch decoding module, along with multi-scale features. Perform element-by-element fusion. and These represent the first branch decoding module and the second branch decoding module, respectively. I is the reference image feature, and d is the normalized depth map. It is a depth map guided by a reference image.

6. The multi-view reconstruction method based on geometric perception and attention fusion according to claim 5, characterized in that: Step S5: Input the geometry-aware enhanced feature map into the multi-stage depth estimation module for iterative optimization, and output a refined depth map; specifically: Step S51: Input the geometric perception enhancement feature map, which contains reference image features and source image features. Using the camera intrinsic and extrinsic parameters provided in the dataset, the source image features are projected onto the reference viewpoint according to the depth assumption of each pixel to achieve feature alignment. Step S52: Calculate the correlation between the features of the reference image and the aligned features of the source image by grouping them. Step S53: The correlation of all source image features is weighted and averaged to construct a preliminary cost volume. The preliminary cost volume is regularized by deep learning to perform three-dimensional spatial consistency modeling. Step S54: The initial cost volume for regularization is normalized by an exponential function to form the probability distribution of each pixel of the image under the depth hypothesis set; Step S55: Obtain the final depth value of each pixel in the image based on probability distribution regression, and combine the final depth values ​​of all pixels in the image to obtain a complete depth map. Step S56: Based on the final depth value of the current stage, generate a more refined set of depth hypotheses for the next stage by performing inverse depth resampling in the inverse depth domain of each pixel of the image. Step S57 involves multiple stages of processing, ultimately outputting the final depth map.

7. A multi-view reconstruction system based on geometric perception and attention fusion, applied to the multi-view reconstruction method based on geometric perception and attention fusion as described in claim 6, characterized in that: A 3D reconstruction model is constructed based on a multi-scale feature enhancement network, a geometric perception feature fusion network, and a multi-stage depth estimation module. The multi-scale feature enhancement network includes a feature selection module and a feature fusion module; The feature selection module includes a preliminary convolution module, a downsampling convolution module, and a lightweight multi-dimensional fusion attention module; The feature fusion module consists of a channel alignment module, an upsampling module, and a multi-scale context aggregation module; Among them, the geometric perception feature fusion network mainly includes a first branch coding module, a second branch coding module and a final fusion module; The first branch encoding module includes an initial convolution module, a geometry-aware convolution module, and a deconvolution module; The second branch encoding module includes an initial convolution module and a geometry-aware convolution module; The final fusion module receives the outputs of the first branch coding module and the second branch coding module, and performs scale fusion of the first branch coding module and the second branch coding module accordingly. The multi-stage depth estimation module consists of feature alignment, correlation calculation, regularization, normalization, depth regression, and inverse depth resampling.

Images