A cascaded multi-view stereo reconstruction method and system based on entropy guidance and attention mechanism, storage medium and product

Abstract

The application discloses a kind of cascade multi-view stereo reconstruction method, system, storage medium and product based on entropy guide and attention mechanism, belong to computer vision, three-dimensional reconstruction, feature pyramid, multi-view stereo reconstruction technical field, solve the three-dimensional reconstruction based on depth learning at present mainly applied in large scene, its focus is the integrity of overall model, and the reconstruction accuracy of detail such as edge region is ignored.The multi-view image set of the stereo to be measured is obtained, and pretreated;The result obtained by pretreatment is input into the cascade multi-view reconstruction network of entropy guide and attention mechanism to carry out cascade multi-view stereo reconstruction.The application is used for cascade multi-view stereo reconstruction.

Description

Technical Field

[0001] A cascaded multi-view stereo reconstruction method, system, storage medium, and product based on entropy guidance and attention mechanism are disclosed. This invention relates to the fields of computer vision, 3D reconstruction, feature pyramid, and multi-view stereo reconstruction technology. Background Technology

[0002] 3D reconstruction is a technology that obtains the geometric information of an object through 3D measurement and converts the real-world object or scene into a 3D model. It is widely used in fields such as autonomous driving, digital cultural relic preservation, real-world mapping, virtual reality, and 3D printing. Based on the data source used, 3D reconstruction methods can be divided into two main categories, such as... Figure 8 As shown, there are two types of 3D reconstruction: sensor-based and image-based. Sensor-based 3D reconstruction typically involves projecting a light beam onto the surface of an object and then determining the distance between the object and the device based on the time difference between the received and transmitted signals, thereby obtaining the size and shape of the object. This method requires specialized sensor equipment and computer software, so image-based 3D reconstruction, which is less expensive and more flexible, has become a research hotspot, especially multi-view stereo reconstruction (MVS Reconstruction).

[0003] MVS refers to a technique that captures images of the object under test from multiple perspectives and uses the multi-view images and corresponding camera parameters to complete stereo matching and depth estimation. It typically includes steps such as image feature extraction, camera pose estimation, and 3D point cloud reconstruction.

[0004] Traditional MVS methods obtain the depth information of feature points by matching corresponding feature points across multiple camera views using triangulation, and finally derive the point cloud model of the object through mathematical calculations. While achieving some success in ideal Lambertian scenes, the reconstruction completeness needs improvement when dealing with weakly textured regions and dense matching of reflective surfaces. Furthermore, the reconstruction effect is significantly affected by external factors such as illumination intensity and sampling angle. Therefore, deep learning-based MVS algorithms have emerged.

[0005] Deep learning-based 3D reconstruction is an evolution of traditional image-based 3D reconstruction techniques. It utilizes neural networks to generate depth maps from reference images, significantly reducing reconstruction time and addressing the challenges encountered by traditional 3D reconstruction methods when reconstructing workpieces with weak textures or non-Lambertian surfaces. This method uses neural networks to extract and match features from multiple images, generating more accurate and comprehensive depth information to produce a more precise 3D point cloud model of the object. Currently, multi-view deep learning-based... Figure 3Reconstruction has become the focus of research for most scholars in this field.

[0006] Since MVSNet was proposed by Yao et al. in 2018, subsequent deep learning-based MVS algorithms have basically been improvements upon it. MVSNet was the first end-to-end multi-view algorithm based on deep learning. Figure 3 The dimensional reconstruction framework, its network structure is as follows: Figure 9 As shown, the network takes a reference image and multiple source images as input. After feature extraction, matching cost volume construction, cost volume regularization, and depth regression, a depth map is obtained.

[0007] Specifically, MVSNet takes N 3-channel images as input (1 ref and N-1 src), with width and height W and H respectively. After passing through a feature extraction module consisting of 8 CNN layers, N feature maps are obtained. By assuming different depth values, the feature map of src is projected onto the feature map of ref using a differentiable homography transformation to obtain the feature volume. Then, the feature volume is constructed into a cost volume using a variance metric. Next, a 3D convolutional network is used to regularize the cost volume, generating the probability distribution of each pixel under different depth assumptions. Finally, the dense depth map is obtained by calculating the maximum probability.

[0008] The feature extraction module in this network structure consists of a simple 8-layer CNN, which cannot obtain more refined features and is difficult to meet the accuracy requirements for object reconstruction in most engineering tasks today. Secondly, the cost volume regularization part generally uses a 3D CNN method, which consumes a lot of video memory resources, has high requirements for device hardware, cannot meet the 3D reconstruction of high-resolution images, and is also difficult to meet real-time requirements.

[0009] In summary, the existing technology has the following technical problems:

[0010] 1. Currently, deep learning-based 3D reconstruction is mainly applied to large-scale scenes, such as buildings and maps, focusing on the integrity of the overall model while ignoring details such as the reconstruction accuracy of edge regions;

[0011] 2. In order to improve reconstruction accuracy, many current 3D reconstruction networks generally adopt large-scale cost volume construction and deep 3D convolutional regularization, which leads to high memory consumption and slow inference speed, making it difficult to deploy in high-resolution images or resource-constrained scenarios. Summary of the Invention

[0012] To address the problems mentioned above, the present invention aims to provide a cascaded multi-view stereo reconstruction method, system, storage medium, and product based on entropy guidance and attention mechanism. This addresses the issue that current deep learning-based 3D reconstruction is mainly applied to large-scale scenes, focusing on the integrity of the overall model while neglecting the reconstruction accuracy of details such as edge regions.

[0013] To achieve the above objectives, the present invention adopts the following technical solution:

[0014] A cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism includes the following steps:

[0015] Step 1: Obtain a multi-view image set of the stereo to be tested and perform preprocessing;

[0016] Step 2: Input the preprocessing results obtained in Step 1 into the cascaded multi-view reconstruction network with entropy guidance and attention mechanism to perform cascaded multi-view stereo reconstruction;

[0017] The cascaded multi-view reconstruction network includes an improved FPN structure that extracts feature maps at three different scales from the preprocessing results obtained in step 1, a depth estimation module that estimates the depth map from coarse to fine using an entropy-guided sampling strategy on the feature maps obtained from the improved FPN structure, and a joint refinement module that optimizes the depth map obtained from the depth estimation module based on the feature maps obtained from the improved FPN structure.

[0018] Furthermore, the specific steps of step 1 are as follows:

[0019] Step 1.1: Obtain a multi-view image set of the stereo image to be tested, and select from it. Zhang Gaowei , width is The image includes one reference image, denoted as [image name missing]. ,and Zhang Yuan's image, denoted as ,in, Indicates the first Zhang Yuan's image;

[0020] Step 1.2, using Indicates the first Zhang Yuan's image camera intrinsic parameters, rotation matrix, and translation vector. Represents reference image The camera's internal and external parameters.

[0021] Furthermore, the improved FPN structure in step 2 replaces the 1×1 convolution in the original FPN structure with a feature selection module and adds a feature alignment module; specifically, it involves downsampling the source image or reference image twice to obtain feature maps of high, medium, and low resolutions, and then performing convolution operations on each to obtain the corresponding features. , and ,right , and The first feature selection module, the second feature selection module, and the third feature selection module perform feature selection respectively. These are the features of the source or reference image;

[0022] The output of the third feature module is processed by a global pooling layer to obtain direction-sensitive attention weights, which are then weighted... The feature map is obtained above ;

[0023] For feature maps Upsampling is performed to obtain the feature map Feature map output by the second feature selection module The second feature alignment module performs feature alignment, and the feature map output by the second feature alignment module is... Then with feature map Adding them together yields the feature map. ;

[0024] For feature maps Upsampling is performed to obtain the feature map Feature map output by the first feature selection module The first feature alignment module performs feature alignment, and the feature map output by the first feature alignment module is shown. Then with feature map Adding them together yields the feature map. .

[0025] Furthermore, the first feature selection module, the second feature selection module, and the third feature selection module include a horizontal global average pooling layer and a vertical global average pooling layer that decompose the input features into horizontal position encoding and vertical position encoding along the horizontal and vertical directions.

[0026] The formula for the horizontal global average pooling layer is: ;

[0027] The formula for the vertical global average pooling layer is: ;

[0028] in, These represent the width and height of the features output by the first feature selection module, the second feature selection module, and the third feature selection module, respectively. and The input tensors represent the feature coordinate positions along the horizontal and vertical directions, respectively. and This represents the pooling output along the horizontal and vertical directions;

[0029] A 1×1 convolutional layer, a batch normalization layer, and a... are used to process the concatenated results of horizontal and vertical position encoding in sequence. The non-linear activation function layer has the following formula:

[0030]

[0031] in, Indicates Intermediate features obtained by processing with a nonlinear activation function layer Indicates a splicing operation; This indicates that a 1×1 convolutional layer scales the number of channels to 1. , Hyperparameters set to limit computational complexity For batch normalization layer, for Nonlinear activation function;

[0032] A 1×1 convolutional layer that processes intermediate features sequentially along the horizontal and vertical directions. The activation function layer has the following formula:

[0033]

[0034]

[0035] in, and From The horizontal and vertical components separated from the middle, and For two Attention weights obtained in the horizontal and vertical directions from the activation function layer, respectively;

[0036] The feature tensor is obtained by multiplying the horizontal and vertical attention weights by the input features, as shown in the formula:

[0037]

[0038] in, This indicates the spatial location of the input features corresponding to the first feature selection module, the second feature selection module, and the third feature selection module. Eigenvalues ​​at;

[0039] Furthermore, the specific implementations of the first feature alignment module and the second feature alignment module are as follows:

[0040] For the corresponding feature maps of the input respectively Upsampling is performed and the resulting feature maps are concatenated along the channel dimension with the feature maps output by the corresponding first and second feature selection modules to generate a fused feature map. Then, convolution is used to predict the offset vector at each position. The formula is:

[0041]

[0042] in, It's a splicing operation. This represents the convolution operation. Indicates the feature map The feature map obtained by upsampling, Indicates the feature The result of feature selection, in the first feature alignment module, The value of is 1, in the second feature alignment module, The value of is 2;

[0043] Based on offset For feature maps Dynamic sampling is performed to shift the sampling positions of high-level features towards the semantic key regions of low-level features, generating aligned features. The formula is:

[0044]

[0045] in, for Activation function It is a deformable convolution;

[0046] Aligned features With features The feature maps are fused together to output the final feature map. The formula is:

[0047] .

[0048] Furthermore, the specific implementation of the depth estimation module is as follows:

[0049] Based on feature maps Obtain the initial depth map The specific process is as follows:

[0050] Will Feature map of Zhang Yuan's image Feature maps projected onto the reference view using the differentiable homography transformation formula The plane, the formula is:

[0051]

[0052] Formula for differentiable homography transformation:

[0053]

[0054] in, and They represent the first Feature maps obtained from Zhang Yuan's image and the reference image using the improved FPN structure Is the depth as The Differentiable homography transformation factor of each source image. Indicates the first Camera intrinsic parameters, rotation matrix, and translation vector for each source image. Represents reference image Camera internal and external parameters, A unit vector refers to the principal axis of the reference camera; Indicates the first All pixels in Zhang Yuan's image, Indicates will Convert to the corresponding pixel in the reference view;

[0055] A depth layer is defined in the projection space, which is discretized onto the depth layer plane, and feature maps of each source image are defined. Pixels after projection Each feature map will be matched on some plane within it. Pixels after projection They all have depth and dimension and obtain the feature body ,in, Indicates the number of channels. and These represent height and width, respectively.

[0056] The obtained first Features of Zhang Yuan's Image Constructing the cost body using variance aggregation The formula is:

[0057] in, This indicates the total number of input reference images and source images. represent The average value of each feature body;

[0058] The cost volume is encoded and decoded. In the encoding stage, the cost volume is downsampled layer by layer through 3D convolution and pooling operations to expand the receptive field and capture global depth and spatial context information. In the decoding stage, coarse global information and local details are combined through upsampling and skip connections.

[0059] The decoded result is regularized, and the regularized cost volume is then applied in the depth dimension. The probability distribution is obtained by finding the probability volume. ;

[0060] For probability bodies The initial depth map is obtained by calculating the expectation along the depth direction. The size is Initial depth map pixels for:

[0061] in, Representing the The actual depth value corresponding to each depth assumption. Represents the initial depth map pixels In the The probability of a depth hypothesis, ;

[0062] Based on feature maps and initial depth map Obtain depth map The specific process is as follows:

[0063] Calculate the probability body The entropy, the formula is:

[0064]

[0065] in, Initial depth map pixels entropy, Initial depth map pixels In the The probability of a depth hypothesis;

[0066] based on Using linear interpolation, each initial depth map is dynamically adjusted. Pixels Depth hypothesis number The formula is:

[0067]

[0068] in, To ensure rounding to the nearest integer It is an integer. The maximum depth hypothesis number, For the minimum depth hypothesis number, and These are the maximum entropy value and the minimum entropy value, respectively.

[0069] based on Define the local depth range using the following formula:

[0070]

[0071] in, It controls the intensity of the influence of entropy on the range. It is the basic depth range. and These are prior parameters, which are updated as the network is trained.

[0072] Based on deep hypothesis number and local depth range The formula for calculating the dynamic depth interval is:

[0073] Based on dynamic depth interval and Based on the initial depth map Depth estimation in Nearby generation The assumption of equal depth intervals is given by the following formula:

[0074]

[0075] Will As a depth dimension, and based on and feature map According to the feature map Obtain the initial depth map The method of obtaining depth map ;

[0076] Based on feature maps and depth map Obtain depth map The specific process is as follows:

[0077] Based on depth map The entropy of the probability volume of each pixel Normalize it to the [0,1] interval, the formula is:

[0078]

[0079] in, , These represent the maximum and minimum entropy values, respectively. Represents pixels Normalized entropy value;

[0080] Based on the normalized entropy value, a different number of depth hypotheses are dynamically assigned to each pixel. The formula is:

[0081]

[0082] in, To ensure rounding to the nearest integer It is an integer. and These are the maximum and minimum depth hypothesis numbers, respectively;

[0083] To estimate Methods for estimating depth values Centered on the depth map The depth entropy of each pixel and the prior parameters obtained during network training define a pixel. Local depth search range:

[0084]

[0085] in, Indicates the base depth interval. It is a scaling factor that controls the width of the search range;

[0086] Based on pixels To determine the local depth search range, a non-uniform sampling strategy is used to generate a set of depth candidate planes, as shown in the formula:

[0087]

[0088] Among them, normalized index index Mapping to interval , used for input The function makes The nearest sampling point is the one closest to the center. and The point is closest to the boundary of the sampling interval; disturbance control Will Interval linear scaling to , Ensure that all depth candidates are concentrated around the center value. Local area;

[0089] The depth candidate plane set outputs a set of non-uniform depth hypothesis values ​​for each pixel location. ;

[0090] Based on the set of non-uniform depth assumptions As a depth dimension combined with feature maps According to the feature map Obtain the initial depth map The method of obtaining depth map .

[0091] Furthermore, the joint refinement module jointly models the photometric consistency constraint and the edge structure constraint to form an objective function that describes the matching error between the current depth map and the real image:

[0092]

[0093] in, It is a balance coefficient used to adjust the weights of photometric data items and edge constraint items. For feature consistency constraints, Indicates pixel position, Represents pixels Eigenvalues ​​at; Indicates the use of depth Reference pixel Feature values ​​of pixels obtained after projection onto the source image For edge constraint terms, This represents the gradient of the depth map. Gradient at edge location compared to reference image Matching, edge weights , It is an edge map obtained from a reference image using the Canny algorithm. At the same time, the gradient error is amplified, making the optimization more sensitive at the edges. Controlling edge contribution; To measure the difference between depth transitions and image edge transitions;

[0094] Edge maps are obtained from reference images using the Canny algorithm. The specific steps are as follows:

[0095] Reference image and Convolution, to obtain a convolutional image, the formula is:

[0096]

[0097]

[0098] in, Indicates the Gaussian kernel in coordinates The value, The standard deviation determines the smoothness of the filter. Represents a convolutional image;

[0099] Through horizontal and vertical directions Operator convolution kernel calculates convolution image gradient components and The formula is:

[0100] ,

[0101] Using gradient components and Calculate the gradient magnitude:

[0102]

[0103] in, Gradient magnitude represents the intensity of a pixel's gradient, i.e., the absolute degree of brightness change. High magnitude values ​​typically correspond to image edges, reflecting abrupt changes in local brightness. and For pixels gradient components;

[0104] Using gradient components and Calculate the gradient direction:

[0105]

[0106] gradient direction Non-maximum suppression is performed on each pixel in four gradient directions: 0°, 45°, 90°, and 135°. ( At point ), compare neighboring pixels along the gradient direction, retaining only local maxima:

[0107]

[0108] in, , Let the points be the neighboring points on both sides of the gradient direction, where, express( ), express , or , express , or ;

[0109] gradient magnitude Normalization to get Based on the normalized results, we obtain ,and , The above are strong edges, which should be preserved directly; ~ A weak edge must be adjacent to a strong edge to be preserved.

[0110] Classify pixels:

[0111]

[0112] Where 1 represents a strong edge. 0 represents a weak edge, and 0 represents a non-edge.

[0113] Then the binary edge map That is, an edge map obtained from the reference image is:

[0114]

[0115] Based on the objective function, the Gauss-Newton method is used for iterative optimization to adjust the depth map. This minimizes the objective function.

[0116] The specific steps are as follows:

[0117] Construct the residual vector:

[0118]

[0119] in, , , by objective function Therefore, after concatenating the two residuals, the objective is to minimize them. , Represents the residual vector;

[0120] Linearization:

[0121] In the current depth estimation Minimize perturbation at the point Then, for the residual vector Perform a first-order Taylor expansion:

[0122]

[0123] Among them, the minimum disturbance It is the increment used in the Gauss-Newton method for iteratively updating variables; it represents the increment in the current depth map. Near the point of view, the minimum correction made to reduce the residual function. It is the Jacobian matrix, used to linearize the variation of residuals with depth. Each element Defined as the first The residuals for the first... The partial derivative of each depth pixel, , (·) indicates the first One residual, ( ) indicates using a depth map The calculated first One error, It is to find the partial derivative. ( ) indicates when depth ;

[0124] Solving the normal equation: Based on the residual vector After performing a first-order Taylor expansion, solving the linear system yields the minimum increment of the current depth, which corresponds to the depth supplement for each pixel. The formula is:

[0125]

[0126] in, for Transpose of;

[0127] Update and convergence:

[0128] After obtaining the minimum increment, the depth value is updated. Through this update, the depth map is... At each location, corrections are made based on local feature inconsistencies and edge gradient mismatches, using the following formula:

[0129]

[0130] If the iteration termination condition is met. To reconstruct the cascaded multi-view stereo, otherwise, use the currently updated depth map as the new initial value and execute the step that minimizes the objective function again.

[0131] A cascaded multi-view stereo reconstruction system based on entropy guidance and attention mechanism includes a memory, a processor, and a computer program stored in the memory. The system is characterized in that the processor executes the computer program to implement the steps of the cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism.

[0132] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism.

[0133] A computer program product includes a computer program that, when executed by a processor, implements the steps of the cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism.

[0134] Compared with the prior art, the beneficial effects of this invention are as follows:

[0135] The cascaded multi-view network proposed in this invention has significant advantages in feature extraction, depth estimation accuracy, edge preservation capability, and computational efficiency. It possesses advantages such as enhanced edge reconstruction, reduced computational resources, and applicability to high resolutions, making it particularly suitable for refined 3D reconstruction scenarios. Overall, this proposal achieves lower memory usage and shorter inference time while ensuring high-precision 3D reconstruction results, demonstrating greater practicality and engineering application value. Specifically:

[0136] I. The improved FPN structure proposed in this invention adopts a multi-scale pyramid structure, combined with coordinate attention mechanism and deformable convolution alignment to enhance the ability to express edge and detail features, effectively enhancing the perception of image edges and geometric details.

[0137] Second, the estimation module in this invention introduces an uncertainty-guided sampling strategy based on information entropy and a non-uniform depth sampling method, which enables the network to adaptively adjust the number and distribution density of depth assumptions according to the prediction confidence of pixels, thereby improving accuracy while reducing redundant computation.

[0138] Third, this invention adopts a three-level cascaded structure to gradually refine the depth map from coarse to fine, avoiding the waste of resources caused by constructing the full cost volume at high resolution. In the final stage, an optimization function is constructed by combining edge information and feature consistency constraints, and Gauss-Newton iteration is used for refinement, so that the generated depth map is superior to the existing technology in terms of edge continuity and geometric consistency. Attached Figure Description

[0139] Figure 1 This is a schematic diagram of the overall structure of the cascaded multi-view reconstruction network ECA-MVSNet in this invention;

[0140] Figure 2 This is a schematic diagram of the overall structure of the improved FPN in this invention;

[0141] Figure 3 This is a schematic diagram of the structure of the first feature selection module, the second feature selection module, and the third feature selection module in this invention;

[0142] Figure 4 This is an example of deformable convolution in this invention;

[0143] Figure 5This is a schematic diagram of the structure of the first feature selection module and the second feature selection module in this invention;

[0144] Figure 6 This is a schematic diagram of the depth space partitioning in this invention;

[0145] Figure 7 This is a schematic diagram of the structure of the joint refinement module in this invention;

[0146] Figure 8 A schematic diagram illustrating the classification of 3D reconstruction techniques in existing technologies;

[0147] Figure 9 This is a schematic diagram of the MVSNet network structure in the existing technology;

[0148] Figure 10 A schematic diagram of the improved FPN structure;

[0149] Figure 11 This is a diagram illustrating the hardware configuration and software environment.

[0150] Figure 12 Training loss function and accuracy convergence curve for cascaded multi-view reconstruction network;

[0151] Figure 13 This is a comparison chart of the experimental results. Detailed Implementation

[0152] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. In terms of improving reconstruction accuracy, the present invention introduces a multi-scale feature extraction architecture to replace the simple 8-layer CNN structure in the MVSNet network. This architecture fuses low-resolution feature maps rich in semantic information with high-resolution feature maps rich in spatial information, enhancing the responsiveness of high-level semantic information at edge details, thereby improving reconstruction accuracy in structurally complex or boundary-blurred regions. Simultaneously, a joint refinement module for the depth map is added. This module, combining the Canny edge detection algorithm and the Gauss-Newton method, significantly improves the continuity and geometric consistency at depth transitions, effectively solving the problem of large edge errors in traditional MVS methods.

[0153] In deep learning MVS methods, the network typically represents the matching probability distribution of pixels under different depth assumptions using a probability volume. A relatively uniform probability distribution indicates high uncertainty in the depth estimation of the current pixel; a concentrated probability distribution indicates a more reliable depth estimation. This proposal introduces an entropy-guided non-uniform depth sampling strategy. It uses entropy to measure the uncertainty of this probability distribution and adaptively adjusts the depth sampling strategy based on the entropy value—increasing sampling density in high-uncertainty regions and decreasing the number of samples in low-uncertainty regions, thereby achieving adaptive allocation of computational resources. Simultaneously, a multi-stage depth estimation framework from coarse to fine is adopted, first performing coarse depth regression at low resolution, and then gradually refining it at medium to high resolution stages. This cascaded structure effectively avoids the memory bottleneck caused by directly constructing the complete cost volume at high resolution, making the entire network more controllable in terms of runtime and resource consumption, while simultaneously achieving an optimization process of layer-by-layer refinement of the depth map.

[0154] A cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism includes the following steps:

[0155] Step 1: Obtain a multi-view image set of the stereo image to be tested and perform preprocessing; the specific steps are as follows:

[0156] Step 1.1: Obtain a multi-view image set of the stereo image to be tested, and select from it. Zhang Gaowei , width is The image includes one reference image, denoted as [image name missing]. ,and Zhang Yuan's image, denoted as ,in, Indicates the first Zhang Yuan's image;

[0157] Step 1.2, using Indicates the first Zhang Yuan's image camera intrinsic parameters, rotation matrix, and translation vector. Represents reference image The camera's internal and external parameters.

[0158] Step 2: Input the preprocessing results obtained in Step 1 into the cascaded multi-view reconstruction network with entropy guidance and attention mechanism to perform cascaded multi-view stereo reconstruction;

[0159] The cascaded multi-view reconstruction network includes an improved FPN structure that extracts feature maps at three different scales from the preprocessing results obtained in step 1, a depth estimation module that estimates the depth map from coarse to fine using an entropy-guided sampling strategy on the feature maps obtained from the improved FPN structure, and a joint refinement module that optimizes the depth map obtained from the depth estimation module based on the feature maps obtained from the improved FPN structure.

[0160] The improved FPN structure replaces the 1×1 convolutions in the original FPN structure with a feature selection module and adds a feature alignment module. Specifically, it involves downsampling the source or reference image twice to obtain feature maps of high, medium, and low resolutions, and then performing convolution operations on each of these to obtain the corresponding feature maps. , and ,right , and The first feature selection module, the second feature selection module, and the third feature selection module perform feature selection respectively. This refers to the feature map of the source or reference image. The improved FPN structure performs depth prediction in three stages: predicting the initial depth map at the coarsest level, and generating a high-resolution depth map at the finest level through iterative optimization via progressive upsampling. The convolution size in the original FPN structure is as follows: Figure 10 As shown. The Feature Selection Module (FSM) and the Feature Alignment Module (FAM) are as follows: Figure 2 As shown, the first, second, and third are used to distinguish the modules or layers corresponding to different objects being processed.

[0161] The output of the third feature module is processed by a global pooling layer to obtain direction-sensitive attention weights, which are then weighted... The feature map is obtained above In terms of lateral connections, unlike traditional feature pyramids that simply use a 1×1 convolution, this invention replaces the 1×1 convolution with a third feature selection module (FSM) based on Coordinate Attention (CA) to select important features from high-level feature maps and suppress redundant features. The third feature selection module, FSM, generates direction-sensitive attention weights by decomposing global pooling in the horizontal and vertical directions, and then weights them accordingly. The above method suppresses noise and enhances edge information, strengthening key regions and thus obtaining a more powerful feature representation. .

[0162] For feature maps Upsampling is performed to obtain the feature map Feature map output by the second feature selection module The second feature alignment module performs feature alignment, and the feature map output by the second feature alignment module is... Then with feature map Adding them together yields the feature map. ;Right now The second feature selection module (FSM) in the horizontal connection performs feature filtering and generates... . After upsampling, it becomes... Feature maps of the same size In the second feature alignment module FAM, the following will be implemented: and Channel concatenation is performed, and the fused feature map is used as input to a deformable convolutional offset prediction network to obtain the offset of each sampling point. Then, the predicted offset is used to... Dynamic adjustments are made to enable upsampling features. and Feature space alignment, generation Finally, the two feature maps are... and Perform feature fusion and output feature maps. .

[0163] For feature maps Upsampling is performed to obtain the feature map Feature map output by the first feature selection module The first feature alignment module performs feature alignment, and the feature map output by the first feature alignment module is shown. Then with feature map Adding them together yields the feature map. That is, high-level characteristics. The enhanced features are generated after passing through the first feature selection module (FSM) with horizontal connections. . After upsampling, the first feature alignment module FAM is used to align the samples with the target data. Feature space alignment, generation Then and Perform feature fusion to generate feature maps .

[0164] The first, second, and third feature selection modules employ a coordinate attention mechanism. Because they consider not only channel information but also spatial coordinate information, this helps to dynamically filter key information from different feature levels during feature selection, suppress redundant features, and more accurately locate important regions, especially in tasks requiring spatial information, such as stereo matching. The overall flow of each feature selection module (FSM) is as follows: Figure 3 As shown.

[0165] The first feature selection module, the second feature selection module, and the third feature selection module include a horizontal global average pooling layer and a vertical global average pooling layer that decompose the input features into horizontal position encoding and vertical position encoding along the horizontal and vertical directions.

[0166] The formula for the horizontal global average pooling layer is: ;

[0167] The formula for the vertical global average pooling layer is: ;

[0168] in, These represent the width and height of the features output by the first feature selection module, the second feature selection module, and the third feature selection module, respectively. and The input tensors represent the feature coordinate positions along the horizontal and vertical directions, respectively. and This represents the pooling output along the horizontal and vertical directions;

[0169] A 1×1 convolutional layer, a batch normalization layer, and a... are used to process the concatenated results of horizontal and vertical position encoding in sequence. The non-linear activation function layer has the following formula:

[0170]

[0171] in, Indicates Intermediate features obtained by processing with a nonlinear activation function layer Indicates a splicing operation; This indicates that a 1×1 convolutional layer scales the number of channels to 1. , Hyperparameters set to limit computational complexity (default r=32). For batch normalization layer, for Nonlinear activation function;

[0172] A 1×1 convolutional layer that processes intermediate features sequentially along the horizontal and vertical directions. The activation function layer has the following formula:

[0173]

[0174]

[0175] in, and From The horizontal and vertical components separated from the middle, and For two Attention weights obtained in the horizontal and vertical directions from the activation function layer, respectively;

[0176] The feature tensor is obtained by multiplying the horizontal and vertical attention weights by the input features, as shown in the formula:

[0177]

[0178] in, This indicates the spatial location of the input features corresponding to the first feature selection module, the second feature selection module, and the third feature selection module. Eigenvalues ​​at;

[0179] In FPN, when high-level features with low resolution and strong semantics are fused with low-level features with high resolution and weak semantics through bilinear upsampling, simple interpolation operations cannot dynamically adapt to the geometric deformation or viewpoint changes of the target. Therefore, this invention uses the Feature Alignment Module (FAM) to process the feature maps to be fused before feature aggregation, so that the features are aligned before fusion.

[0180] In FAM (Feature Alignment Model), deformable convolution is introduced as a feature alignment function. The principle is based on the convolutional network learning an offset, allowing the convolutional kernel to dynamically adjust its sampling position on the input feature map. Simply put, traditional convolution has a fixed sampling pattern (such as a 3×3 rectangle), while the sampling point position of deformable convolution is automatically adjusted according to the geometric characteristics of the input content, such as... Figure 4 As shown;

[0181] Feature alignment uses deformable convolution to predict the offset between high-level and low-level features, dynamically mapping high-level features to the spatial coordinates of low-level features. This solves the geometric misalignment problem in multi-scale feature fusion and achieves alignment in the feature space. The overall structure of FAM is as follows: Figure 5 As shown.

[0182] The specific implementations of the first feature alignment module and the second feature alignment module are as follows:

[0183] For the corresponding feature maps of the input respectively Upsampling is performed and the resulting feature maps are concatenated along the channel dimension with the feature maps output by the corresponding first and second feature selection modules to generate a fused feature map. Then, convolution is used to predict the offset vector at each position. The formula is:

[0184]

[0185] in, It's a splicing operation. This represents the convolution operation. Indicates the feature map The feature map obtained by upsampling, Indicates the feature The result of feature selection, in the first feature alignment module, The value of is 1, in the second feature alignment module, The value of is 2;

[0186] Based on offset For feature maps Dynamic sampling is performed to shift the sampling positions of high-level features towards the semantic key regions of low-level features, generating aligned features. The formula is:

[0187]

[0188] in, for Activation function It is a deformable convolution;

[0189] Aligned features With features The feature maps are fused together to output the final feature map. The formula is:

[0190] .

[0191] Traditional MVSNet constructs the complete cost volume at a fixed resolution, resulting in memory requirements that increase cubically with image resolution, limiting the processing of high-resolution images. The network of this invention first constructs the cost volume at a low resolution for coarse depth estimation, and then refines it by constructing a local cost volume around the previously estimated depth value on a higher-resolution image. This approach significantly reduces the computational cost and memory requirements of each layer, making the network more compact and efficient.

[0192] An entropy-guided sampling strategy based on probabilistic volume entropy is introduced. After each stage of depth estimation, the network obtains the probability distribution of the corresponding pixel under different depth assumptions. By calculating the entropy of this probability distribution, the uncertainty of the current pixel's depth estimation can be reflected. When the entropy value is high, it indicates that the depth probability distribution of the pixel is relatively dispersed, with significant estimation uncertainty. In this case, the number of depth samples in that region will be increased and the sampling interval will be reduced in subsequent depth estimation stages. When the entropy value is low, it indicates that the depth distribution is relatively concentrated, so the sampling density will be reduced, thereby reducing computational overhead. Through this entropy-guided adaptive sampling mechanism, the network can allocate more computational resources in complex regions (weak textures, edges, occluded regions) while reducing redundant computation in stable regions, thus significantly reducing the overall computational cost while ensuring reconstruction accuracy.

[0193] The specific implementation of the depth estimation module is as follows:

[0194] Based on feature maps Obtain the initial depth map The specific process is as follows:

[0195] Will Feature map of Zhang Yuan's image Feature maps projected onto the reference view using the differentiable homography transformation formula The plane, the formula is:

[0196]

[0197] Formula for differentiable homography transformation:

[0198]

[0199] in, and They represent the first Feature maps obtained from Zhang Yuan's image and the reference image using the improved FPN structure Is the depth as The Differentiable homography transformation factor of each source image. Indicates the first Camera intrinsic parameters, rotation matrix, and translation vector for each source image. Represents reference image Camera internal and external parameters, A unit vector refers to the principal axis of the reference camera; Indicates the first All pixels in Zhang Yuan's image, Indicates will Convert to the corresponding pixel in the reference view;

[0200] The public dataset DTU used for network training has a depth range of 425mm-925mm, so uniform sampling is performed within this range. The spatial depth is set to 48 layers, resulting in the projection space being discretized into 48 planes. All points on the feature maps will be matched onto one of these planes, such as... Figure 6 As shown, the feature maps of each source image Pixels after projection Each feature map will be matched on some plane within it. Pixels after projection They all have depth and dimension and obtain the feature body ,in, Indicates the number of channels. and These represent height and width, respectively.

[0201] The obtained first Features of Zhang Yuan's Image Constructing the cost body using variance aggregation The formula is:

[0202] in, This indicates the total number of input reference images and source images. represent The average value of each feature body;

[0203] The obtained feature volumes are used to construct a cost volume through variance aggregation. Because all source view feature maps are projected onto the corresponding depth reference view feature map, if the true depth of a pixel is close to... If so, then the transformed eigenvalues ​​at that point should be approximate, meaning the variance at that point should be very small. Therefore, through variance aggregation, the smaller the variance at a point on a certain depth plane, the higher the confidence level at that depth.

[0204] A structure similar to 3D U-Net is adopted for encoding and decoding of the cost volume. In the encoding stage, the cost volume is downsampled layer by layer through 3D convolution and pooling operations to expand the receptive field and capture global depth and spatial context information. In the decoding stage, upsampling and skip connections combine coarse global information with local details, resulting in a smoother and more robust cost volume feature output.

[0205] The decoded result is regularized, and the regularized cost volume is then applied in the depth dimension. The probability distribution is obtained by finding the probability volume. ;

[0206] For probability bodies The initial depth map is obtained by calculating the expectation along the depth direction. The size is Initial depth map pixels for:

[0207]

[0208] in, Representing the The actual depth value corresponding to each depth assumption. Represents the initial depth map pixels In the The probability of a depth hypothesis, ;

[0209] In this step, the network can automatically learn how to "filter out" false matches and noise, while reinforcing those that are effective in multi-view scenarios. Figure 1 A match that is particularly prominent in terms of compatibility.

[0210] In multi-view stereo reconstruction, cascaded multi-view reconstruction networks typically construct depth probability volumes to represent the matching probability distribution of pixels under different depth assumptions. For each pixel, its probability distribution under each depth assumption reflects the degree of uncertainty in the depth estimate of that pixel. When the probability distribution is relatively uniform, the entropy value is large, indicating that the depth estimate of that pixel has high uncertainty; when the probability distribution is relatively concentrated, the entropy value is small, indicating that the depth estimate is more reliable.

[0211] Based on the above characteristics, this invention utilizes entropy to guide the depth estimation process, that is, dynamically adjusting the depth search range and sampling density according to the entropy value of the depth probability distribution. When the entropy value is high in certain areas, it indicates that there is significant uncertainty in the depth estimation of those areas. In subsequent stages, the depth sampling density is increased and the depth search range is narrowed for those areas. Conversely, for areas with low entropy values, the sampling density is reduced, thereby reducing redundant computation.

[0212] Here, the obtained higher resolution feature map Homography transformation projection is also performed. The difference is that this layer does not use a fixed depth hypothesis and depth interval, but instead utilizes the entropy of the probability volume in the feature map. The depth map obtained in the coarse stage is further refined, and the depth assumption is adaptively adjusted by combining an entropy-guided strategy, so that the network can pay more attention to areas with high uncertainty in depth estimation, thereby improving the depth estimation accuracy of complex areas.

[0213] Based on feature maps and initial depth map Obtain depth map The specific process is as follows:

[0214] Calculate the probability body The entropy, the formula is:

[0215]

[0216] in, Initial depth map pixels entropy, Initial depth map pixels In the The probability of a depth hypothesis;

[0217] based on Using linear interpolation, each initial depth map is dynamically adjusted. Pixels Depth hypothesis number The formula is:

[0218]

[0219] in, To ensure rounding to the nearest integer It is an integer. The maximum depth hypothesis number, For the minimum depth hypothesis, here, In the initial depth map In the calculation, the fixed depth assumption number is 48; while in the depth map In the calculation, to balance computational efficiency and accuracy, a depth hypothesis number is dynamically set for each pixel to ensure sufficient depth sampling density in high-uncertainty regions (maximum 32) while retaining basic depth search capability in low-uncertainty regions (minimum 8). This value is an empirical parameter that achieves a good balance between accuracy and computational cost, and can be adjusted according to image resolution, scene complexity, video memory resources, or training dataset.

[0220] and These are the maximum entropy value and the minimum entropy value, respectively.

[0221] based on Define the local depth range using the following formula:

[0222]

[0223] in, It controls the intensity of the influence of entropy on the range. , It is the basic depth range. mm, and These are prior parameters, which are updated as the network is trained.

[0224] Based on deep hypothesis number and local depth range The formula for calculating the dynamic depth interval is:

[0225] Based on dynamic depth interval and Based on the initial depth map Depth estimation in Nearby generation The assumption of equal depth intervals is given by the following formula:

[0226]

[0227] Will As a depth dimension, and based on and feature map According to the feature map Obtain the initial depth map The method of obtaining depth map That is, a cost body is constructed by variance aggregation, the cost body is regularized to obtain a probability body, and the expectation along the depth direction is calculated to obtain the depth map. ;

[0228] Based on feature maps and depth map Obtain depth map The specific process is as follows:

[0229] Depth map The entropy of the probability volume of each pixel This is used to measure the uncertainty of the pixel depth estimate. To better control the number of samples at the current stage, it is normalized to the [0,1] interval, as shown in the formula:

[0230]

[0231] in, , These represent the maximum and minimum entropy values, respectively. Represents pixels Normalized entropy value;

[0232] Based on the normalized entropy value, a different number of depth hypotheses are dynamically assigned to each pixel. To focus on regions of high uncertainty while maintaining computational efficiency, the formula is:

[0233]

[0234] in, To ensure rounding to the nearest integer It is an integer. and These are the maximum and minimum depth hypothesis numbers, respectively. ;

[0235] To estimate Methods for estimating depth values Centered on the depth map The depth entropy of each pixel and the prior parameters obtained during network training define a pixel. Local depth search range:

[0236]

[0237] in, Indicates the base depth interval. This is a scaling factor that controls the width of the search range; set it to 3.0.

[0238] To more effectively capture depth abrupt changes such as edges, based on pixels To determine the local depth search range, a non-uniform sampling strategy is used to generate a set of depth candidate planes, as shown in the formula:

[0239]

[0240] Among them, normalized index index Mapping to interval , used for input The function makes The nearest sampling point is the one closest to the center. and The point is closest to the boundary of the sampling interval; disturbance control Will Interval linear scaling to , Ensure that all depth candidates are concentrated around the center value. Local region; final depth value equal Add a perturbation to apply all sampling offsets. , forming A set of depths that are centered on a non-uniformly distributed depth.

[0241] The sampling range is defined. =10, .

[0242] at this time If uniform sampling is followed, 8, 9, 10, 11, 12; but according to the present invention, At this point, the density is even higher around the middle 10.

[0243] The depth candidate plane set outputs a set of non-uniform depth hypothesis values ​​for each pixel location. ;

[0244] Based on the set of non-uniform depth assumptions As a depth dimension combined with feature maps According to the feature map Obtain the initial depth map The method of obtaining depth map That is, the cost body is constructed sequentially through variance aggregation, the cost body is regularized to obtain the probability body, and finally the expectation is calculated to obtain the depth map. .

[0245] like Figure 7 As shown, the joint refinement module jointly models the photometric consistency constraint and the edge structure constraint to form an objective function that describes the matching error between the current depth map and the real image:

[0246]

[0247] in, It is a balance coefficient used to adjust the weights of photometric data items and edge constraint items. This is a feature consistency constraint term. Its goal is to minimize the difference between the two images, thereby ensuring that the feature values ​​of the projected pixels match the reference image at the correct depth. Indicates pixel position, Represents pixels Eigenvalues ​​at; Indicates the use of depth Reference pixel Feature values ​​of pixels obtained after projection onto the source image For edge constraint terms, This represents the gradient of the depth map. Gradient at edge location compared to reference image Matching, edge weights , It is an edge map obtained from a reference image using the Canny algorithm. At the same time, the gradient error is amplified, making the optimization more sensitive at the edges. Controlling edge contribution; To measure the difference between depth transitions and image edge transitions;

[0248] Edge maps are obtained from reference images using the Canny algorithm. The specific steps are as follows:

[0249] Reference image and Convolution, to obtain a convolutional image, the formula is:

[0250]

[0251]

[0252] in, Indicates the Gaussian kernel in coordinates The value, The standard deviation determines the smoothness of the filter. Represents a convolutional image;

[0253] Through horizontal and vertical directions Operator convolution kernel calculates convolution image gradient components and The formula is:

[0254] ,

[0255] Using gradient components and Calculate the gradient magnitude:

[0256]

[0257] in, Gradient magnitude represents the intensity of a pixel's gradient, i.e., the absolute degree of brightness change. High magnitude values ​​typically correspond to image edges, reflecting abrupt changes in local brightness. and For pixels gradient components;

[0258] Using gradient components and Calculate the gradient direction:

[0259]

[0260] gradient direction Non-maximum suppression is performed on each pixel in four gradient directions: 0°, 45°, 90°, and 135°. ( At point ), compare neighboring pixels along the gradient direction, retaining only local maxima:

[0261]

[0262] in, , Let the points be the neighboring points on both sides of the gradient direction, where, express( ), express , or , express , or ;

[0263] The core of nonmaximum suppression lies in comparing pixel values ​​along the gradient direction to preserve local maxima. Since the gradient direction is continuous, it is usually discretized for ease of implementation. Therefore, nonmaximum suppression is performed only along four gradient directions: 0°, 45°, 90°, and 135°. The gradient direction of each pixel is replaced by one of these four directions based on their similarity.

[0264] Taking the 0° direction (horizontal) as an example, the current pixel position is If the gradient direction is 0°, then compare the gradient magnitudes of its left and right adjacent pixels:

[0265] like and If it is true, then keep it; otherwise, set it to zero.

[0266] A pixel is considered a valid edge only when its gradient is a local maximum; otherwise, it is set to 0 to achieve edge thinning.

[0267] gradient magnitude Normalization to get Based on the normalized results, we obtain ,and , The above are strong edges, which should be preserved directly; ~ A weak edge must be adjacent to a strong edge to be preserved.

[0268] Classify pixels:

[0269]

[0270] Where 1 represents a strong edge. 0 represents a weak edge, and 0 represents a non-edge.

[0271] Then the binary edge map That is, an edge map obtained from the reference image is:

[0272]

[0273] Based on the objective function, the Gauss-Newton method is used for iterative optimization to adjust the depth map. To minimize the objective function, an iterative solution to a nonlinear least squares problem is used to gradually refine the initial depth value. This involves calculating the error between the current depth and the true depth, and then adjusting the direction based on this error estimate. This process gradually optimizes the depth value of each point to ensure geometric consistency and accurate data matching across multiple views. The specific steps are as follows:

[0274] Construct the residual vector:

[0275]

[0276] in, , , by objective function Therefore, after concatenating the two residuals, the objective is to minimize them. , Represents the residual vector;

[0277] Linearization:

[0278] In the current depth estimation Minimize perturbation at the point Then, for the residual vector Perform a first-order Taylor expansion:

[0279]

[0280] Among them, the minimum disturbance It is the increment used in the Gauss-Newton method for iteratively updating variables; it represents the increment in the current depth map. Near the point of view, the minimum correction made to reduce the residual function. It is the Jacobian matrix, used to linearize the variation of residuals with depth. Each element Defined as the first The residuals for the first... The partial derivative of each depth pixel, , (·) indicates the first One residual, ( ) indicates using a depth map The calculated first One error, It is to find the partial derivative. ( ) indicates when depth When small changes occur, the amount of change in the residuals Representing a depth map The Middle The depth value of each pixel;

[0281] Solving the normal equation: Based on the residual vector After performing a first-order Taylor expansion, solving the linear system yields the minimum increment of the current depth, which corresponds to the depth supplement for each pixel. The formula is:

[0282]

[0283] in, for Transpose of;

[0284] Update and convergence:

[0285] After obtaining the minimum increment, the depth value is updated. Through this update, the depth map is... At each location, corrections are made based on local feature inconsistencies and edge gradient mismatches, using the following formula:

[0286]

[0287] If the iteration termination condition is met. To reconstruct the cascaded multi-view stereo, otherwise, use the currently updated depth map as the new initial value and execute the step that minimizes the objective function again.

[0288] This invention integrates attention mechanisms, dynamic depth sampling strategies, and joint edge optimization modules to form a cascaded 3D reconstruction network. It has advantages such as enhanced edge reconstruction, saving computational resources, and applicability to high resolution, making it particularly suitable for refined 3D reconstruction scenarios.

[0289] Dataset introduction:

[0290] To fully verify the performance of the proposed Cascaded Multi-View Reconstruction Network (ECA-MVSNet) on mechanical part reconstruction, the experiment used a combination of publicly available standard datasets and self-built mechanical part datasets for evaluation.

[0291] (1) DTU dataset

[0292] The DTU dataset is currently the most comprehensive dataset for multi-view applications. Figure 3 This dataset is a commonly used evaluation benchmark in the field of 3D reconstruction, containing images of 124 different objects from 49 viewpoints, with each viewpoint accompanied by 7 different lighting conditions. This dataset provides high-precision ground truth values ​​for structured light scanning as a reference. According to the design of the cascaded multi-view reconstruction network of this invention, during the training phase, the depth range (425mm-925mm) of this dataset is selected for non-uniform sampling to ensure that the model can learn depth perception capabilities for industrial-grade scenes.

[0293] (2) Self-built mechanical parts dataset

[0294] To address the unique characteristics of mechanical parts, such as their weakly textured surfaces and metallic reflective properties, this invention utilizes a self-developed experimental platform to collect image data of various typical parts. During the acquisition process, the part under test is placed at the center of an electric turntable, and three industrial cameras at different angles simultaneously capture images every 6° of turntable rotation, obtaining a total of 60 high-resolution images for each part. These images, after camera calibration and distortion correction, serve as test data to verify the generalization ability of the algorithm in a real industrial environment.

[0295] Evaluation indicators:

[0296] To quantitatively evaluate the geometric accuracy and completeness of the reconstructed point cloud, this invention adopts the standard evaluation metrics officially recommended by the DTU dataset, specifically including accuracy, completeness, and overall score.

[0297] (3) Accuracy )

[0298] Accuracy is defined as the average Euclidean distance between the reconstructed point cloud and the ground truth point cloud. This metric reflects how closely the reconstructed point cloud points approximate the actual surface of the object; a smaller value indicates higher accuracy. The calculation formula is as follows:

[0299]

[0300] In the formula, This is the set of reconstructed point clouds generated by the algorithm of this invention. This is a set of ground truth point clouds acquired by a high-precision scanner. To reconstruct point clouds Total number of points in To reconstruct a specific point in a point cloud, For a point in the truth cloud, Represents the point Find the nearest point in the ground truth point cloud and calculate the L2 norm (Euclidean distance) between them.

[0301] (4) Completeness )

[0302] Completeness is defined as the average Euclidean distance between the ground truth point cloud and the reconstructed point cloud. This metric measures the comprehensiveness of the reconstruction's coverage of the object's surface, reflecting the algorithm's ability to restore areas with weak textures and deep holes. Its calculation formula is as follows:

[0303]

[0304] in, This represents the number of points in the true point cloud set G.

[0305] (5) Overall Indicators

[0306] The comprehensive index, which is the arithmetic mean of accuracy and completeness, is the most critical indicator for measuring the overall performance of a 3D reconstruction system. Its calculation formula is as follows:

[0307]

[0308] The smaller the value, the better the coverage of the algorithm while ensuring accuracy.

[0309] (6) Chamfer Distance )

[0310] When assessing the overall distribution differences of point clouds, chamfer distance is commonly used to measure the similarity between two point cloud sets. It combines the bidirectional minimum sum of squared distances. The calculation formula is as follows:

[0311]

[0312] In the formula, and These are the cardinality of the two sets, respectively. This formula, by summing and averaging, is more sensitive to outliers in the reconstruction process and can effectively evaluate the fitting degree of the edges of mechanical parts.

[0313] (7) Depth map evaluation metrics (Mean Absolute Error) )

[0314] Because this invention uses a cascaded structure to generate multi-scale depth maps, the intermediate layer depth maps need to be evaluated during the training phase. The mean absolute error (MAE) is used to measure the difference between the predicted depth and the ground truth depth.

[0315]

[0316] In the formula, This represents the resolution of the depth map. and pixels The predicted depth value and the true depth at that location.

[0317] To ensure the efficiency and stability of large-scale 3D point cloud reconstruction experiments, all model training and testing in this study were performed on a high-performance computing server. Specific hardware configurations and software environments are as follows: Figure 11 As shown.

[0318] During the specific model training phase, ECA-MVSNet performs end-to-end parameter optimization on the DTU training set. The number of input views is set to... Image resolution downsampled to To balance computational load and accuracy, the depth sampling number of the cascaded structure within the model was set to 48, 32, and 8 in the three-level network. The Adam optimizer was used during training with an initial learning rate of 0.001, which was decayed to half its initial value at 10, 12, and 14 training epochs. The total training time was 16 epochs, and the loss function steadily decreased with increasing iterations. Figure 12 The results visually demonstrate the performance of the training and validation sets during the convergence process. It can be seen that the losses of the cascaded multi-view reconstruction network tend to stabilize after the 12th round, and the feature extraction module and the depth estimation module achieve a good match.

[0319] Experimental comparison and analysis:

[0320] To objectively evaluate the improvement in 3D reconstruction accuracy of ECA-MVSNet, this paper compares it with classic and cutting-edge algorithms such as MVSNet, CasMVSNet, and PatchmatchNet on the DTU test set. The evaluation criteria uniformly adopt recognized metrics in the field of 3D reconstruction, including... , and the arithmetic mean of the two. Result Figure 13 As shown.

[0321] Through analysis Figure 13 The data shows that the algorithm presented in this paper demonstrates significant advantages across all key metrics. The overall score of ECA-MVSNet (Cascaded Multi-View Reconstruction Network) reaches 0.305mm, an improvement of approximately 14% compared to the baseline CasMVSNet method with a cascaded structure. Particularly noteworthy is its accuracy, with a score of 0.288mm significantly outperforming other models. This confirms the effectiveness of the Feature Integration (FI) module in processing fine features of mechanical parts, generating point cloud distributions that more closely resemble the true geometry of the object.

Claims

1. A cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism, characterized in that, Includes the following steps: Step 1: Obtain a multi-view image set of the stereo to be tested and perform preprocessing; Step 2: Input the preprocessing results obtained in Step 1 into the cascaded multi-view reconstruction network with entropy guidance and attention mechanism to perform cascaded multi-view stereo reconstruction; The cascaded multi-view reconstruction network includes an improved FPN structure that extracts feature maps at three different scales from the preprocessing results obtained in step 1, a depth estimation module that estimates the depth map from coarse to fine using an entropy-guided sampling strategy on the feature maps obtained from the improved FPN structure, and a joint refinement module that optimizes the depth map obtained from the depth estimation module based on the feature maps obtained from the improved FPN structure.

2. The cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism as described in claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Obtain a multi-view image set of the stereo image to be tested, and select from it. Zhang Gaowei , width is The image includes one reference image, denoted as [image name missing]. ,and Zhang Yuan's image, denoted as ,in, Indicates the first Zhang Yuan's image; Step 1.2, using Indicates the first Zhang Yuan's image camera intrinsic parameters, rotation matrix, and translation vector. Represents reference image The camera's internal and external parameters.

3. The cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism as described in claim 2, characterized in that, The improved FPN structure in step 2 is to replace the 1×1 convolution in the original FPN structure containing 1×1 convolution with a feature selection module and add a feature alignment module. Specifically, this involves downsampling the source or reference image twice to obtain feature maps of high, medium, and low resolutions, and then performing convolution operations on each to obtain the corresponding feature maps. , and ,right , and The first feature selection module, the second feature selection module, and the third feature selection module perform feature selection respectively. That is, the feature map of the source image or reference image; The output of the third feature module is processed by a global pooling layer to obtain direction-sensitive attention weights, which are then weighted... The feature map is obtained above ; For feature maps Upsampling is performed to obtain the feature map Feature map output by the second feature selection module The second feature alignment module performs feature alignment, and the feature map output by the second feature alignment module is... Then with feature map Adding them together yields the feature map. ; For feature maps Upsampling is performed to obtain the feature map Feature map output by the first feature selection module The first feature alignment module performs feature alignment, and the feature map output by the first feature alignment module is shown. Then with feature map Adding them together yields the feature map. .

4. The cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism as described in claim 3, characterized in that, The first feature selection module, the second feature selection module, and the third feature selection module include a horizontal global average pooling layer and a vertical global average pooling layer that decompose the input features into horizontal position encoding and vertical position encoding along the horizontal and vertical directions. The formula for the horizontal global average pooling layer is: ; The formula for the vertical global average pooling layer is: ; in, These represent the width and height of the features output by the first feature selection module, the second feature selection module, and the third feature selection module, respectively. and The input tensors represent the feature coordinate positions along the horizontal and vertical directions, respectively. and This represents the pooling output along the horizontal and vertical directions; A 1×1 convolutional layer, a batch normalization layer, and a... are used to process the concatenated results of horizontal and vertical position encoding in sequence. The non-linear activation function layer has the following formula: in, Indicates Intermediate features obtained by processing with a nonlinear activation function layer Indicates a splicing operation; This indicates that a 1×1 convolutional layer scales the number of channels to 1. , Hyperparameters set to limit computational complexity For batch normalization layer, for Nonlinear activation function; A 1×1 convolutional layer that processes intermediate features sequentially along the horizontal and vertical directions. The activation function layer has the following formula: in, and From The horizontal and vertical components separated from the middle, and For two Attention weights obtained in the horizontal and vertical directions from the activation function layer, respectively; The feature tensor is obtained by multiplying the horizontal and vertical attention weights by the input features, as shown in the formula: in, This indicates the spatial location of the input features corresponding to the first feature selection module, the second feature selection module, and the third feature selection module. The eigenvalue at that location.

5. A cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism as described in claim 4, characterized in that: The specific implementations of the first feature alignment module and the second feature alignment module are as follows: For the corresponding feature maps of the input respectively Upsampling is performed and the resulting feature maps are concatenated along the channel dimension with the feature maps output by the corresponding first and second feature selection modules to generate a fused feature map. Then, convolution is used to predict the offset vector at each position. The formula is: in, It's a splicing operation. This represents the convolution operation. Indicates the feature map The feature map obtained by upsampling, Indicates the feature The result of feature selection, in the first feature alignment module, The value of is 1, in the second feature alignment module, The value of is 2; Based on offset For feature maps Dynamic sampling is performed to shift the sampling positions of high-level features towards the semantic key regions of low-level features, generating aligned features. The formula is: in, for Activation function It is a deformable convolution; Aligned features With features The feature maps are fused together to output the final feature map. The formula is: 。 6. A cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism as described in claim 5, characterized in that: The specific implementation of the depth estimation module is as follows: Based on feature maps Obtain the initial depth map The specific process is as follows: Will Feature map of Zhang Yuan's image Feature maps projected onto the reference view using the differentiable homography transformation formula The plane, the formula is: Formula for differentiable homography transformation: in, and They represent the first Feature maps obtained from Zhang Yuan's image and the reference image using the improved FPN structure Is the depth as The Differentiable homography transformation factor of each source image. Indicates the first Camera intrinsic parameters, rotation matrix, and translation vector for each source image. Represents reference image Camera internal and external parameters, A unit vector refers to the principal axis of the reference camera; Indicates the first All pixels in Zhang Yuan's image, Indicates will Convert to the corresponding pixel in the reference view; A depth layer is defined in the projection space, which is discretized onto the depth layer plane, and feature maps of each source image are defined. Pixels after projection Each feature map will be matched on some plane within it. Pixels after projection They all have depth and dimension and obtain the feature body ,in, Indicates the number of channels. and These represent height and width, respectively. The obtained first Features of Zhang Yuan's Image Constructing the cost body using variance aggregation The formula is: in, This indicates the total number of input reference images and source images. represent The average value of each feature body; The cost volume is encoded and decoded. In the encoding stage, the cost volume is downsampled layer by layer through 3D convolution and pooling operations to expand the receptive field and capture global depth and spatial context information. In the decoding stage, coarse global information and local details are combined through upsampling and skip connections. The decoded result is regularized, and the regularized cost volume is then applied in the depth dimension. The probability distribution is obtained by finding the probability volume. ; For probability bodies The initial depth map is obtained by calculating the expectation along the depth direction. The size is Initial depth map pixels for: in, Representing the The actual depth value corresponding to each depth assumption. Represents the initial depth map pixels In the The probability of a depth hypothesis, ; Based on feature maps and initial depth map Obtain depth map The specific process is as follows: Calculate the probability body The entropy, the formula is: in, Initial depth map pixels entropy, Initial depth map pixels In the The probability of a depth hypothesis; based on Using linear interpolation, each initial depth map is dynamically adjusted. Pixels Depth hypothesis number The formula is: in, To ensure rounding to the nearest integer It is an integer. The maximum depth hypothesis number, For the minimum depth hypothesis number, and These are the maximum entropy value and the minimum entropy value, respectively. based on Define the local depth range using the following formula: in, It controls the intensity of the influence of entropy on the range. It is the basic depth range. and These are prior parameters, which are updated as the network is trained. Based on deep hypothesis number and local depth range The formula for calculating the dynamic depth interval is: Based on dynamic depth interval and Based on the initial depth map Depth estimation in Nearby generation The assumption of equal depth intervals is given by the following formula: Will As a depth dimension, and based on and feature map According to the feature map Obtain the initial depth map The method of obtaining depth map ; Based on feature maps and depth map Obtain depth map The specific process is as follows: Based on depth map The entropy of the probability volume of each pixel Normalize it to the [0,1] interval, the formula is: in, , These represent the maximum and minimum entropy values, respectively. Represents pixels Normalized entropy value; Based on the normalized entropy value, a different number of depth hypotheses are dynamically assigned to each pixel. The formula is: in, To ensure rounding to the nearest integer It is an integer. and These are the maximum and minimum depth hypothesis numbers, respectively; To estimate Methods for estimating depth values Centered on the depth map The depth entropy of each pixel and the prior parameters obtained during network training define a pixel. Local depth search range: in, Indicates the base depth interval. It is a scaling factor that controls the width of the search range; Based on pixels To determine the local depth search range, a non-uniform sampling strategy is used to generate a set of depth candidate planes, as shown in the formula: Among them, normalized index index Mapping to interval , used for input The function makes The nearest sampling point is the one closest to the center. and The point is closest to the boundary of the sampling interval; disturbance control Will Interval linear scaling to , Ensure that all depth candidates are concentrated around the center value. Local area; The depth candidate plane set outputs a set of non-uniform depth hypothesis values ​​for each pixel location. ; Based on the set of non-uniform depth assumptions As a depth dimension combined with feature maps According to the feature map Obtain the initial depth map The method of obtaining depth map .

7. A cascaded multi-view stereo reconstruction method based on entropy guidance and attention mechanism as described in claim 6, characterized in that: The joint refinement module jointly models the photometric consistency constraint and the edge structure constraint to form an objective function that describes the matching error between the current depth map and the real image: in, It is a balance coefficient used to adjust the weights of photometric data items and edge constraint items. For feature consistency constraints, Indicates pixel position, Represents pixels Eigenvalues ​​at; Indicates the use of depth Reference pixel Feature values ​​of pixels obtained after projection onto the source image For edge constraint terms, This represents the gradient of the depth map. Gradient at edge location compared to reference image Matching, edge weights , It is an edge map obtained from a reference image using the Canny algorithm. At the same time, the gradient error is amplified, making the optimization more sensitive at the edges. Controlling edge contribution; To measure the difference between depth transitions and image edge transitions; Edge maps are obtained from reference images using the Canny algorithm. The specific steps are as follows: Reference image and Convolution, to obtain a convolutional image, the formula is: in, Indicates the Gaussian kernel in coordinates The value, The standard deviation determines the smoothness of the filter. Represents a convolutional image; Through horizontal and vertical directions Operator convolution kernel calculates convolution image gradient components and The formula is: ， Using gradient components and Calculate the gradient magnitude: in, Gradient magnitude represents the intensity of a pixel's gradient, i.e., the absolute degree of brightness change. High magnitude values ​​typically correspond to image edges, reflecting abrupt changes in local brightness. and For pixels gradient components; Using gradient components and Calculate the gradient direction: gradient direction Non-maximum suppression is performed on each pixel in four gradient directions: 0°, 45°, 90°, and 135°. ( At point ), compare neighboring pixels along the gradient direction, retaining only local maxima: in, , Let the points be the neighboring points on both sides of the gradient direction, where, express( ), express , or , express , or ; gradient magnitude Normalization to get Based on the normalized results, we obtain ,and , The above are strong edges, which should be preserved directly; ~ A weak edge must be adjacent to a strong edge to be preserved. Classify pixels: Where 1 represents a strong edge. 0 represents a weak edge, and 0 represents a non-edge. Then the binary edge map That is, an edge map obtained from the reference image is: Based on the objective function, the Gauss-Newton method is used for iterative optimization to adjust the depth map. This minimizes the objective function. The specific steps are as follows: Construct the residual vector: in, , , by objective function Therefore, after concatenating the two residuals, the objective is to minimize them. , Represents the residual vector; Linearization: In the current depth estimation Minimize perturbation at the point Then, for the residual vector Perform a first-order Taylor expansion: Among them, the minimum disturbance It is the increment used in the Gauss-Newton method for iteratively updating variables; it represents the increment in the current depth map. Near the point of view, the minimum correction made to reduce the residual function. It is the Jacobian matrix, used to linearize the variation of residuals with depth. Each element Defined as the first The residuals for the first... The partial derivative of each depth pixel, , (·) indicates the first One residual, ( ) indicates using a depth map The calculated first One error, It is to find the partial derivative. ( ) indicates when depth When small changes occur, the amount of change in the residuals Representing a depth map The Middle The depth value of each pixel; Solving the normal equation: Based on the residual vector After performing a first-order Taylor expansion, solving the linear system yields the minimum increment of the current depth, which corresponds to the depth supplement for each pixel. The formula is: in, for Transpose of; Update and convergence: After obtaining the minimum increment, the depth value is updated. Through this update, the depth map is... At each location, corrections are made based on local feature inconsistencies and edge gradient mismatches, using the following formula: If the iteration termination condition is met. To reconstruct the cascaded multi-view stereo, otherwise, use the currently updated depth map as the new initial value and execute the step that minimizes the objective function again.

8. A cascaded multi-view stereo reconstruction system based on entropy guidance and attention mechanism, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-7.

Images