An attention-guided multi-view depth estimation method

Abstract

The present invention relates to an attention-guided multi-view depth estimation method, and belongs to the field of computer vision technology. The method comprises: extracting multi-scale features from an input image to obtain a pyramid feature map; constructing an attention-based global feature weight based on consistency information and contextual global clues; applying the global feature weight to the feature volume of all feature maps to obtain a cost volume; generating an attention weight by extracting correlation geometric information between the source feature map and the reference feature map after homography transformation; filtering the cost volume using the attention weight to obtain an attention cost volume; and using a deep regression algorithm to infer a depth map and a corresponding confidence map after preliminary processing of the attention cost volume. The method can improve the accuracy, robustness and efficiency of matching, and bring more satisfactory results to three-dimensional reconstruction tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to an attention-guided multi-view depth estimation method. Background Art

[0002] The goal of multi-view stereo (MVS) is to reconstruct the observed 3D scene from multi-view images and corresponding camera parameters in a dense representation. This method has been widely studied for decades and covers a wide range of applications, such as image entertainment, smart home, cultural relic reconstruction, AR tourism, autonomous driving, robot navigation, etc. It has gradually become a core issue in photogrammetry and computer vision tasks. Traditional MVS methods usually follow the sparse point cloud generation process to achieve it. In order to reconstruct a dense 3D point cloud, it is necessary to obtain the camera intrinsic and extrinsic parameters and sparse points restored for each image. For example, clustered multi-view stereo (CMVS) and patch-based multi-view stereo (PMVS) are very popular dense 3D reconstruction methods. CMVS uses SfM filters to merge the extracted feature points, decompose the input image into a set of image clusters of controllable size, and then uses MVS software for 3D reconstruction. PMVS takes the clustered images of CMVS as input and generates dense 3D point clouds through matching, expansion and filtering. Semi-global matching (SGM) is also a popular 3D reconstruction method, which is proposed to estimate dense disparity maps from rectified stereo image pairs and introduce inconsistency penalties. Since the SGM algorithm makes a trade-off between computational time and result quality, it is faster than PMVS and has been widely adopted in real-time stereo vision applications. Although these methods perform well in ideal Lambert scenes in high-texture areas, there are problems of incomplete reconstruction in some low-texture and reflective areas, and the accuracy and robustness of dense matching are reduced. At the same time, traditional methods are usually performed sequentially, usually requiring steps such as camera calibration, camera motion estimation, and dense image matching to achieve, which is a time-consuming and memory-intensive process, limiting the application of scenes with high efficiency requirements. Therefore, traditional MVS methods still need to be improved to obtain more robust and complete reconstruction results.

[0003] The multi-view stereo MVSNet method and the Cas_MVSnet cascade network and method are important research directions in the field of computer vision in recent years. They aim to obtain the three-dimensional structure information of the scene by shooting the same scene with multiple cameras. These methods have broad application prospects in robot vision, virtual reality, augmented reality, autonomous driving and other fields.

[0004] MVSNet is a multi-view stereo vision method based on deep learning. It learns the depth information of the scene by training a deep neural network. The MVSNet method uses a convolutional neural network (CNN) as the basic architecture, uses multi-view images as input, and achieves depth estimation of the scene through end-to-end training. The advantage of the MVSNet method is that it can automatically learn the depth information of the scene without manual intervention. In addition, the MVSNet method also uses multi-view images as input, so that richer scene information can be obtained. However, the MVSNet method also has some problems. First, due to the complexity of deep neural networks, training MVSNet requires a lot of computing resources and time. Secondly, the MVSNet method requires a lot of training data to obtain accurate depth estimation results. In addition, the cost body in the existing MVSNet method has the problem of high information redundancy and insufficient correct matching information, which affects the correctness of depth estimation.

[0005] Cas_MVSnet is a multi-view stereo vision method based on cascade and attention mechanism. It realizes multi-level depth prediction of scenes by cascading multiple depth estimation networks together. The method also introduces the attention mechanism to adaptively extract and match features of images taken by different cameras. The advantage of the Cas_MVSnet method is that it can make full use of the depth information already obtained in the low-level network and reduce the learning difficulty of the high-level network. At the same time, the method also improves the accuracy and stability of depth prediction by fusing multiple depth estimation results together. In addition, the method also introduces the attention mechanism to adaptively extract and match features of images taken by different cameras, further improving the accuracy of depth prediction. However, the Cas_MVSnet method also has some problems. First, due to the structural characteristics of the cascade network, it requires a lot of computing resources and time during training. Second, in the cascade network, the depth estimation results of the low-level network often have errors, which will be transmitted to the high-level network, affecting the accuracy of depth prediction. In addition, the existing Cas_MVSnet method often only focuses on improving the accuracy of depth prediction, while ignoring the impact of global context consistency information on depth estimation.

[0006] In general, in current technologies, the overall contextual information of the scene is often ignored in the process, resulting in certain matching problems in matching confidence. At the same time, the coarse-to-fine strategy inevitably involves cumulative errors, which are difficult to compensate in the later stage. At the same time, redundant and irrelevant information will appear in the cost body of deep learning methods, resulting in insufficient correct matching information, thus affecting the correctness of depth estimation. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide an improved multi-view stereo MVSNet method and Cas_MVSnet cascade network method, that is, an attention-guided multi-view depth estimation method to solve the shortcomings and problems existing in the prior art. The method improves the accuracy of depth prediction by introducing an attention mechanism and a cost volume filtering mechanism, thereby improving the performance of the overall method.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] An attention-guided multi-view depth estimation method, the method comprising the following steps:

[0010] S1, extracting multi-scale features from the input image to obtain a pyramid feature map;

[0011] S2, combining consistency information and contextual global cues to construct attention-based global feature weights;

[0012] S3, applying the global feature weight to the feature volumes of all feature maps to obtain the cost volume;

[0013] S4, generating attention weights by extracting correlation geometric information between the source feature map and the reference feature map after homography transformation;

[0014] S5. Filter the cost body using the attention weight to obtain the attention cost body;

[0015] S6. After preliminary processing of the attention cost body, a deep regression algorithm is used to infer the depth map and the corresponding confidence map.

[0016] Further, in step S1, referring to the feature pyramid network, feature maps of multi-scale spatial resolution are used to construct cost volumes of different resolutions; for each input image i∈{0,1,…,N}, where I0 is the reference image; an L-level image pyramid is constructed starting from the reference image, where the bottom layer of the image pyramid corresponds to the input image Then, a convolutional neural network is used to extract features to obtain the feature maps of each layer, where the number of channels of layers conv0, conv1, and conv2 is increased; represents the feature map extracted by the Lth layer, where f l i represents the i-th feature map in the l-th layer, R represents the set of feature maps, H represents height, W represents width, and Ch represents the number of output channels.

[0017] Further, in step S2, the following steps are specifically included:

[0018] S21, splicing the feature maps of the reference image and the source image in the same batch;

[0019] S22, apply global average pooling to reduce all feature maps to their respective channel dimensions;

[0020] S23, calculating the weight of the global feature according to the context channel information of each scene;

[0021] The specific calculation process is as follows:

[0022]

[0023] w=Sigmoid(L1(ReLU(L2(GAP(F),s1)),s2) (2)

[0024] V i =v i *w (3)

[0025] In formula (1), f i represents the reference image, f0 represents the original image, concat() is the concatenation function, and F represents the concatenated feature map; in formula (2), L1(·,·) and L2(·,·) are two linear transformations, where s1 and s2 represent the corresponding transformation parameters respectively; GAP() represents the average pooling of the weight value of the feature map F; ReLU() is the activation layer for fitting; Sigmoid() represents the normalization function, which normalizes the output to the range of 0-1; in formula (3), v i represents the feature volume obtained by a differentiable homography transformation that transforms all features at assumed depth d to the frontal-parallel plane of the reference view.

[0026] Further, in step S3, by calculating the global feature weight w i , improve the matching graph f i , where ⊕ is the channel multiplication and M is the mapping defined by the cost metric:

[0027] M:R V ×…×R V ——→R V that:

[0028]

[0029] in is the average of all confidence maps matched by attention enhancement. All the above operations are performed pixel by pixel. C represents the cost volume, which is constructed by calculating the mean variance between the reference and source features.

[0030] Further, in step S4, the group-wise method of GWCNet is used to group the features, calculate the relevant graph information group by group, and divide the T channels into G groups on average, so that feature graphs of different levels will not interfere with each other;

[0031] First, the matching similarity is calculated; among them, F0(p), F i (p i,j ) are the reference feature map and the source feature map respectively, then the feature similarity S of the Gth group i (p,j) is:

[0032]

[0033] Where <·,·> is the similarity inner product between the reference feature map and the source feature map, and S i (p,j)∈R G represents the similarity vector within different groups, g represents the group number of G groups into which the T channel is evenly divided, and a pixel-by-pixel view weight {W is estimated for each source view i (p)}, based on the initial depth hypothesis space obtained in the first iteration, the visible information of all source views is integrated; using W i (p) represents the source view I i The visibility of pixel P in the similarity matrix S is calculated by using a 3D convolution containing a 1*1*1 convolution kernel and a Sigmoid activation layer. i Processing, get a weight P between 0-1 i , for source view I i The pixel point p, its weight value is expressed as:

[0034] w i (p) = max{P i (p,j)|j=0,1,…,D-1} (6)

[0035] Where P i Represents the visible probability of point p at depth j, weight w i is regarded as the global attention weight of the i-th view relative to the reference view, and D is the predicted depth.

[0036] Further, in step S5, using the weight w i Pair similarity S i Weighted to increase its robustness, for all source views of the pixel point P of the j-th depth layer, its group similarity expression is as follows:

[0037]

[0038] Using two 3D convolutions and a 3D hourglass network Regularization is performed, and the convolution layer is used to compress the depth channel to 1, and then the softmax is used to process the depth dimension into a probability value between 0 and 1, so as to obtain the attention weight A∈R C×1×H / 4×W / 4 ; Use the attention weight A to eliminate the redundant information in the initial cost body, and the attention cost body C on channel i A Calculated as:

[0039] C A =A⊙C (8)

[0040] ⊙ represents the pixel-by-pixel product. By filtering the initial cost body C using the attention weight A, a more accurate attention cost body C is obtained. A .

[0041] Further, in step S6, the attention cost body C A Regularization is performed by encoding and then decoding it using a 3D convolutional layer to filter out the noise pollution and object occlusion errors of the original cost volume C; finally, a deep regression algorithm is used to infer the depth map and the corresponding confidence map; the depth range [d min ,d max The probability P(d) of each sampled depth d in ] and the predicted depth D at each labeled pixel are calculated as:

[0042]

[0043] where d max , d min are the maximum depth and minimum depth of the depth estimation range respectively;

[0044] For the intermediate outputs generated by the cost body of N layers and the intermediate outputs of the group-related attention cost body, the total loss function for supervising all outputs is:

[0045]

[0046] Among them, use Smooth L1 To calculate the loss; d att is the group-related attention weight to obtain the depth map, d i is the depth map obtained by depth regression of the multi-scale cost volume; att is the coefficient of the predicted data, λ i is the corresponding loss weight.

[0047] The beneficial effects of the present invention are:

[0048] Compared with the prior art, the adaptive feature weighting method of the present invention fully utilizes the overall context information of the scene by introducing a spatial attention mechanism. This makes the matching process more globally aware and improves the matching accuracy, especially when dealing with large-scale scenes and complex structures.

[0049] Since the present invention adopts an adaptive feature weight method, the cumulative error introduced by the coarse-to-fine strategy is reduced. By more effectively fusing feature information at each level, the method of the present invention can maintain matching consistency at different scales and improve the robustness of the overall matching.

[0050] The method based on the group-related attention mechanism can effectively filter out redundant information and mismatched information in the cost body, thereby improving the accuracy and completeness of the matching. This means that the present invention can restore the three-dimensional scene more accurately and provide more reliable data support for application fields such as computer vision and virtual reality.

[0051] The adaptive feature weight method of the present invention has adaptive learning capability and can automatically adjust the feature weights according to different scenarios and tasks. This flexibility makes the method of the present invention more versatile and can adapt to various stereo matching requirements.

[0052] The method based on the group-related attention mechanism introduces relevant weights for different groups in the cost body, which helps to alleviate the problem of redundant information and mismatched information in the cost body. This technical means enables the present invention to capture the geometric structure of the scene more accurately and improves the quality of the reconstruction results.

[0053] By filtering the redundant information in the cost volume, the present invention not only improves the matching accuracy, but also improves the efficiency of overall matching and reconstruction, which is of great significance for processing large-scale data sets and real-time application scenarios.

[0054] The method of the present invention increases the robustness of the algorithm to interferences such as noise and occlusion by introducing spatial attention mechanism and group-related attention mechanism. This makes the algorithm of the present invention more reliable in practical applications and can handle complex and dynamic scenes.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 A flowchart of the attention-guided multi-view depth estimation method of the present invention;

[0058] Figure 2 A schematic diagram of a process of enhancing structural features of global features according to attention in the present invention;

[0059] Figure 3 It is a schematic diagram of qualitative comparison between the CMVD-AANET of the present invention and other existing methods in the DTU validation set scan1;

[0060] Figure 4 It is a schematic diagram of qualitative comparison between the CMVD-AANET of the present invention and other existing methods in scan10 of the DTU validation set;

[0061] Figure 5 This is an example diagram of the result of point cloud reconstruction performed by CMVD-AANET of the present invention on the middle set of Tanks and Temples data set;

[0062] Figure 6 1. It is an example diagram of the point cloud fusion results of the CMVD-AANET of the present invention in four scenes of the BlendedMVS data set;

[0063] Figure 7 This is an example diagram comparing the point cloud reconstruction results of a large outdoor scene in the BlendedMVS dataset using the CMVD-AANet of the present invention and the existing MVSNet model. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0065] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0066] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0067] See also Figure 1 to Figure 7 , which is an attention-guided multi-view depth estimation method.

[0068] Example 1

[0069] Under the current MVS multi-view stereo technology, there is a common problem in the multi-view stereo matching process, that is, the overall context information of the scene is often easily ignored, resulting in problems with the matching confidence. At the same time, the coarse-to-fine strategy will inevitably introduce cumulative errors in the processing process, which is difficult to be effectively compensated in the subsequent stages. In addition, in deep learning methods, the cost body often contains redundant information and information irrelevant to the task, which further affects the accuracy of matching and reconstruction.

[0070] In order to solve the above problems, the present invention proposes an innovative adaptive feature weighting method, which can effectively enhance the accuracy of matching by introducing a spatial attention mechanism, thereby compensating for the problem of insufficient matching reliability caused by ignoring overall context information. The core idea of ​​this method is to better capture the global information of the scene through sensitivity to spatial relationships, thereby improving the accuracy and robustness of matching.

[0071] In addition, the present invention also proposes a method based on group-related attention mechanism, which aims to filter redundant information and mismatched information in the cost body by introducing group-related weights. The key to this method is to improve the matching accuracy and completeness by fine-grained screening of the cost body, thereby obtaining more accurate reconstruction results.

[0072] In the adaptive feature weighting method, the present invention adopts a spatial attention mechanism, which dynamically weights the input feature map based on the principle of deep learning, so that the network pays more attention to important areas. By introducing spatial attention, the present invention can better utilize the global structural information of the scene in the matching process, thereby improving the accuracy of matching. In addition, the method of the present invention can adaptively learn the weights of features, so that the network can be more adaptable and generalized when processing different scenes.

[0073] In the method based on the group-related attention mechanism, the present invention divides the cost body into different groups and introduces relevant weights for each group. In this way, the present invention can eliminate irrelevant information in the cost body and reduce redundant calculations, thereby improving the efficiency of matching and reconstruction. This method performs well in processing complex scenes and can better adapt to the characteristics of different scenes and objects, thereby obtaining more accurate three-dimensional reconstruction results.

[0074] Specifically, the attention-guided multi-view depth estimation method of the present invention comprises the following steps:

[0075] S1, extracting multi-scale features from the input image to obtain a pyramid feature map;

[0076] S2, combining consistency information and contextual global cues to construct attention-based global feature weights;

[0077] S3, applying the global feature weight to the feature volumes of all feature maps to obtain the cost volume;

[0078] S4, generating attention weights by extracting correlation geometric information between the source feature map and the reference feature map after homography transformation;

[0079] S5. Filter the cost body using the attention weight to obtain the attention cost body;

[0080] S6. After preliminary processing of the attention cost body, a deep regression algorithm is used to infer the depth map and the corresponding confidence map.

[0081] In this embodiment, the attention-guided multi-view depth estimation method of the present invention is implemented based on the constructed CMVD-AANet network architecture, such as Figure 1As shown, the design of the CMVD-AANet network architecture fully draws on the multi-scale feature extraction method of the Cas-MVSnet network. In the CMVD-AANet network architecture, the multi-view images are first downsampled to form an image pyramid, and then a feature extraction pyramid with weight sharing is constructed to extract and generate feature maps from coarse to fine. The feature map is homographically transformed at the assumed depth to construct a cost body, and the cost body is regularized into an initial depth map DL using 3D convolution and softmax operations. The initial depth map DL is upsampled to the image size of the next layer (L-1), and at the same time, the depth assumption plane is estimated and the cost body is regularized, and then the depth map DL-1 is estimated using softmax operation, and it is upsampled to the L-2 layer image size, and then deep reasoning is performed. Therefore, an iterative depth map estimation process can be formed. Specifically, the present invention uses the FPN feature pyramid to extract features and global feature weights based on attention-enhanced global feature matching. Then, while the cost body generated by the feature map is being used, the group-related attention mechanism is used to filter out useless redundant information in the cost body. Finally, the regularized cost body is subjected to deep regression to generate a depth map and a corresponding confidence map. More specifically, the proposed CMVD-AANet architecture extracts multi-scale features from the input image through the FPN module (feature pyramid). To enhance the credibility of feature matching, AWF (Attention Weight Filtering) constructs attention-based global feature weights in combination with consistency information and contextual global cues. In AGF (Attention Augmented Global Features), associated geometric information is extracted from the source feature map and the reference feature map, and attention weights are generated to filter out irrelevant information and reduce redundancy in the cost volume. The proposed method emphasizes effective information to obtain more accurate high-quality dense depth maps.

[0082] Further, in step S1, the feature extraction pipeline of the present invention refers to the feature pyramid network and uses feature maps of multi-scale spatial resolution to construct cost volumes of different resolutions. First, for each input image i∈{0,1,…,N}, where I0 is a reference image. The present invention starts to construct an L-level image pyramid from the reference image, where the bottom layer of the image pyramid corresponds to the input image Then the present invention uses a convolutional neural network (CNN) to extract features to obtain feature maps at each level, wherein the present invention increases the number of channels of layers conv0, conv1, and conv2 from 8 to 32. Since the extraction module is implemented on images of different perspectives and scales, the learning weights of the module need to be shared. As shown in the figure below, the present invention uses represents the feature map extracted by the Lth layer, where f l irepresents the i-th feature map in the l-th layer, R represents the set of feature maps, H represents height, W represents width, and Ch represents the number of output channels. The architecture of the FPN network for multi-scale feature extraction is shown in Table 1:

[0083] Table 1

[0084]

[0085] Furthermore, in step S2, in current multi-view stereo reconstruction based on deep learning methods, feature extraction mainly relies on local perceptual features at the pixel level to construct matching costs, which results in ignoring overall contextual information during the entire extraction process. The present invention proposes an attention mechanism to enhance the confidence of feature matching by combining consistency information with global contextual clues from feature maps of a reference image and its corresponding source image. Specifically, Figure 2 As shown, the specific steps include:

[0086] S21, splicing the feature maps of the reference image and the source image in the same batch;

[0087] S22, apply global average pooling to reduce all feature maps to their respective channel dimensions;

[0088] S23, calculating the weight of the global feature according to the context channel information of each scene;

[0089] The specific calculation process is as follows:

[0090]

[0091] w=Sigmoid(L1(ReLU(L2(GAP(F),s1)),s2) (2)

[0092] V i =v i *w (3)

[0093] Among them, v i represents the feature volume obtained by the differentiable homography transformation, which transforms all features with a hypothetical depth of d to the frontal parallel plane of the reference view, L1(·,·) and L2(·,·) are two linear transformations, where s1 and s2 represent the corresponding transformation parameters. In formula (2), in order to enhance robustness, the F weight value is average pooled to reduce the negative impact of smooth noise and local changes. Secondly, the ReLU activation layer is used to better fit complex nonlinear data and improve the expressiveness and pan-capability of the network; then, the sigmoid function is used to normalize the output to the range of 0-1, which makes it possible to compare the output data between different neurons or layers.

[0094] Furthermore, in step S3, by calculating the global feature weight w i , improve the matching graph f i Where ⊕ is the channel multiplication, and M is the mapping defined by the cost metric of the present invention:

[0095] M:R V ×…×R V ——→R V that:

[0096]

[0097] in is the average of all confidence maps matched by attention enhancement. All the above operations are performed pixel by pixel. C represents the cost volume, which is constructed by calculating the mean variance between the reference and source features.

[0098] Further, in step S4, the purpose of filtering the cost body with attention weights is to emphasize valid information, suppress irrelevant information and reduce redundant information in the cost body. Therefore, the present invention is prepared to generate attention weights by extracting correlation geometric information between the source feature map and the reference feature map after the homography transformation. The present invention will adopt the group-wise method of GWCNet to group the features and calculate the correlation map information group by group. The present invention divides T (T = 128) channels into G groups (G = 16) on average, and feature maps of different levels will not interfere with each other.

[0099] First, the matching similarity is calculated; among them, F0(p), F i (p i,j )F0(p),Fi(pij) are the reference feature map and the source feature map respectively, then the feature similarity S of the G group is i (p,j) is:

[0100]

[0101] Where <·,·> is the similarity inner product between the reference feature map and the source feature map. At the same time, the present invention uses S i (p,j)∈R G represents the similarity vectors within different groups, g represents the group number of the G group into which the T channel is evenly divided; for each source view, the present invention estimates a pixel-by-pixel view weight {W i (p)}, based on the initial depth hypothesis space obtained in the first iteration, the visible information of all source views is integrated to make the similarity vector more robust. i (p) represents the source view I iThe weight is calculated only once and used in subsequent iterations. The present invention uses a 3D convolution including a 1*1*1 convolution kernel and a sigmoid activation layer to perform similarity matrix S i Processing, get a weight P between 0-1 i , for source view I i The weight value of the pixel p is expressed as follows:

[0102] w i (p) = max{P i (p,j)|j=0,1,…,D-1} (6)

[0103] Where P i Represents the visible probability of point p at depth j. At the same time, the weight w here can be i is regarded as the global attention weight of the ith view relative to the reference view.

[0104] Further, in step S5, we have the pixel visual weight w of each source view. i Similarity with group S i , we can use the weight w i Pair similarity S i Weighting is performed to increase its robustness. For the pixel point P of the jth depth layer of all source views, its group similarity expression is as follows:

[0105]

[0106] Finally, two 3D convolutions and a 3D hourglass network are used to Regularization is performed, and a simple convolution layer is used to compress the depth channel to 1, and then softmax is used to process the depth dimension into a probability value between 0 and 1 to obtain the attention weight A∈R C×1×H / 4×W / 4 The present invention uses the ground truth depth to supervise the attention weight A to obtain accurate weights. After obtaining the attention weight A, the present invention uses it to eliminate redundant information in the initial cost body, thereby enhancing its representation ability. The attention cost body C on channel i A Calculated as:

[0107] C A =A⊙C (8)

[0108] ⊙ represents the pixel-by-pixel product. By filtering the initial cost body C using the attention weight A, a more accurate attention cost body C is obtained. A .

[0109] Further, in step S6, after obtaining the filtered attention cost body CA After that, firstly, the attention cost body C A Regularization is performed, and the original cost volume C is encoded and then decoded by using a 3D convolutional layer to filter out possible noise pollution and object occlusion errors. Finally, the Yao Zhong depth regression algorithm is used to infer the depth map. The probability P(d) of each sampled depth d in the depth range [dmin, dmax] is calculated by using Softmax. The predicted depth D at each marked pixel is calculated as:

[0110]

[0111] where d max , d min are the maximum depth and minimum depth of the depth estimation range respectively.

[0112] For the intermediate outputs generated by the cost body of N layers and the intermediate outputs of the group-related attention cost body, the present invention supervises all outputs. The total loss function is:

[0113]

[0114] Among them, the present invention uses To calculate the loss; d att is the group-related attention weight to obtain the depth map, d i is the depth map obtained by depth regression of the multi-scale cost volume; att is the coefficient of the predicted data, λ i is the corresponding loss weight.

[0115] Example 2

[0116] This example introduces the data sets used for network training and testing, and studies the experimental results.

[0117] The DTU dataset is an indoor scene dataset for multi-view reconstruction, which contains images, camera parameters and 3D models of multiple scenes. The dataset was created by the Computer Vision and Graphics Research Group at the Technical University of Denmark (DTU) and is widely used in the field of computer vision.

[0118] The DTU dataset contains multi-view images of 124 different scenes, each scene contains 40-60 images and corresponding camera parameters. All images are high-resolution, containing 2000x1500 to 3072x2048 pixels. The camera of each scene is fixed with different positions and orientations to simulate images captured from different perspectives.

[0119] In addition to images and camera parameters, the DTU dataset also provides 3D models for evaluating the performance of multi-view reconstruction algorithms. Each scene has a complete 3D model, which is a fusion of reconstruction results from multiple perspectives. These 3D models contain accurate geometry and texture information.

[0120] Using the DTU dataset for multi-view reconstruction research allows algorithm developers to evaluate the performance of their algorithms in real-world scenarios.

[0121] Multi-view reconstruction refers to the reconstruction of a 3D scene using images from multiple viewpoints. Tanks and Temple is a classic multi-view reconstruction dataset, which was created by researchers at Princeton University to evaluate the performance of multi-view reconstruction algorithms. The dataset consists of two parts: one is the tank dataset and the other is the temple dataset. Each dataset contains image sequences taken from multiple viewpoints, and each sequence rotates around a 3D object (tank or temple). These sequences are taken from a rotating platform with a camera and an object on it.

[0122] The Tanks dataset contains 57 sequences, each of which contains 41 640x480 JPEG images. The Temple dataset contains 105 sequences, each of which contains 30 640x480 JPEG images. The Tanks and Temple datasets are widely used datasets to evaluate the performance of multi-view reconstruction algorithms.

[0123] This embodiment will use image pyramid and ground truth for training. The network of this embodiment is first trained on the DTU dataset training set to obtain training weights, and then tested on the DTU dataset test set. At the same time, the Tanks and Temple dataset is tested using the weights trained on the DTU dataset to evaluate the generalization of the multi-view reconstruction algorithm.

[0124] The depth map is fused. Before the depth map is fused to generate a point cloud, the noise and erroneous estimates of the background and occluded areas need to be filtered out. The depth fusion filter mainly includes the following two points: Photometric consistency robust filtering filters outliers. This embodiment improves the quality of depth estimation by screening probability values. For example, pixels with a depth probability less than 0.8 are used as outliers, thereby improving the accuracy of depth estimation. Geometric consistency robust filtering can be used to evaluate the quality of depth consistency between multiple views. By projecting the depth d1 of the pixel point in the reference image to the original image and then reprojecting it back to the reference image, the coordinates p of the reprojected pixel point are i and the reprojected depth d i satisfy:

[0125] ||p reproj-p1||2<τ1 (11)

[0126] ||d reproj -d1||2<τ2 (12)

[0127] It is assumed that the depth estimate d1 of pixel p1 is consistent in these two views.

[0128] For the evaluation of the reconstruction results of the DTU dataset, this embodiment uses average accuracy (acc) to summarize the quality of the reconstructed MVS points, average completeness (comp) to summarize how much surface the MVS reconstruction captures, and overall accuracy (overall) to measure the reconstruction results. Overall accuracy OA:

[0129]

[0130] Acc and Comp can be calculated by using the official MATLAB script provided by the DTU dataset.

[0131] This embodiment uses the DTU dataset for training, and the obtained training weights are used to fuse into a three-dimensional point cloud. The official MATLAB script provided by the DTU dataset is then used to calculate the accuracy and completeness, and the average of the two is calculated to comprehensively evaluate the overall reconstruction quality. The quantitative table is shown in Table 2. Although Gipuma shows the best accuracy in terms of progress, the CMVD-AANet of this embodiment shows excellent technical levels in terms of completeness and overall reconstruction quality, which is far behind previous methods. Figure 3 and Figure 4 In this embodiment, the 3D reconstruction results of the first scene and the tenth scene will be reconstructed. In these examples, CMVD-AANet shows quite good completeness. However, the point cloud generated by Cas-MVSNset lacks completeness. This is because it ignores the combination of global feature information and contextual information, which leads to the lack of completeness of the reconstructed point cloud. However, the 3D point cloud model generated by the MVSNet method is relatively sparse and has more cluttered points. This is because it does not use a multi-scale cascade network to refine the depth hypothesis range, resulting in inaccurate depth estimation. The reconstructed point cloud has more cluttered points and is sparse. Figure 3 and Figure 4 As shown, the point cloud model generated by the method of this embodiment not only performs well in terms of integrity, but also the reconstruction result of this embodiment is smoother and easier to identify from the three-dimensional reconstruction result.

[0132] Table 2

[0133]

[0134]

[0135] The quantitative results of the DTU dataset are shown in Table 2, where lower values ​​indicate better results. Compared with some other deep learning methods (such as standard MVSNet) and traditional multi-view stereo methods, CMVD-AANet outperforms all of the above methods in terms of completeness and overall scores.

[0136] like Figure 3 As shown, for this particular scene, this embodiment shows the fused point cloud model in the top row and the enlarged local area in the rendered image in the bottom row. Compared with other methods such as MVSNet, the method proposed in this embodiment shows superiority in the completeness and accuracy of reconstructing the point cloud.

[0137] like Figure 4 As shown, the top row shows the fused point cloud model, and the bottom row shows the enlarged local area. The method of this embodiment is tested with other methods such as CasMVSNet and returns a denser reconstruction result.

[0138] Furthermore, in order to evaluate the generalization ability of the method of the present invention, this embodiment uses the intermediate set of the tank and temple datasets. In this embodiment, the size of the input image is set to 2048×1056, and the number of views N is 7. Since the images of the tank and temple datasets are taken outdoors, in order to improve the performance in real-world scenes, this embodiment uses a model trained on the BlendedMVS dataset covering indoor and outdoor scenes for evaluation. The quantitative results of the model of this embodiment are shown in Table 3. The indicator is the average F score, which takes into account both accuracy and completeness to comprehensively measure the quality of reconstruction. The quality of the corresponding reconstructed point cloud is better than the listed classic MVS methods (Schonberger et al., 2016) and the recently proposed learning-based methods (Yao et al., 2018; Yao et al., 2019; Chen et al., 2019; Zehao et al., 2020; Patchmatchnet et al., 2021). The model of this embodiment achieved higher F scores in the home, horse, M60 and Francis scenes. At the same time, this embodiment Figure 5 The reconstruction results of this embodiment are shown in FIG. 4 . In some scenes with poor textures, the method successfully captures position-related features and fuses global information in different views.

[0139] Table 3

[0140]

[0141]

[0142] The benchmark results on the Tanks and Temples dataset are shown in Table 3. The evaluation metric of the point cloud in this dataset is F-score, and the higher the value, the better. AGF-MVSNet achieves excellent performance on the middle leaderboard of the Tanks and Temples dataset.

[0143] Furthermore, both the DTU dataset and the Tanks and Temples dataset contain evaluation algorithms for point cloud results. However, the BlendedMVS dataset does not provide any algorithm to compare the quality of point cloud reconstruction. Therefore, in the CMVD-AANet model proposed in this embodiment, this embodiment uses formula (13) to evaluate the quality of the reconstructed depth map prediction. For the sake of fairness, this embodiment is compared with MVSNet, CasMVSNet and other networks. The dataset used in this embodiment only contains a large-scale outdoor scene of a drone aerial image, and the image resolution is set to 512x640. The reconstructed result is as follows: Figure 6 and Figure 7 As shown, the evaluation algorithm using formula (13) was evaluated as shown in Table 2. Where EPE represents the endpoint error, that is, the average L-1 distance between the predicted depth map and the true depth map, <1pix err and <3pixerr represent the proportion of pixels (%) with depth errors greater than 1 and greater than 3, respectively.

[0144] Table 4

[0145]

[0146] As shown in Table 4, the quantitative results of the predicted depth map error on the BlendedMVS dataset, the smaller the value, the better. It can be seen that the algorithm of this embodiment is superior to other methods in all experimental methods, especially in EPE, the network model of this embodiment predicts the smallest error for the BlendedMVS data, proving its ability to generate high-quality depth maps, and at the same time proving the generalization ability of the network model of this embodiment.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An attention-guided multi-view depth estimation method, characterized in that: The method comprises the following steps: S1, extracting multi-scale features from the input image to obtain a pyramid feature map; S2, combining consistency information and contextual global cues to construct attention-based global feature weights; S3, applying the global feature weight to the feature volumes of all feature maps to obtain the cost volume; S4, generating attention weights by extracting correlation geometric information between the source feature map and the reference feature map after homography transformation; S5. Filter the cost body using the attention weight to obtain the attention cost body; S6. After preliminary processing of the attention cost body, a deep regression algorithm is used to infer the depth map and the corresponding confidence map; In step S1, referring to the feature pyramid network, feature maps of multi-scale spatial resolutions are used to construct cost volumes of different resolutions; for each input image i∈{0,1,…,N}, where I0 is the reference image; Starting from the reference image, an L-level image pyramid is constructed, where the bottom layer of the image pyramid corresponds to the input image Then, a convolutional neural network is used to extract features to obtain the feature maps of each layer, where the number of channels of layers conv0, conv1, and conv2 is increased; represents the feature map extracted by the Lth layer, where f l i represents the i-th feature map in the l-th layer, R represents the set of feature maps, H represents height, W represents width, and Ch represents the number of output channels; In step S2, the following steps are specifically included: S21, splicing the feature maps of the reference image and the source image in the same batch; S22, apply global average pooling to reduce all feature maps to their respective channel dimensions; S23, calculating the weight of the global feature according to the context channel information of each scene; The specific calculation process is as follows: w=Sigmoid(L1(ReLU(L2(GAP(F),s1)),s2) (2) V i =v i *w (3) In formula (1), f i represents the reference image, f0 represents the original image, concat() is the concatenation function, and F represents the concatenated feature map; in formula (2), L1(·,·) and L2(·,·) are two linear transformations, where s1 and s2 represent the corresponding transformation parameters respectively; GAP() represents the average pooling of the weight value of the feature map F; ReLU() is the activation layer for fitting; Sigmoid() represents the normalization function, which normalizes the output to the range of 0-1; in formula (3), v i represents the feature volume obtained by a differentiable homography transformation that transforms all features at assumed depth d to the frontal-parallel plane of the reference view.

2. The attention-guided multi-view depth estimation method according to claim 1, characterized in that: In step S3, the global feature weight w is calculated i , improve the matching graph f i , where ⊕ is the channel multiplication and M is the mapping defined by the cost metric: in is the average of all confidence maps matched by attention enhancement. All the above operations are performed pixel by pixel. C represents the cost volume, which is constructed by calculating the mean variance between the reference and source features.

3. The attention-guided multi-view depth estimation method according to claim 2, characterized in that: In step S4, the group-wise method of GWCNet is used to group the features, calculate the relevant graph information group by group, and divide the T channels into G groups on average, so that feature graphs of different levels will not interfere with each other; First, the matching similarity is calculated; among them, F0(p), F i (p i,j ) are the reference feature map and the source feature map respectively, then the feature similarity S of the Gth group i (p,j) is: Where <·,·> is the similarity inner product between the reference feature map and the source feature map, and S i (p,j)∈R G represents the similarity vectors within different groups, g represents the group number of the G group into which the T channel is evenly divided; For each source view, estimate a pixel-wise view weight {W i (p)}, based on the initial depth hypothesis space obtained in the first iteration, the visible information of all source views is integrated; using W i (p) represents the source view I i The visibility of pixel P in the similarity matrix S is calculated by using a 3D convolution containing a 1*1*1 convolution kernel and a Sigmoid activation layer. i Processing, get a weight P between 0-1 i , for source view I i The pixel point p, its weight value is expressed as: w i (p)=max{P i (p,j)|j=0,1,…,D-1} (6) Where P i Represents the visible probability of point p at depth j, weight w i is regarded as the global attention weight of the i-th view relative to the reference view, and D is the predicted depth.

4. The attention-guided multi-view depth estimation method according to claim 3, characterized in that: In step S5, the weight w is used i Pair similarity S i Weighted to increase its robustness, for all source views of the pixel point P of the j-th depth layer, its group similarity expression is as follows: Using two 3D convolutions and a 3D hourglass network Regularization is performed, and the convolution layer is used to compress the depth channel to 1, and then the softmax is used to process the depth dimension into a probability value between 0 and 1, so as to obtain the attention weight A∈R C×1×H / 4×W / 4 ; Use the attention weight A to eliminate the redundant information in the initial cost body, and the attention cost body C on channel i A Calculated as: C A =A⊙C (8) ⊙ represents the pixel-by-pixel product. By filtering the initial cost body C using the attention weight A, a more accurate attention cost body C is obtained. A .

5. The attention-guided multi-view depth estimation method according to claim 4, characterized in that: In step S6, the attention cost body C A Regularization is performed by encoding and then decoding it using a 3D convolutional layer to filter out the noise pollution and object occlusion errors of the original cost volume C; finally, a deep regression algorithm is used to infer the depth map and the corresponding confidence map; the depth range [d min ,d max The probability P(d) of each sampled depth d in ] and the predicted depth D at each labeled pixel are calculated as: where d max , d min are the maximum depth and minimum depth of the depth estimation range respectively; For the intermediate outputs generated by the cost body of N layers and the intermediate outputs of the group-related attention cost body, the total loss function for supervising all outputs is: Among them, use To calculate the loss; d att is the group-related attention weight to obtain the depth map, d i is the depth map obtained by depth regression of the multi-scale cost volume; att is the coefficient of the predicted data, λ i is the corresponding loss weight.

Images