Mvsnet cost volume construction method based on co visibility

Abstract

Based on the cost volume construction method of MVSNet, the importance of different views in depth estimation of each pixel is measured by using attention mechanism, and the cost volume is constructed accordingly, which finally improves the accuracy of DSM generation (MAE index is reduced, <1-interval index and <3-interval index are improved).

Description

Technical Field

[0001] This invention relates to the field of digital surface model (DSM) generation from satellite remote sensing images, specifically a method for constructing cost voxels based on common-viewability MVSNet. Background Technology

[0002] Remote sensing technology primarily utilizes equipment such as drones and satellites to capture ground feature data, and then processes and analyzes this data to obtain the target's location, attributes, and other states, as well as their changing patterns. As early as the last century, remote sensing technology was listed as a top priority in national scientific and technological research. With the development of related scientific research, this technology has been applied to all aspects of my country's socio-economic development, such as military target reconnaissance and monitoring, crop growth surveys, environmental quality monitoring, surface resource exploration, weather forecasting, and disaster observation; bringing significant benefits to my country's economy and social security. As one of the many technologies in the field of remote sensing, satellite remote sensing image DSM generation technology aims to construct a shared-view area DSM based on multiple satellite remote sensing images, providing a three-dimensional description of the area's topographic and geomorphological features from a microscopic perspective. DSM is the most important spatial information data in geographic information databases, a core data system for topographic analysis, and also an important framework data for building national spatial data infrastructure.

[0003] Current mainstream traditional satellite remote sensing imagery DSM generation schemes are mostly based on planar scanning algorithms. This involves pre-assuming multiple hypothetical depth planes and constructing matching costs using constraints such as photometric consistency, local structure invariance, and depth continuity, then estimating depth through optimization. This approach has the following drawbacks: First, depth estimation can only be performed between two images. For multi-view stereo (MVS) scenes, this scheme can only perform pairwise matching to estimate depth before fusing the results to obtain the final depth; this is inefficient and time-consuming. Second, planar scanning algorithms essentially estimate depth through pixel matching between two images; since matching algorithms are generally inefficient in weak textures, repetitive textures, and occluded areas, they cannot achieve depth estimation in these regions.

[0004] To address the aforementioned shortcomings, current research is transferring MVSNet, a deep neural network that excels in 3D reconstruction of close-up scenes, to remote sensing scenes to generate depth maps. These maps are then combined with pixel latitude and longitude coordinates to transform them into DSM data. The main idea of ​​MVSNet is to use 3D voxels to describe the matching cost, then generate depth probabilities for each pixel through 3D convolution, and finally perform a weighted average to generate the final depth map. In the matching cost construction process, it assumes that "all images are equally important for the pixel depth estimation of the reference image." This is now being combined with real-world scenarios (such as...). Figure 1 A feasibility analysis of this hypothesis will be conducted.

[0005] In the figure, a, b and c are three spatial points located in the field of view of the reference camera, and depth estimation is performed on them. Among them, point c can be "seen" by all other auxiliary cameras; therefore, for the depth estimation of this point, all remote image features should be equally important. Point b can be "seen" by auxiliary camera 1 but cannot be "seen" by auxiliary camera 2; therefore, for the depth estimation of this point, the weight of the auxiliary remote image 2 feature should be as small as possible; the depth of the current pixel should be determined by the remaining two image features as much as possible. Point a is only visible to the reference camera and invisible to the auxiliary cameras; therefore, the depth of the point needs to be estimated in combination with the depth of its neighbor pixels. As can be seen from the example, when performing depth estimation on different pixels of the reference image, the weight of the feature corresponding to different views should be determined according to the co-visibility of the auxiliary image and the reference image, and the hypothesis of "all image features are equally important" of MVSNet is not reasonable.

[0006] As can be seen, when performing depth estimation on the reference image, all auxiliary images cannot be treated equally; a reasonable cost volume should be constructed in combination with the visibility of the spatial point to the auxiliary camera, and then aggregated to estimate a more accurate depth map. SUMMARY

[0007] To overcome the shortcomings of the prior art, the present application proposes a MVSNet cost voxel construction method based on co-visibility. MVSNet first constructs a plurality of depth hypothetical planes when constructing the cost volume, and then converts all auxiliary image features to the reference camera coordinate system for subsequent processing according to the depth plane and the camera internal and external parameters. This process is a homographic transformation. The specific transformation formula is as shown in formula 1.

[0008]

[0009] Among them, K1, R1, t1 and K i , R i , t i respectively represent the internal parameter matrix, the rotation matrix and the translation vector of the reference camera and the auxiliary camera, I represents the unit matrix, n1 represents the optical axis direction vector of the reference camera, and d represents the plane depth. As can be seen from the formula, under the condition that the internal and external parameters of the reference camera and the auxiliary camera i are known, the homographic transformation is a function of the depth d. It can map the pixel p on the reference image to the pixel p' i on the auxiliary image i

[0010]

[0011] For different depth hypothetical planes d j , j ∈ (1, 2, …, n), there is a series of homographic transformation matrices, from which a series of candidate matching pixels to form a candidate matching pixel set. The similarity of the pixel pair is calculated as the score of the candidate matching pixel ; secondly, for each depth hypothesis plane d j , the candidate matching pixel with the highest score is selected from the auxiliary images i to construct the cost of the pixel p on the depth hypothesis plane d j .

[0012]

[0013]

[0014] In formula 3, is used to describe the similarity of the pixel p and the candidate matching pixel ; and represents the pixel most similar to the pixel p in the candidate matching pixel set; in formula 4, represents the cost of the pixel p on the depth hypothesis plane d j . It can be seen that the most important part of the present application is the establishment of the similarity function f(·). Similar to the feature point method in the image registration task, the present application constructs a suitable descriptor for each pixel to encode the gray value of the pixel itself and the surrounding information, and then calculates the distance between the two pixel descriptors to measure whether the two pixels are similar or not.

[0015] The technical solution of the present application is as follows:

[0016] A cost voxel construction method based on MVSNet of common visibility, characterized in that it comprises the following steps:

[0017] ① Descriptor extraction: features are extracted from multiple remote sensing images (including reference images and auxiliary images) by using an encoder, and the extracted features are used as the descriptors of the corresponding pixels; the encoder fuses the features of the pixel neighborhood while down-sampling the remote sensing images to extract the local texture structure of the images. Thus, the present application directly uses the channel dimension information corresponding to each pixel in the encoder output feature as the descriptor of the pixel with respect to the original satellite remote sensing image.

[0018] ② Cost calculation: for the auxiliary image i, there will be a series of candidate matching pixels j , j∈(1, 2,..., n) due to the difference of the depth hypothesis plane d to form a candidate matching pixel set. In the candidate matching pixel set, the similarity of the pixel pair is calculated as the score of the candidate matching pixel . For each depth hypothesis plane d jwhere j is the index of the auxiliary image i, and the cost of pixel p on the depth hypothesis plane d is constructed by weighting the scores of the candidate matching pixels of auxiliary image i corresponding to the reference image descriptor of pixel p according to the cross-attention mechanism. j

[0019] ③ Depth estimation: the obtained cost volume based on the cross-attention is aggregated and regressed by using 3D convolution to estimate the optimal depth.

[0020] Further, for the step ②, the application designs a candidate matching pixel similarity calculation module based on the cross-attention mechanism, and the main steps are as follows:

[0021] First step: taking the block corresponding to a certain depth hypothesis plane of the reference image descriptor and the auxiliary image descriptor obtained by step ① and rearranging them into a one-dimensional vector.

[0022] Second step: performing multiplication operation on the two to obtain a weight matrix; using the weight matrix to weight each auxiliary image descriptor to generate an optimal auxiliary descriptor.

[0023] Third step: taking the difference between the optimal auxiliary descriptor and the reference descriptor and squaring to construct the MVSNet cost volume based on the cross-attention.

[0024] Compared with the prior art, the application has the beneficial effects that the importance of different views in depth estimation of each pixel is measured by using the attention mechanism, and the cost volume is constructed accordingly, and finally the DSM generation accuracy is improved (the MAE index is reduced, and the <1-interval index and the <3-interval index are improved). BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a common cross-attention phenomenon in existing multi-view tasks.

[0026] Figure 2 is a specific implementation of the MVSNet cost volume construction method based on the cross-attention of the application, that is, a neural network module based on the attention mechanism.

[0027] Figure 3 is a schematic diagram of the overall process of the MVSNet cost volume construction method based on the cross-attention of the application.

[0028] Figure 4 ​​is the depth map estimated by the MVSNet series model adopting the cost voxel construction method based on co-visibility of the application. Wherein, a is the original satellite remote sensing image; b is the depth map generated by the CasMVSNet adopting the construction method; c is the depth map generated by the UniMVSNet adopting the construction method; d is the depth map generated by the Fast-SatMVSNet-16 adopting the construction method; e is the depth map generated by the Fast-SatMVSNet-8 adopting the construction method. DETAILED DESCRIPTION

[0029] The application will be described in detail below in combination with the drawings and examples, but the protection scope of the application should not be limited thereby.

[0030] The application designs a network module based on attention mechanism as shown in Figure 2 The following will be described in terms of "how does the neural network extract the descriptor" and "how to construct the cost by using the attention mechanism".

[0031] On the one hand, the neural network has a natural advantage in descriptor construction. The role of the encoder itself is to fuse the neighborhood features while down-sampling the image to extract the texture structure features of the local region of the image. In fact, this is not inconsistent with the original intention of descriptor construction. Therefore, the channel dimension C information corresponding to each pixel in the encoder output feature can be considered as the descriptor of the receptive field of the pixel in the original satellite remote sensing image.

[0032] On the other hand, the attention mechanism can be understood as a whole: for each depth hypothesis plane, the reference features are used to filter the auxiliary features. The filtering process mainly includes first multiplying the current pixel descriptor of the reference feature with the corresponding pixel descriptor in different auxiliary features to generate a weight matrix; and then using the weight matrix to weight the corresponding pixels in multiple auxiliary features to select the optimal pixel from the corresponding pixels in multiple auxiliary features.

[0033] Figure 2 In the figure, the three blocks marked F represent the reference features and auxiliary features (their size is [H, W, C]) for one depth hypothesis plane, and the blocks are taken and rearranged into one-dimensional vectors (the one-dimensional vector is relative to each block, and the whole processed feature should be the result obtained by spatial domain connection of the one-dimensional vector according to the original block position) and then input to the subsequent cross-attention module to learn the importance of different views for the depth estimation of the current pixel under different depth hypothesis planes.

[0034] In this embodiment: assuming the block size is 1, the feature size obtained is still [H, W, C]; this feature is divided into "reference" and "auxiliary" two categories and reorganized, and the reference feature (size [H, W, C, 1]) and the auxiliary feature (size [H, W, C, 2]) can be obtained. Next, the two are integrated to obtain the weight matrix (size [H, W, 1, 2]); the optimal auxiliary feature (size [H, W, C, 1]) is generated by screening the auxiliary feature using the weight matrix. Finally, the optimal auxiliary feature is subtracted from the reference feature, and the square is constructed to construct the cost volume based on the common view with the size of [H, W, C, 1] (equation 4).

[0035] Cost volume construction comparison test:

[0036] The data set used in this experiment is TLC SatMVS, which is made of mountain optical remote sensing images collected by ZY-3 satellite. Compared with the Worldview series of satellites, ZY-3 can capture images from three angles at the same time for the same scene, thereby weakening the influence of seasonal weather alternation and light and dark changes caused by time change, which is conducive to the reconstruction of the earth's surface model. The data set contains two versions, one is a large size remote sensing image set, and the other is a preprocessed remote sensing image set suitable for machine learning. The second version corresponds to two imaging models (i.e. RPC model and its approximate pinhole camera model), and the latter is selected as the data set to train various models to test its performance in the task of satellite remote sensing image DSM generation. The data set contains a total of 6802 sets of mountain image data, and the three images obtained by shooting the same scene from three perspectives are called a group of information here. Specifically, a group of data includes: (1) three optical remote sensing images for the same scene, with a size of 768x384; (2) the corresponding depth ground truth of the scene, which is used as the label for model training and testing; (3) the approximate camera internal and external parameters corresponding to the three images. Among the 6802 groups of data, 5011 groups of data are used for model training, and another 1791 groups of data are used for model testing.

[0037] In the experiment, the models participating in the comparison are all variants of the classic network MVSNet in the field of MVS, mainly including: (1) CasMVSNet, which adopts a three-stage depth estimation mode from coarse to fine; a smaller cost volume is used to estimate a rough depth map, and then the depth map is used to limit the range of the next stage of depth hypothesis plane setting. (2) UniMVSNet, which divides the entire depth estimation problem into two modules of depth calculation and depth optimization; the former can be regarded as a classification problem, and the latter can be regarded as a regression problem. Based on this analysis, a new type of loss (unified focal loss) is designed, so as to more unified and reasonable describe the performance of the model. (3) Fast-SatMVSNet-16 and Fast-SatMVSNet-8, which reduce the calculation amount of 3D convolution by simplifying the cost volume, and transfer the calculation pressure of 3D convolution to 2D network, thereby reducing the inference time of the model; the simplification process mainly considers the redundancy of the cost volume in the D, H, W and C dimensions.

[0038] The evaluation indexes used in the experiment include: (1) <1-interval and <3-interval, which represent the percentage of pixels with an error less than 1 times and 3 times the minimum hypothesis depth plane interval in the depth map generated by the model, respectively; the larger the values of the two indexes, the higher the completeness of the DSM generated by the model. (2) Mean Absolute Error (MAE, Mean Absolute Error), also known as L1 divergence, represents the average absolute error of the depth map generated by the model and the corresponding true value; the smaller the value of the index, the higher the accuracy of the DSM generated by the model.

[0039] The comparison results of the final cost volume construction are shown in Table 1, wherein the models marked with "*" take the feature volume variance as the cost volume to represent the hypothesis that "all image pairs are equally important for reference image pixel depth estimation"; and the remaining comparison models use the cost volume construction scheme based on the common visibility described in the present application.

[0040] Table 1 Comparison of model performance with different cost volume construction methods

[0041]

[0042] As can be seen from the above table, the cost volume construction method based on common visibility is better than the cost volume construction method based on variance in terms of the three precision indexes of MAE, <1-interval and <3-interval, especially the <1-interval index is improved by nearly 3 percentage points; thereby verifying the effectiveness of the present application.

Claims

1.A method for constructing cost volume based on co visibility MVSNet, characterized in that, The method comprises the following steps: ① description sub-extraction: using an encoder to extract features from multiple remote sensing images, the remote sensing images including reference images and auxiliary images, and taking the extracted channel dimension information features as the descriptors of the corresponding pixels with respect to the original satellite remote sensing images; ii. Cost computation: For the auxiliary image i, there will be a set of candidate matching pixels j , d , d , d , d , d , d , d , d , d , d , d , d , d , d < In the candidate matching pixel set, the similarity of the pixel pair is calculated based on the descriptor generated in the previous step as the score of the candidate matching pixel . ​ For each depth hypothesis plane d j , j e (1, 2,..., n), the cost of pixel p at depth hypothesis plane d is constructed as a weighted sum of the scores of the candidate matching pixels j according to the auxiliary image i. ③ depth estimation: using 3D convolution to aggregate and regress the obtained cost voxel based on co-visibility to estimate the optimal depth; The step ② cost calculation comprises the following specific steps: S21. Taking the reference image descriptors and the auxiliary image descriptors obtained from the step ①, taking the descriptors corresponding to a certain depth imaginary plane as blocks and rearranging them into one-dimensional vectors; S22. performing multiplication operation on the two to obtain a weight matrix, and using the weight matrix to weight each auxiliary image descriptor to generate an optimal auxiliary image descriptor; S23. taking the difference between the optimal auxiliary image descriptor and the reference image descriptor and taking the square to construct the MVSNet cost voxel based on co-visibility.

Images