Multi-view remote sensing image depth estimation method

Abstract

The invention provides a multi-view remote sensing image depth estimation method. The method comprises the following steps: acquiring a training sample set and a test sample set; constructing a remote sensing image depth estimation network model and carrying out iterative training on the remote sensing image depth estimation network model; and obtaining a remote sensing image depth estimation result. In the process of training a remote sensing image depth estimation network model and obtaining a depth estimation result, clearer matching distribution is formed through richer high-order features of a low-texture region extracted by a single-view remote sensing image depth estimation branch Mono; an epipolar attention aggregation module in the multi-view remote sensing image depth estimation branch MVS improves the feature discrimination degree in the epipolar direction, and provides stronger geometric constraints in a weak texture region; a single-view depth estimation result correction module Correct reduces systematic deviation by aligning single-view and multi-view depth estimation results and learning global scale transformation parameters and offset parameters, and further improves the estimation precision of remote sensing images.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and relates to a multi-view remote sensing image depth estimation method, which can be used in the fields of remote sensing scene depth estimation, three-dimensional reconstruction and geographic mapping. BACKGROUND

[0002] Remote sensing image depth estimation aims to recover the spatial depth information of each pixel in the scene by processing remote sensing images obtained by aerial photography of unmanned aerial vehicles, satellites and the like. Generally, due to the difficulty in feature extraction and matching caused by weak texture information in suburban scenes, the depth estimation effect is poor, which makes the existing remote sensing image depth estimation method not suitable for scene reconstruction in weak texture areas. Therefore, how to design a more robust feature extraction network, and enhance the feature expression ability by introducing multi-scale information fusion, spatial attention mechanism and other technologies, is a key factor to improve the discrimination ability of weak texture areas.

[0003] For example, Wuhan Tuoke Intelligent Technology Co., Ltd. discloses a multi-view depth estimation method suitable for unmanned aerial vehicle platforms in its patent document "a multi-view depth estimation method and device suitable for unmanned aerial vehicle platforms" (patent application number: CN202210874554.9, publication number: CN115082540A). According to each group of images and homographic transformation, a plurality of two-dimensional cost volumes are obtained. Each two-dimensional cost volume is decoded and edge feature extraction is performed to obtain its corresponding feature map and probability map. Each probability map and the feature map are multiplied point by point to obtain a plurality of affinity matrices. Each affinity matrix and each two-dimensional cost volume are iteratively propagated multiple times to obtain the final edge information guided cost volume and its corresponding edge optimized depth map. The depth map is optimized and decoded to obtain the final depth map estimation. This invention improves the estimation accuracy of remote sensing images, but it only considers the effective feature representation of the edge region and the occluded region, and does not fully solve the problem of weak texture region feature extraction in remote sensing scenes. The feature extraction ability is weak, resulting in low depth estimation accuracy in weak texture areas of remote sensing scenes. SUMMARY

[0004] The present application aims to solve the technical problem of low depth estimation accuracy in weak texture areas of remote sensing scenes due to weak feature extraction ability in the prior art.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application includes the following steps:

[0006] (1) Obtain a training sample set and a test sample set:

[0007] N unmanned aerial vehicle aerial scene remote sensing images and depth value labels are acquired, T remote sensing images and real depth value labels are used to form a training sample set, and the remaining N-T remote sensing images and real depth value labels are used to form a test sample set, wherein N>30000,

[0008] (2) Construct a remote sensing image depth estimation network model:

[0009] A remote sensing image depth estimation network model O is constructed, which includes a single-view remote sensing image depth estimation branch Mono arranged in parallel for extracting a single-view depth estimation image and a single-view depth estimation confidence map, a multi-view remote sensing image depth estimation branch MVS for extracting a multi-view depth estimation image and a multi-view depth estimation confidence map, and a fusion optimization calculation unit Fusion cascaded with the output ends of Mono and MVS, wherein Mono includes a feature extraction module, MVS includes an epipolar attention aggregation module, and the output end of the feature extraction module is bridged with the input end of the epipolar attention aggregation module; a single-view depth estimation result correction module Correct is loaded between Mono and Fusion, which is used to correct the single-view depth estimation image;

[0010] (3) Iterative training of the remote sensing image depth estimation network model O:

[0011] The remote sensing image depth estimation network model O is iteratively trained through the training sample set, and a trained remote sensing image depth estimation network model O * is obtained.

[0012] (4) Obtain the remote sensing image depth estimation result:

[0013] The test sample set is used as the input of the remote sensing image depth estimation network model O * for forward propagation, and the remote sensing image depth estimation results corresponding to N-T test samples are obtained.

[0014] Compared with the prior art, the present application has the following advantages:

[0015] The application enriches the high-order features of the low-texture area extracted by the single-view remote sensing image depth estimation branch Mono, so that a clearer matching distribution is formed in the cost volume; the epipolar attention aggregation module in the multi-view remote sensing image depth estimation branch MVS improves the feature discrimination in the epipolar direction and provides stronger geometric constraints in the weak texture area; the single-view depth estimation result correction module Correct aligns the preliminary depth estimation interval and learns the global scale transformation parameter and offset parameter, so that the single-view depth estimation result and the multi-view depth estimation result are aligned in scale, the systematic deviation is reduced, and the accuracy of the remote sensing image depth estimation is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart for the implementation of the application.

[0017] Figure 2 The structural diagram of the remote sensing image depth estimation network model in the application.

[0018] Figure 3 The unmanned aerial vehicle aerial image used in the simulation of the application.

[0019] Figure 4 The error image of the simulation results of the depth estimation of the application and the prior art. Figure 3 DETAILED DESCRIPTION

[0020] The application will be further described in detail below with reference to the drawings and specific embodiments.

[0021] With reference to Figure 1 , the application comprises the following steps:

[0022] Step 1) Obtain a training sample set and a test sample set:

[0023] Obtain N unmanned aerial vehicle aerial scene remote sensing images and their real depth value labels, and group T remote sensing images and their real depth value labels into a training sample set, and group the remaining N-T preprocessed remote sensing images and their real depth value labels into a test sample set, N=31960 and T=27460 in this embodiment;

[0024] Obtain the real depth value label of the remote sensing image, including obtaining the three-dimensional point C 3d project to the camera coordinate system by using the camera external parameter (R, T), then project to the camera plane corresponding to the current remote sensing image by using the camera internal parameter F, to obtain the pixel (x, y); then traverse each remote sensing image pixel (x, y), and use the inverse matrix F -1 ​, Camera extrinsic parameters (R, T) are calculated by a pixel ray L(t) passing through the camera optical center direction, and the ray is matched with a three-dimensional point C 3d The intersection point is calculated, and the distance between the pixel and the intersection point is the depth value d corresponding to the pixel position x,y The depth value corresponding to each pixel is composed into a depth map D with the same size WxH as the original remote sensing image gt The depth map is used as the real depth value label corresponding to the remote sensing image, and the specific implementation is as follows:

[0025] (x, y) = F · (R · (C 3d -T))

[0026] L(t) = T + tK -1 ·[x, y, 1] T

[0027] d x,y = ||L(t) - C 3d ||

[0028]

[0029] Where, · represents matrix multiplication operation, F -1 represents the inverse operation of F, [x, y, 1] T represents the transpose operation of the vector [x, y, 1], and t represents the positive real number parameter in the corresponding ray equation. In this embodiment, W = 768 and H = 384.

[0030] Step 2) Construct a remote sensing image depth estimation network model, the structure of which is shown in Figure 2

[0031] The remote sensing image depth estimation network model O is constructed to include a single-view remote sensing image depth estimation branch Mono arranged in parallel for extracting a single-view depth estimation map and a single-view depth estimation confidence map, a multi-view remote sensing image depth estimation branch MVS for extracting a multi-view depth estimation map and a multi-view depth estimation confidence map, and a fusion optimization calculation unit Fusion cascaded with the output ends of Mono and MVS, wherein Mono includes a feature extraction module, MVS includes an epipolar attention aggregation module, the output end of the feature extraction module is bridged with the input end of the epipolar attention aggregation module, the feature extraction module includes a plurality of ViT layers stacked therein, the epipolar attention aggregation module adopts a convolution layer structure, a feature projection calculation unit, a Softmax function layer for attention weight calculation, and an attention weighted summation calculation unit loaded after an activation function layer; a single-view depth estimation result correction module Correct is loaded between Mono and Fusion, which is used to correct the single-view depth estimation map, wherein:

[0032] ​The monocular remote sensing image depth estimation branch Mono further includes a depth estimation regression head cascaded with the feature extraction module; the depth estimation regression head includes stacked convolution layers and a Softmax function layer, and a weighted summation calculation unit loaded at an output end of the Softmax function layer;

[0033] The multi-view remote sensing image depth estimation branch MVS further includes a hierarchical feature extraction module cascaded with the epipolar attention aggregation module, a cost volume construction calculation unit, a cost volume aggregation module and a depth estimation regression head; the hierarchical feature extraction module includes stacked 3x3 convolution layers and 5x5 convolution layers, a global context enhancement module and a decoding and feature recovery module, the global context enhancement module includes a global average pooling layer and ReLU function layers and Sigmoid function layers cascaded with the global average pooling layer, and full connection layers loaded between the global average pooling layer and the ReLU function layers and between the ReLU function layers and the Sigmoid function layers, and the decoding and feature recovery module includes stacked deconvolution layers; the cost volume aggregation module includes stacked 3D convolution layers, 3D batch normalization layers and ReLU function layer combinations and corresponding 3D deconvolution layers, 3D batch normalization layers and ReLU function layer combinations; the depth estimation regression head includes stacked convolution layers and a Softmax function layer cascaded with the convolution layers, and a weighted summation calculation unit loaded after the Softmax function layer;

[0034] The monocular depth estimation result correction module Correct includes stacked multiple convolution layers, batch normalization layers and ReLU function layers, and a calculation unit loaded after the ReLU function layers;

[0035] Step 3) iteratively training the remote sensing image depth estimation network model O:

[0036] The remote sensing image depth estimation network model O is iteratively trained through the training sample set, and a trained remote sensing image depth estimation network model O is obtained * , and the specific steps are as follows:

[0037] (3a) initializing the iteration number as a, the maximum iteration number as A, A>8000, the weight and bias parameters in the remote sensing image depth estimation network model O a in the a-th iteration are w a and b a , and a=0;

[0038] (3b) randomly selecting P training samples in the training sample set as the input of the remote sensing image depth estimation network model O for forward propagation, to obtain the depth estimation results of P remote sensing images, and in this embodiment, P=2;

[0039] (3b1) The feature extraction module in the monocular remote sensing image depth estimation branch Mono performs feature extraction on each training sample; the depth estimation regression head performs depth estimation regression on the extracted monocular feature map to obtain a monocular depth estimation map and a monocular depth estimation confidence map

[0040] (3b2) The monocular depth estimation result correction module Correct corrects the monocular depth estimation map to obtain a corrected monocular depth estimation result The implementation is as follows:

[0041] The monocular depth estimation map is input into the monocular depth estimation result correction module Correct, and after passing through two 1×1 convolution layers and a fully connected layer, global scale parameter s and offset parameter o are obtained, and the monocular depth estimation map is mapped to a scale space consistent with the multi-view depth estimation map to obtain the corrected monocular depth estimation result

[0042]

[0043] (3b3) The epipolar attention aggregation module in the multi-view remote sensing image depth estimation branch MVS performs epipolar attention aggregation on each training sample and the monocular feature map ; the hierarchical feature extraction module performs feature extraction on the aggregated feature map ; the cost volume construction calculation unit performs cost volume construction on the extracted feature map ; the cost volume aggregation module performs cost volume aggregation on the constructed initial cost volume ; the depth estimation regression head performs depth estimation regression on the aggregated cost volume to obtain a multi-view depth estimation map and a multi-view depth estimation confidence map

[0044] Among them, the epipolar attention aggregation module extracts attention features of the monocular feature map , including query feature Q, key feature K, and value feature V; and projects K to D different depth values to obtain key feature K corresponding to depth value d d , d∈D, D represents a set of depth values; simultaneously calculates the attention weight A of Q and K; then performs weighted summation on K through A to obtain the feature fusion result wherein:

[0045]

[0046] K d =Warp(K,d)

[0047]

[0048] wherein, W q , W k , W v represent the convolution kernel of Q, K, V respectively, * represents the convolution operation, Warp(·) represents the homographic transformation operation, Softmax(·) represents the weight calculation operation, represents the transpose result of K d , C represents the dimension of K, h represents the number of attention heads, V d represents the value feature corresponding to the depth value d.

[0049] (3b4) Single view and multi-view depth estimation result fusion optimization calculation unit Fusion carries out confidence weighted fusion calculation on the corrected single view depth estimation result and confidence map and multi-view depth estimation map and multi-view depth estimation confidence map to obtain the final depth estimation result map The specific implementation is as follows:

[0050]

[0051] wherein, ε represents a smoothing parameter.

[0052] (3c) Smooth L1 loss function is adopted, and the network model loss value L a is calculated through the depth estimation result of each remote sensing image and its corresponding real label, then the gradient descent method is adopted, and the weight, bias parameters w a , b a are updated through L a to obtain the remote sensing image depth estimation network model O a of this iteration.

[0053] The network model loss value L a is calculated, and the specific implementation is as follows:

[0054]

[0055] wherein, represents that multiple Smooth L1 loss functions are used to calculate the loss function of the depth estimation task, represents that the Smooth L1 loss is calculated for and , represents the true depth value label; and Smooth L1 loss is calculated, represents the true depth value label; and Smooth L1 loss is calculated, represents the true depth value label; represents a loss function with a confidence constraint, β is a confidence consistency penalty coefficient, |·| represents an absolute value operation, and Σ represents a summation operation, represents a loss function of the gradient matching task, and respectively represent gradient calculation of the depth map in the horizontal and vertical directions; λ conf ,λ grad are weight coefficients corresponding to the loss functions.

[0056] The network model loss value L a is updated to the model O, and the update formula is:

[0057]

[0058] where w′ a and b′ a represent the update results of w a and b a , respectively, and l r is a learning rate, respectively represent the partial derivative operations of L a with respect to w a and b a .

[0059] (3d) determining whether a > A is true, if so, obtaining the trained remote sensing image depth estimation network model O * , otherwise, setting a = a + 1, O a = O, and performing step (3b).

[0060] Step 4) obtaining a remote sensing image depth estimation result:

[0061] The test sample set is input into the remote sensing image depth estimation network model O * for forward propagation, and the remote sensing image depth estimation result corresponding to the N-T test samples is obtained.

[0062] The technical effects of the present application will be described below in combination with a simulation experiment:

[0063] 1. Experimental conditions and contents:

[0064] The hardware platform for the simulation experiment is: the processor is 13th Gen Core TM i9-13900Kx32, memory is 125.60GB, GPU is 24G of GeForce RTX4090, the software platform of simulation experiment is Ubuntu20.04 operating system, uses Pytorch deep learning framework, adopts Python language to realize, and the data of depth estimation algorithm simulation is as shown in Figure 3 It is a remote sensing image under the unmanned aerial vehicle aerial scene, and the data contains suburban, city, farmland, river and other scenes.

[0065] The present application and the existing remote sensing image depth estimation method are compared in three indexes of measuring model precision, mean absolute error (Mean Absolute Error, MAE), less than 3 times interval ratio (<3Interval%) and less than 0.6m accuracy ratio (<0.6m accuracy%), and the results are shown in Table 1. Figure 3 The depth estimation algorithm simulation result error graph is as shown in Figure 4 (a) is the simulation result error graph of prior art, Figure 4 (b) is the simulation result error graph of the present application. Figure 4

[0066] 2. Experimental result analysis:

[0067] Table 1 Comparison of evaluation index numerical results of the present application and prior art

[0068] Prior art The invention Mean absolute error 0.160 0.114 Proportion of errors less than 0.3m 91.4％ 95.3％ Proportion of errors less than 0.6m 96.7％ 98.6％

[0069] Referring to Table 1 and Figure 4 Compared with the prior art, the present application has a significant decrease in the mean absolute error (Mean Absolute Error, MAE) index on the OMVS data set, and the less than 3 times interval ratio (<3Interval%) and the less than 0.6m accuracy ratio (<0.6m accuracy%) indexes are improved; from Figure 4 The results of (a) show that there are some areas with large depth estimation error in the simulation results of the prior art, and from Figure 4 (b), the present application can optimize the feature extraction ability of weak texture area, and the overall depth estimation is more accurate.​

Claims

1. A method for depth estimation of a multi-view remote sensing image, characterized in that, Comprising the following steps: (1) Obtain a training sample set and a test sample set: N remote sensing images and their real depth value labels are obtained, and T remote sensing images and their real depth value labels are used to form a training sample set, and the remaining N-T remote sensing images and their real depth value labels are used to form a test sample set, wherein N>30000, (2) Construct a remote sensing image depth estimation network model: Construct a remote sensing image depth estimation network model O comprising a single-view remote sensing image depth estimation branch Mono arranged in parallel for extracting a single-view depth estimation map and a single-view depth estimation confidence map, a multi-view remote sensing image depth estimation branch MVS for extracting a multi-view depth estimation map and a multi-view depth estimation confidence map, and a fusion optimization calculation unit Fusion cascaded with the outputs of Mono and MVS, wherein Mono comprises a feature extraction module, MVS comprises an epipolar attention aggregation module, and the output of the feature extraction module is bridged with the input of the epipolar attention aggregation module; a single-view depth estimation result correction module Correct is loaded between Mono and Fusion, used to correct the single-view depth estimation map; (3) Iteratively train the remote sensing image depth estimation network model O: The remote sensing image depth estimation network model O is iteratively trained through a training sample set, to obtain a trained remote sensing image depth estimation network model O * ; (4) Obtain a remote sensing image depth estimation result: Forward propagate the test sample set as the input of the remote sensing image depth estimation network model O * to obtain the remote sensing image depth estimation results corresponding to N-T test samples.

2. The method of claim 1, wherein, The true depth value label of each remote sensing image in step (1) is obtained by: Projecting a three-dimensional model in an actual scene onto the camera plane corresponding to the current remote sensing image, then traversing each remote sensing image pixel, calculating the pixel ray passing through the camera optical center direction using the camera parameters, finding the intersection point of the ray and the three-dimensional scene, and the distance between the pixel and the intersection point is the depth value corresponding to the pixel position. The depth value corresponding to each pixel is composed into a depth map with the same size as the original remote sensing image, and the depth map is used as the true depth value label of the corresponding remote sensing image.

3. The method of claim 1, wherein, The remote sensing image depth estimation network model O in step (2) comprises: The single-view remote sensing image depth estimation branch Mono further comprises a depth estimation regression head cascaded with the feature extraction module; the feature extraction module comprises a plurality of ViT layers stacked, the depth estimation regression head comprises a convolution layer and a Softmax function layer stacked, and a weighted summation calculation unit is loaded at the output of the Softmax function layer; The single-view depth estimation result correction module Correct comprises a plurality of convolution layers, batch normalization layers and ReLU function layers stacked, and a calculation unit is loaded after the ReLU function layer; The multi-view remote sensing image depth estimation branch MVS further comprises a hierarchical feature extraction module, a cost volume construction calculation unit, a cost volume aggregation module and a depth estimation regression head cascaded with the epipolar attention aggregation module.

4. The method of claim 3, wherein, The implementation step of the iterative training of the remote sensing image depth estimation network model O in step (3) is: (3a) initialize the iteration number as a, the maximum iteration number as A, A > 8000, the weight, bias parameter in the remote sensing image depth estimation network model O in the a-th iteration as w a, b a respectively, and let a = 0; a a a ​​​ (3b) Randomly select P training samples from the training sample set as the input of the remote sensing image depth estimation network model O for forward propagation, and obtain the depth estimation results of P remote sensing images; (3c) using the Smooth L1 loss function, and calculating the network model loss value L through the depth estimation result of each remote sensing image and its corresponding depth real value label a , and then using the gradient descent method to update the weight and bias parameters w a , b a , b a , and obtaining the remote sensing image depth estimation network model O a of this iteration; (3d) judging whether a>A is true, if yes, obtaining the trained remote sensing image depth estimation network model O * , otherwise, letting a=a+1, O a =O, and executing step (3b).

5. The method of claim 4, wherein, The depth estimation result of the p-th remote sensing image in step (3b) is obtained by: (3b1) The feature extraction module in the monocular remote sensing image depth estimation branch Mono performs feature extraction on each training sample; the depth estimation regression head performs depth estimation regression on the extracted monocular feature map to obtain a monocular depth estimation map and a monocular depth estimation confidence map (3b2) a single view depth estimation result correction module Correct corrects the single view depth estimation map The correction is performed to obtain a corrected single view depth estimation result (3b3) The epipolar attention aggregation module in the multi-view remote sensing image depth estimation branch MVS performs epipolar attention aggregation on each training sample and the single-view feature map performs epipolar attention aggregation; hierarchical feature extraction module extracts features from the aggregated feature maps performs feature extraction; a cost volume construction computing unit constructs a cost volume from the extracted feature maps performs cost volume construction; a cost volume aggregation module aggregates the constructed initial cost volume performs cost volume aggregation; a depth estimation regression head estimates depth from the aggregated cost volume performs depth estimation regression to obtain a multi-view depth estimation map and a multi-view depth estimation confidence map (3b4) a single view and multi-view depth estimation result fusion optimization calculation unit Fusion fuses the corrected single view depth estimation result with a confidence map and a multi-view depth estimation map with a multi-view depth estimation confidence map confidence weighted fusion is performed to obtain a depth estimation result map of the pth remote sensing image 6. The method of claim 5, wherein, The epipolar attention aggregation module described in step (3b3) performs epipolar attention aggregation on each training sample and single-view feature map The implementation steps of the epipolar attention aggregation are as follows: Extreme line attention aggregation module extracts single-view feature map Attention features, including query features Q, key features K, and value features V And project K to D different depth values, get the key feature K corresponding to the depth value d d , d∈D, D represents the depth value set; while calculating the attention weight A of Q and K; then summing up K by weighting through A, get the feature fusion result Wherein: K d = Warp(K, d) wherein, W q , W k , W v respectively represent the convolution kernel of Q, K, V, * represents the convolution operation, Warp(·) represents the homographic transformation operation, Softmax(·) represents the weight calculation operation, represents the transpose result of K d , C represents the dimension of K, h represents the number of attention heads, V d represents the value feature corresponding to the depth value d.

7. The method of claim 5, wherein, The monocular depth estimation result correction module Correct described in step (3b2) corrects the monocular depth estimation map The correction is made according to the following formula: Wherein s and o represent the global scale parameter and the offset parameter obtained by passing through the multi-layer convolution layer and the fully connected layer.

8. The method of claim 5, wherein, the depth estimation result map described in step (3b4) The acquisition method is: Wherein ε represents a smoothing parameter.

9. The method of claim 4, wherein, The network model loss value L described in step (3c) a The calculation formula is: wherein, denotes a loss function for the depth estimation task computed using multiple Smooth L1 loss functions, denotes a loss function for the depth estimation task computed using multiple Smooth L1 loss functions, and Smooth L1 loss is computed, denotes a loss function for the depth estimation task computed using multiple Smooth L1 loss functions, denotes a loss function for the depth estimation task computed using multiple Smooth L1 loss functions, Smooth L1 loss is computed, denotes a loss function for the depth estimation task computed using multiple Smooth L1 loss functions, denotes a loss function for the depth estimation task computed using multiple Smooth L1 loss functions, Smooth L1 loss is computed, denotes a real depth value label; denotes a loss function for the confidence constraint, β is a confidence consistency penalty coefficient, | · | denotes an absolute value operation, and Σ denotes a summation operation, denotes a loss function for the gradient matching task, and denote gradient computation in the horizontal and vertical directions of the depth map, respectively; λ conf , λ grad are weight coefficients corresponding to the loss functions.

10. The method of claim 4, wherein, the network model loss value L described in step (3c) a The model O is updated, and the update formula is: Among them, w′ a b′ a They represent w respectively a b a The update result, l r It's the learning rate. L a For w a b a The partial derivative operation.

Images