Remote sensing image multi-view reconstruction method based on deformable self-attention transformer

Abstract

The application discloses a kind of based on deformable self-attention transformer's remote sensing image multi-view reconstruction method, comprising the following steps: S1, for given reference remote sensing image and the preset quantity of source remote sensing image, extract multi-scale feature map;S2, then take the first scale remote sensing image feature and carry out deformable self-attention and feature compression processing;S3, the source remote sensing image feature processed by homographic transformation is under the reference remote sensing image visual angle, and matching cost is formed by calculating cost volume;S4, the probability volume is obtained by the cost volume after regularization network, and the initial depth map is obtained according to the probability volume;S5, the second stage depth hypothesis value range is optimized using the initial depth map again combined with the second stage feature to obtain the second stage depth map, and the final depth map is obtained in turn by analogy;S6, then calculate the loss between the final depth map and true value, and the final depth map is obtained by training the entire network according to the obtained loss value to obtain the three-dimensional structure of remote sensing image.

Description

Technical Field

[0001] This invention belongs to the field of multi-view reconstruction technology, and particularly relates to a method for multi-view reconstruction of remote sensing images based on deformable self-attention Transformer. Background Technology

[0002] With the development and expansion of deep learning, deep learning models have made progress in various fields, including remote sensing. Multi-view reconstruction of remote sensing images, as a major application in remote sensing, has attracted much research from related professionals. Multi-view reconstruction of remote sensing images is mainly divided into two categories: one using traditional non-deep learning methods, and the other combining deep learning methods.

[0003] Traditional methods use handcrafted similarity metrics and engineered regularizations (e.g., normalized cross-correlation and semi-global matching) to compute dense correspondences and recover 3D points. While these methods show good results in ideal Lambertian scenes, they are subject to some common limitations. For example, low-texture scenes, specular reflections, and reflective regions make dense matching difficult to handle, leading to incomplete reconstructions. With the rapid development of deep learning, some methods employ convolutional operations for end-to-end reconstruction of multiple views; however, existing deep learning-based multi-view reconstruction methods suffer from the following limitations:

[0004] (1) The locality of convolutional features organizes information about the global context, which is crucial for robust depth estimation of challenging regions in MVS, such as poor texture, repeating patterns and non-Lambertian surfaces.

[0005] (2) When calculating the matching cost, the features to be compared are simply extracted from each image itself, that is, the potential correspondence between images is not considered.

[0006] (3) Most existing multi-view reconstruction methods are designed to produce satisfactory results for the reconstruction of objects at close range, but there are few methods for multi-view reconstruction of large-scale multi-view remote sensing datasets. Summary of the Invention

[0007] Objective: The purpose of this invention is to overcome the shortcomings of existing technologies and propose a multi-view reconstruction method for remote sensing images based on a deformable self-attention Transformer. Compared with other existing multi-view reconstruction methods, this invention utilizes a deformable self-attention mechanism and a win transformer block to enhance the correlation between views within and between remote sensing images, combining multi-faceted feature information to form a global relational model. Simultaneously, the coarse-to-fine depth map generation process further improves the accuracy and 3D structural integrity of the multi-view reconstruction. Furthermore, unlike previous methods that simply calculate the cross-entropy loss between predicted and true values ​​as the loss function, this multi-view reconstruction method employs a multi-scale metric. Specifically, it not only introduces a pixel-level loss function but also designs a feature-level loss function to address the lack of texture and improve the integrity of the reconstructed point cloud.

[0008] Technical Solution: To achieve the objectives of this invention, this invention proposes a multi-view reconstruction method for remote sensing images based on a deformable self-attention transformer. This method includes the following steps:

[0009] S1. A reference remote sensing image for a given building remote sensing image. and a preset number of building source remote sensing images H represents the height of the remote sensing image, W represents the width of the remote sensing image, and N represents the total number of remote sensing images; the reference remote sensing image R0 and the source remote sensing image S are compared. i Simultaneously, after processing by a joint network based on ResNet and CNN, multi-scale feature maps are obtained. l represents the scale index;

[0010] S2. Obtain multi-scale feature maps Then, the feature set F formed by the feature map F1 at scale l=1 in the first stage is input into the swin transformer for feature self-attention processing, and the features F′ of the reference remote sensing image and the source remote sensing image are obtained respectively. R and The feature F′ obtained by self-attention processing R and The input is fed into the feature compression module to obtain the final reference remote sensing image features F″. R Heyuan remote sensing image features

[0011] S3. For a given reference remote sensing image, the final feature F″ R Heyuan remote sensing image final features Using the camera rotation matrix corresponding to the remote sensing image Translation matrix Intrinsic parameter matrix And the depth assumption value d∈[d min ,d max The source remote sensing image is bent onto a parallel conical section of the reference remote sensing image through a differentiable homography transformation, forming a three-dimensional cost volume. Where, d min d represents the minimum assumed depth value. max This represents the maximum assumed depth value;

[0012] S4. The obtained 3D cost volume COST is regularized by the regularization network 3D-Unet. After regularization, the probability volume P is obtained. The probability volume is normalized by performing a softmax operation along the depth direction. The depth hypothesis value at the maximum probability is selected as the predicted depth value to obtain the initial depth map I0.

[0013] S5. Process the initial depth map to obtain the optimized residual depth value Δd. Process the feature map of the second-stage scale through steps S2 and S3. At this time, the range of the depth assumption value of the homography matrix in S3 becomes d∈[d]. min +Δd,d max +Δd], and after processing S4, a second depth map is obtained; the second depth map is processed again to optimize the range of depth assumption values, and the features of the third stage scale are processed through steps S2, S3, and S4 to obtain a third depth map; the third depth map is processed again to obtain an optimized range of depth assumption values, and the features of the fourth stage scale are processed through steps S2, S3, and S4 to obtain the final depth map.

[0014] S6. Calculate the loss between the depth map and the ground truth. Train the entire network based on the loss value until a limit is reached. The resulting depth map is the final depth map. Calculate the 3D depth value of each pixel in the remote sensing image based on the depth value in the depth map to reconstruct the 3D remote sensing image.

[0015] Furthermore, in step S1, the specific calculation method for multi-scale feature extraction is as follows:

[0016] S11. For a given multi-view remote sensing image of a building, select one view as the reference remote sensing image and the other views as the source remote sensing images to form the reference remote sensing image. and a series of source remote sensing images

[0017] S12, Combine the reference remote sensing image R0 and the source remote sensing image S i The data are fed into a ResNet network for preprocessing, and then a CNN convolutional network is used on top of the ResNet to construct multi-scale remote sensing image features. Both the reference remote sensing image and the source remote sensing image yielded feature maps at four scales. The feature map of the reference remote sensing image is denoted as F. R The features of the source remote sensing image are denoted as Specifically, the feature maps obtained from stages C3 to C5 of ResNet are taken, and these three scales of feature maps are convolved with a 1×1 convolution with a stride of 1 to obtain features at three different scales. The lowest resolution feature map is obtained by convolving with a 3×3 convolution with a stride of 2 in the final C5 stage. The final feature map is then obtained. The scales, from largest to smallest, are as follows:

[0018] Furthermore, in step S2, the specific calculation methods for self-attention processing and feature compression of the features are as follows:

[0019] S21. Regarding the reference remote sensing image features F obtained above... R Heyuan remote sensing image features Take the same scale feature F from the reference remote sensing image and the source remote sensing image. l R and Where l∈{1,2,3,4}, both the reference remote sensing image features and the source remote sensing image features are features, and both features are subsequently processed in the same way, with the reference remote sensing image features and the source remote sensing image features being uniformly represented by F. l This means that, firstly, the features at scale l=1 in the first stage are extracted and input into the Swin transformer for feature self-attention. The specific feature self-attention includes the following steps:

[0020] (1) For the input remote sensing image features F l By using linear embedding, the number of channels is changed from 256 to C, at which point the initial input features are obtained, denoted as g. u-1 ;

[0021] (2) The obtained g u-1 The input is processed in a window-based transformer, g u-1 After normalization and window-based deformable self-attention processing, it is added to itself to obtain... The result After processing by a normalization layer and a multilayer perceptron, the result is added to itself, ultimately yielding a window-based output g. u The formula is as follows:

[0022]

[0023]

[0024] Where LN represents the normalization layer, MLP represents the multilayer perceptron; W-MSDA represents window-based deformable self-attention, which performs deformable self-attention calculation according to formula (2) within the window; formula (2) is the formula for the deformable self-attention mechanism, z q Represents the query vector; p q Indicates the position of the query vector; x is the input feature; M is the number of attention heads; W k and W v These represent the weights of the key and value, respectively; C is the number of channels; Δp mqk It represents the relative position offset in positional encoding; φ represents the softmax function; W m This indicates that a linear transformation is performed on the result after attention is applied to the value to obtain the output results of different heads;

[0025] (3) The obtained g u The input is processed in a displacement window-based transformer, g u After normalization and deformational self-attention processing based on displacement windows, the result is added to itself. Will After processing by a normalization layer and a multilayer perceptron, the result is added to itself, ultimately yielding the output g based on a displacement window. u+1 The formula is as follows:

[0026]

[0027] Where LN represents the normalization layer, MLP represents the multilayer perceptron, and SW-MSDA represents the deformable self-attention processing based on the displacement window. Here, MSDA is calculated using formula (2).

[0028] (4) Set the initial input g u-1 After processing by the spatial interaction module, it interacts with g. u+1 After addition, the sum is then processed by window-based self-attention in step (2) above to obtain the output g. u+2 The final output g is obtained by the self-attention operation based on the displacement window in step (3). u+3 g u+3 The output feature F' is obtained by taking the input reference remote sensing image features and source remote sensing image features through steps (1)(2)(3)(4) and obtaining the same representation.

[0029] The spatial interaction module operates as follows: for a given initial input features... First, it is reshaped using reshape. Size, all dimensions become 2 times the original. n-1Size; The reshaped features are input into a 3×3 dilated convolutional layer with a dilation rate of 2. Average pooling is used to calculate the statistics of the feature map in both the vertical and horizontal directions along channel c. The vertical and horizontal statistics are then matrix-multiplied, followed by a 1×1 convolution and then combined with g. l+1 The feature F1' generated by the swin transformer is obtained by adding elements together. The calculation formula is as follows:

[0030]

[0031] Among them, c i c represents the horizontal index along the channel direction. j This represents the vertical index along the channel direction, where w is the image width and h is the image height. This indicates the characteristics of the channel space in the horizontal direction. This represents the characteristics in the vertical direction of the channel space. Represents matrix multiplication, representing Element-level addition, It is a function with batch normalization and GELU activation function;

[0032] S22. Perform feature compression on the generated feature F' by dividing the operation into two branches. One branch performs three convolution operations on F' to obtain the feature compression. Where c is the channel size after reshaping, and its calculation method is expressed as:

[0033] F1'=conv(convD3(conv(F'))) (5)

[0034] Where conv is a 1×1 convolution, and convD3 is a 3×3 dilated convolution with a dilation factor of 2.

[0035] Another branch involves using soft pooling to exponentially weight the pixels in the pooling kernel. For each pixel in a specific kernel neighborhood v, the soft-pooled pixel is then fed into a convolutional layer to obtain the output. The calculation formula is as follows:

[0036]

[0037] Where i represents the horizontal index of the neighborhood, j represents the vertical index of the neighborhood, v is the scale of the kernel neighborhood, e represents the exponential operation, and conv is the convolution operation.

[0038] The features obtained from the two branches are added together proportionally to obtain the final output feature F”. F” is represented as:

[0039]

[0040] in, This represents element-wise addition;

[0041] For the input reference remote sensing image features F R Heyuan remote sensing image features The characteristics of the unified representation in the first stage F l After processing through steps S21-S22, the final output feature F” is obtained, where F” is the reference remote sensing image feature F″. R Heyuan remote sensing image features A unified representation.

[0042] Furthermore, the specific method for the differentiable homography transformation in step S3 is as follows:

[0043] S31. For a given reference remote sensing image, the final feature F″ R Heyuan remote sensing image final features Add the corresponding camera rotation matrix Translation matrix Intrinsic parameter matrix And the depth assumption value d∈[d min ,d max ], where d min d represents the minimum assumed depth value. max The maximum depth hypothesis value is represented by multiplying the source remote sensing image features by the differentiable homography transformation matrix to bend the source remote sensing image features onto a parallel conical cross-section of the reference remote sensing image features. The matching cost is then calculated using the following formula:

[0044]

[0045] Where i represents the index of the camera parameter in the source remote sensing image, 0 subscript represents the index of the camera parameter in the reference remote sensing image, I represents the identity matrix, and r i and t i Let n and n0 represent the rotation and translation matrices, respectively, which are the extrinsic parameters of the camera. Let n0 represent the normal vector of the reference camera, and T represent the transpose of the matrix.

[0046] Given a depth hypothesis value d, multiply the source remote sensing image features by the given homography transformation matrix H mentioned above. i (d) Obtain the source remote sensing image features at depth d from the perspective of the reference remote sensing image.

[0047] S32. After multiplying the above source remote sensing image features by the homography matrix transformation, subtracting the reference image in the depth direction and squaring the result, the final three-dimensional cost volume is formed by aggregating multiple views. The calculation method is as follows:

[0048]

[0049] Where N is the total number of views, D is the total number of depth assumptions, and d0 is the initial depth assumption value. F″ represents the feature of the i-th source remote sensing image at depth d. R Represents the features of the reference remote sensing image, |||| 2 This indicates the square calculation.

[0050] Furthermore, the specific method for cost volume regularization in step S4 is as follows:

[0051] S41. After obtaining the 3D cost volume COST, the cost volume is regularized. Specifically, the 3D cost volume is input into 3D-UNet for processing. The specific structure of the network is as follows:

[0052] In the first convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 1 for normalization.

[0053] In the second convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 2 for normalization.

[0054] In the third convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 3 for normalization.

[0055] In the fourth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 4 for normalization.

[0056] In the fifth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 5 for normalization.

[0057] In the sixth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 6 for normalization.

[0058] Deconvolutional layers are used in layers 7, 8, and 9, with a kernel size of 3×3×3, a stride of 2, padding of 1, and output padding of 1. A batch normalization layer is then added after these layers for normalization.

[0059] S42. Use the softmax operation to perform probability normalization to obtain multiple probability values, forming a probability volume. Take the depth hypothesis value with the maximum probability value of the probability volume as the prediction value, and finally obtain the initial depth map I0.

[0060] Furthermore, the specific method for step S5 is as follows:

[0061] S51. After obtaining the initial depth map, perform convolution processing on the initial depth map to obtain the residual depth hypothesis value Δd. After concatenating the reference image with the disparity value corresponding to each pixel in the reference image, process it through three convolutional layers. The specific processing method is as follows:

[0062] Δd=conv3(conv2(conv1(cat(R0,d)))) (10)

[0063] Where cat represents the connection operation, conv1, conv2, and conv3 represent three convolutional layers, R0 is the reference image, and d is the depth value in the depth map;

[0064] Repeat step S2 on the feature map at scale l=2 in the second stage to obtain the source remote sensing image features and reference remote sensing image features after feature self-attention in the second stage. After step S3, the depth hypothesis value range of the homography matrix in S3 becomes d∈[d min +Δd,d max +Δd], after the S4 step operation, the second depth map I1 is obtained;

[0065] S52. The second depth map I1 is calculated using formula (10) in S51 to obtain the range of depth residual value Δd to optimize the depth assumption value. The features of the third stage scale are processed through steps S2 and S3. Here, the range of depth assumption value of the homography matrix changes in step S3. The third depth map I2 is obtained through step S4.

[0066] S53. The third depth map I2 is calculated using formula (10) in S51 to obtain the depth residual value Δd again to optimize the range of the depth assumption value. The features of the fourth stage scale are processed through steps S2 and S3. Here, the range of the depth assumption value of the homography matrix changes in step S3. Then, the final refined depth map I3 is obtained through step S4.

[0067] Furthermore, step S6 reconstructs the three-dimensional remote sensing image, and the specific method is as follows:

[0068] S61. After obtaining the refined depth map, calculate the loss between the depth map and the ground truth value. Train the entire network based on the obtained loss value until the training limit is reached. The loss function is calculated as follows:

[0069] (1) First, calculate the cross-entropy loss between the final generated depth map and the ground truth, and construct the pixel-level loss L. xs For all valid pixel sets {p} belonging to reliable pixels v For pixel p in}, perform the following calculations:

[0070]

[0071] Among them, G d (p) and P d (p) represent the true and predicted values ​​of pixel p at the depth hypothesis d, respectively. min Assuming a minimum depth, d max This is the assumed maximum depth value;

[0072] (2) Input the reference remote sensing image into the VGG16 network to extract features, and construct the feature-level loss L. tz The features from layers 8, 15, and 22 are extracted for calculation, and the specific calculation is expressed as follows:

[0073]

[0074] in, The i-th layer is obtained by passing through the convolutional layers in VGG16 to obtain the features of the reference remote sensing image. The source remote sensing image features are obtained from the convolutional layer of the i-th layer of VGG16 in the n-th view, where N is the total number of views, k is the index of the weight value β, and || represents the absolute value operation.

[0075] (3) Adding the two loss functions together yields the final, complete loss function, expressed as follows:

[0076] L = L xs +L tz (13)

[0077] Among them, L xs L represents the pixel-level loss in (1). tz This represents the feature-level loss in (2);

[0078] S62. The depth map obtained after training stops is the final depth map. The depth map is also known as the disparity map. After obtaining the depth map of the reference remote sensing image, the depth value of each pixel in the reference remote sensing image is calculated using the following formula to obtain the three-dimensional structure of the image, that is, the three-dimensional remote sensing image is reconstructed.

[0079] z=(b×f) / d (14)

[0080] Where b is the baseline, f is the focal length, d is the depth value in the depth map, and z is the depth value in three-dimensional space.

[0081] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0082] First, the deformable self-attention mechanism is introduced into multi-view reconstruction and then applied to the field of multi-view reconstruction of remote sensing images to solve the problem of three-dimensional reconstruction of remote sensing images, thereby improving the accuracy and completeness of three-dimensional reconstruction of remote sensing images.

[0083] Second, for the input remote sensing images, a convolutional neural network without feature pyramid (FPN) is used on the basis of ResNet to extract multi-scale feature maps, which takes into account multi-scale image information and facilitates subsequent regression of fine depth maps from coarse to fine.

[0084] Third, in order to better aggregate information between and within views, remote sensing image features are input into the transformer block for further feature aggregation and feature self-attention. At the same time, a spatial aggregation module is added to improve the global modeling capability of the transformer, and a feature compression module is added to obtain more detailed feature information.

[0085] Fourth, unlike other networks that only calculate cross-entropy loss between the obtained depth map and the ground truth, this network considers both cross-entropy loss and feature-level loss by passing the original image through a pre-trained VGG network. The entire loss function is a sum of pixel-level loss and feature-level loss. Attached Figure Description

[0086] Figure 1 This is the network structure diagram of the present invention. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0088] Please see Figure 1 This invention discloses a method for multi-view reconstruction of remote sensing images based on a deformable self-attention transformer, comprising the following steps:

[0089] S1. For the given remote sensing image dataset WHU Dataset, select the image with ID 1 as the reference remote sensing image, and select the corresponding source remote sensing images with IDs 0, 2, 3, and 4 as the source remote sensing images. Given the reference remote sensing image... and a series of source remote sensing images H is the height of the remote sensing image, W is the width of the remote sensing image, and N is the total number of views; the reference remote sensing image R0 and the source remote sensing image S are compared.i Simultaneously, after processing by a joint network based on ResNet and CNN, multi-scale feature maps are obtained. l represents the scale index;

[0090] S11. First, for the given remote sensing image dataset WHU Dataset, select the image with ID 1 as the reference remote sensing image, and select the corresponding IDs 0, 2, 3, and 4 as the source remote sensing images. This forms the reference remote sensing image. and a series of source remote sensing images

[0091] S12, Combine the reference remote sensing image R0 and the source remote sensing image S i The data are fed into a ResNet network for preprocessing, and then a CNN convolutional network is used on top of ResNet to construct multi-scale remote sensing image features. Both the reference remote sensing image and the source remote sensing image will yield feature maps at four scales. The feature map of the reference remote sensing image is denoted as F. R The features of the source remote sensing image are denoted as The specific approach is as follows: Take the feature maps obtained from stages C3 to C5 of ResNet, and perform a 1×1 convolution with a stride of 1 on these three scales to obtain features at three different scales. The lowest resolution feature map is obtained by performing a 3×3 convolution with a stride of 2 in the final C5 stage. The resulting feature map... The scales, from largest to smallest, are as follows:

[0092]

[0093] S2. Obtain multi-scale feature maps Then, features F at the same scale are taken from both the reference remote sensing image and the source remote sensing image. l R and Where l∈{1,2,3,4}, since both the reference remote sensing image features and the source remote sensing image features are features, and both features are subsequently processed in the same way, we will use F to represent both the reference remote sensing image features and the source remote sensing image features. l This means that the feature set F formed by the feature map F1 at scale l=1 in the first stage is input into the swin transformer for feature self-attention, and the features F′ of the reference remote sensing image and the source remote sensing image are obtained respectively. R and The features F′ obtained through self-attention R and The input is fed into the feature compression module to obtain the final reference remote sensing image features F″. R Heyuan remote sensing image features

[0094] S21. Regarding the reference remote sensing image features F obtained above... R Heyuan remote sensing image features Take the same scale feature F from the reference remote sensing image and the source remote sensing image. l R and Where l∈{1,2,3,4}, both the reference remote sensing image features and the source remote sensing image features are features, and both features are subsequently processed in the same way, with the reference remote sensing image features and the source remote sensing image features being uniformly represented by F. l This means that, firstly, the features at scale l=1 in the first stage are extracted and input into the Swin transformer for feature self-attention. The specific feature self-attention includes the following steps:

[0095] (1) For the input remote sensing image features F l By using linear embedding, the number of channels is changed from 256 to C, at which point the initial input features are obtained, denoted as g. u-1 ;

[0096] (2) The obtained g u-1 The input is processed in a window-based transformer, g u-1 After passing through a normalization layer and window-based deformable self-attention, the result is added to itself. The result After processing by a normalization layer and a multilayer perceptron, the result is added to itself, ultimately yielding a window-based output g. u The formula is as follows:

[0097]

[0098]

[0099] Where LN represents the normalization layer, MLP represents the multilayer perceptron; W-MSDA represents window-based deformable self-attention, which performs deformable self-attention calculation according to formula (2) within the window; formula (2) is the formula for the deformable self-attention mechanism, z q Represents the query vector; p q Indicates the position of the query vector; x is the input feature; M is the number of attention heads; W k and W v These represent the weights of the key and value, respectively; C is the number of channels; Δp mqk It represents the relative position offset in positional encoding; φ represents the softmax function; W m This indicates that a linear transformation is performed on the result after attention is applied to the value to obtain the output results of different heads;

[0100] (3) The obtained g u The input is processed in a displacement window-based transformer, g u After normalization and displacement window-based deformable self-attention, it is added to itself to obtain... Will After processing by a normalization layer and a multilayer perceptron, the result is added to itself, ultimately yielding the output g based on a displacement window. u+1 The formula is as follows:

[0101]

[0102] Where LN represents the normalized layer, MLP represents the multilayer perceptron, and SW-MSDA represents deformable self-attention based on displacement window. Here, MSDA is still calculated using formula (2).

[0103] (4) Set the initial input g u-1 After processing by the spatial interaction module, it interacts with g. u+1 After addition, the sum is then processed through window-based self-attention in step (2) above to obtain the output g. u+2 The final output g is obtained by the self-attention operation based on the displacement window in step (3). u+3 g u+3 The output feature F' is the same representation obtained after the input reference remote sensing image features and source remote sensing image features are processed through steps (1)(2)(3)(4);

[0104] The spatial interaction module operates as follows: for a given initial input features... First, it is reshaped using reshape. Size, all dimensions become 2 times the original. n-1 Size; The reshaped features are input into a 3×3 dilated convolutional layer with a dilation rate of 2, and then average pooling is used to calculate the statistics of the feature map in both the vertical and horizontal directions along channel c; the statistics in the vertical and horizontal directions are matrix multiplied, and then convolved with g after a 1×1 convolution. l+1 The elements are added together to obtain the feature F1' generated by the swin transformer. The calculation formula is as follows:

[0105]

[0106] Among them, c i c represents the horizontal index along the channel direction. j This represents the vertical index along the channel direction, where w is the image width and h is the image height. This indicates the characteristics of the channel space in the horizontal direction. This represents the characteristics in the vertical direction of the channel space. Represents matrix multiplication, representing Element-level addition, It is a function with batch normalization and GELU activation function;

[0107] S22. Perform feature compression on the generated feature F' by dividing the operation into two branches. One branch performs three convolution operations on F' to obtain the feature compression. Where c is the channel size after reshaping, and the calculation method is expressed as:

[0108] F'1=conv(convD3(conv(F'))) (5)

[0109] Wherein, conv is a 1×1 convolution, and convD3 is a 3×3 dilated convolution with a dilation factor of 2.

[0110] Another branch involves using soft pooling to exponentially weight the pixels in the pooling kernel. For each pixel in a specific kernel neighborhood v, the soft-pooled pixel is then fed into a convolutional layer to obtain the output. The calculation formula is as follows:

[0111]

[0112] Where i represents the horizontal index of the neighborhood, j represents the vertical index of the neighborhood, v is the scale of the kernel neighborhood, e represents the exponential operation, and conv is the convolution operation.

[0113] Therefore, by adding the features obtained from the two branches in equal proportions and merging them, we obtain the final output feature F”. F” can be expressed as:

[0114]

[0115] in, This represents element-wise addition.

[0116] Thus, for the input reference remote sensing image features F R Heyuan remote sensing image features The characteristics of the unified representation in the first stage F l Where l∈{1,2,3,4}, after processing through steps S21-S22, the final output feature F” is obtained, where F” is the reference remote sensing image feature F″. R Heyuan remote sensing image features A unified representation.

[0117] S3. For a given reference remote sensing image, the final feature F″ R Heyuan remote sensing image final features Add the corresponding camera rotation matrix Translation matrix Intrinsic parameter matrix And the depth assumption value d∈[d min ,d max ], where d min d represents the minimum assumed depth value. max Representing the maximum depth hypothesis, the source remote sensing image is bent onto a parallel conical section of the reference remote sensing image through a differentiable homography transformation. This allows for the calculation of the matching cost, forming a three-dimensional cost volume.

[0118] S31. For a given reference remote sensing image, the final feature F″ R Heyuan remote sensing image final features Add the corresponding camera rotation matrix Translation matrix Intrinsic parameter matrix And the depth assumption value d∈[d min ,d max ], where d min d represents the minimum assumed depth value. max The maximum depth hypothesis value is represented by multiplying the source remote sensing image features by the differentiable homography matrix. This bends the source remote sensing image features onto a parallel conical cross-section of the reference remote sensing image features, which is necessary for the subsequent matching cost calculation. The formula for calculating the homography matrix transformation matrix is ​​as follows:

[0119]

[0120] Where i represents the index of the camera parameter in the source remote sensing image, 0 subscript represents the index of the camera parameter in the reference remote sensing image, I represents the identity matrix, and r i and t i denoted by and respectively, indicating that the rotation and translation matrices are the extrinsic parameters of the camera, and n represents the normal vector of the reference camera;

[0121] Given a depth hypothesis value d, multiply the source remote sensing image features by the given homography transformation matrix H mentioned above. i (d) can then be used to obtain the source remote sensing image features at depth d from the perspective of the reference remote sensing image.

[0122] S32. After multiplying the above source remote sensing image features by the homography transformation matrix, the depth direction is subtracted from the reference image and the square is calculated. Then, multiple views are aggregated to form the final three-dimensional cost volume. The calculation method is as follows:

[0123]

[0124] Where N is the total number of views, D is the total number of depth assumptions, and d0 is the initial depth assumption value. F″ represents the feature of the i-th source remote sensing image at depth d. R Represents the features of the reference remote sensing image, |||| 2 This indicates the square calculation.

[0125] S4. The obtained 3D cost volume COST regularization network 3D-Unet is subjected to cost volume regularization. After regularization, the probability volume P is obtained. The probability volume is normalized by performing softmax operation along the depth direction. The depth hypothesis value at the maximum probability is selected as the predicted depth value, thus obtaining the initial depth map I0.

[0126] S41. After obtaining the 3D cost volume (COST) as described above, it is necessary to regularize the cost volume to reduce the influence of noise. Specifically, the 3D cost volume is input into a 3D-UNet for processing. The network structure is as follows:

[0127] In the first convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 1 for normalization.

[0128] In the second convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 2 for normalization.

[0129] In the third convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 3 for normalization.

[0130] In the fourth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 4 for normalization.

[0131] In the fifth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 5 for normalization.

[0132] In the sixth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 6 for normalization.

[0133] Deconvolutional layers are used in layers 7, 8, and 9, with a kernel size of 3×3×3, a stride of 2, padding of 1, and output padding of 1. A batch normalization layer is then added after these layers for normalization.

[0134] S42. Finally, the probability is normalized using the softmax operation to obtain multiple probability values, forming a probability volume. The depth hypothesis value with the highest probability value of the probability volume is taken as the prediction value, and finally the initial depth map I0 is obtained.

[0135] S5. After obtaining the initial depth map, process it to obtain the optimized residual depth value Δd. Then process the feature map of the second-stage scale through steps S2 and S3. At this point, the range of the depth hypothesis value of the homography matrix in S3 becomes d∈[d]. min +Δd,d max +Δd], and after processing S4, the second depth map is obtained; then the second depth map is processed again to optimize the range of depth hypothesis values, and the features of the third stage scale are processed through steps S2, S3, and S4 to obtain the third depth map; then the third depth map is processed again to obtain the optimized range of depth hypothesis values, and the features of the fourth stage scale are processed through steps S2, S3, and S4 to obtain the final depth map.

[0136] S51. After obtaining the initial depth map, perform convolution processing on the initial depth map to obtain the residual depth hypothesis value Δd. After concatenating the reference image with the disparity value corresponding to each pixel in the reference image, process it through three convolutional layers. The specific processing method is as follows:

[0137] Δd=conv3(conv2(conv1(cat(R0,d)))) (10)

[0138] Where cat represents the connection operation, conv1, conv2, and conv3 represent three convolutional layers, R0 is the reference image, and d is the depth value in the depth map;

[0139] Next, the feature map at scale l=2 in the second stage is repeated using step S2 to obtain the source remote sensing image features and reference remote sensing image features after feature self-attention in the second stage. Then, step S3 is performed, at which point the range of the depth hypothesis value of the homography matrix in S3 becomes d∈[d min +Δd,d max +Δd], and then the second depth map I1 is obtained through step S4;

[0140] S52. Next, the second depth map I1 is calculated using formula (10) in S51 to obtain the depth residual value Δd again to optimize the range of the depth assumption value. Then, the features of the third stage scale are processed through steps S2 and S3. Here, the range of the depth assumption value of the homography matrix changes in step S3. Then, the third depth map I2 is obtained through step S4.

[0141] S53. Next, the third depth map I2 is calculated using formula (10) in S51 to obtain the depth residual value Δd again to optimize the range of the depth assumption value. Then, the features of the fourth stage scale are processed through steps S2 and S3. Here, the range of the depth assumption value of the homography matrix changes in step S3. Then, the final refined depth map I3 is obtained through step S4.

[0142] S6. After obtaining the refined depth map, we calculate the loss between the depth map and the ground truth. We then train the entire network based on the loss value until a certain number of iterations are reached. The resulting depth map is the final depth map. Based on the depth values ​​in the depth map, we calculate the 3D depth value of each pixel in the remote sensing image to reconstruct the 3D remote sensing image. This is how the 3D remote sensing image is reconstructed.

[0143] S61. After obtaining the refined depth map, we calculate the loss between the depth map and the ground truth. We then train the entire network based on the obtained loss value. However, if the loss value decreases, we will continue training until the training limit is reached. The loss function can be calculated in the following way:

[0144] (1) First, calculate the cross-entropy loss between the final generated depth map and the ground truth, and construct the pixel-level loss L. xs For all valid pixel sets {p} belonging to reliable pixels v For pixel p in}, perform the following calculations:

[0145]

[0146] Among them, G d (p) and P d (p) represent the true and predicted values ​​of pixel P at the depth hypothesis d, respectively. min Assuming a minimum depth, d max This is the assumed maximum depth value;

[0147] (2) Next, the reference remote sensing image is input into the VGG16 network to extract features, and a feature-level loss L is constructed. tz The features from layers 8, 15, and 22 are extracted for calculation. The specific calculation can be expressed as follows:

[0148]

[0149] in The i-th layer is obtained by passing through the convolutional layers in VGG16 to obtain the features of the reference remote sensing image. Let N be the source remote sensing image features obtained from the convolutional layer of the i-th layer of VGG16 in the n-th view, where N is the total number of views, k is the index of the weight value β, and || represents the absolute value operation.

[0150] (3) Finally, by adding the two loss functions above, we can obtain the final complete loss function, which can be expressed as:

[0151] L = L xs +L tz (13)

[0152] Among them, L xs L represents the pixel-level loss in (1). tz The feature level loss in (2) is represented.

[0153] S62. After training stops, the resulting depth map is the final depth map, also known as the disparity map. After obtaining the depth map of the reference remote sensing image, we can easily calculate the depth value of each pixel in the reference remote sensing image, thus obtaining the three-dimensional structure of the image. This reconstructs the three-dimensional remote sensing image. The formula can be expressed as:

[0154] z=(b×f) / d (14)

[0155] Where b is the baseline, f is the focal length, d is the depth value in the depth map, and z is the depth value in 3D space.

[0156] The dataset of this invention will be further described below:

[0157] Experimental dataset:

[0158] The experiments in this invention used the WHU dataset remote sensing dataset.

[0159] The WHU dataset is a publicly available remote sensing dataset released in 2020. Each image in the dataset is a complete and accurate 3D urban scene simulation generated from a collection of real multi-view aerial images using software and careful manual editing. The dataset includes thousands of simulated images covering 6.7 × 2.2 square kilometers, along with ground truth depth and camera parameters for multi-view images, and disparity maps for corrected epipolar images. The dataset contains six regions: Region 0: The complete dataset consists of 1776 virtual aerial images, each 5376 × 5376 pixels. To facilitate machine learning methods, regions 1, 4, 5, and 6 are assigned to a training set of 261 images each. Regions 2 and 3, totaling 93 images, serve as the test set. Region 1 is a flat suburb with tall and short factory buildings. Region 2 includes trees, roads, buildings, and open spaces. Region 3 is a residential area with mixed tall and short buildings. Regions 4 and 5 are densely built-up city centers with complex roof structures. Region 6 is a mountainous area covered by farmland and forest. A total of 261 virtual images from Regions 1 / 4 / 5 / 6 were used as the training set, and 93 images from Region 2 were used as the training set in both the training and testing regions. The images were also cropped to a scale of 768×384 pixels for a single GPU.

[0160] The foregoing has provided a detailed description of a remote sensing image multi-view reconstruction method based on a deformable self-attention transformer, as provided by the embodiments of the present invention. For those skilled in the art, there may be changes in the specific implementation methods and application scope based on the ideas of the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for multi-view reconstruction of remote sensing images based on a deformable self-attention transformer, characterized in that, The method includes the following steps: S1. A reference remote sensing image for a given building remote sensing image. and a preset number of building source remote sensing images , For the height of the remote sensing image, Width of the remote sensing image Total number of remote sensing images; reference remote sensing images Heyuan remote sensing images Simultaneously, after processing by a joint network based on ResNet and CNN, multi-scale feature maps are obtained. , For scale indexing; S2. Obtain multi-scale feature maps After that, take the first stage. Scale feature map The formed feature group The features of the reference remote sensing image and the source remote sensing image are respectively input into a swin transformer for feature self-attention processing to obtain the features of the reference remote sensing image and the source remote sensing image. and Features obtained through self-attention processing and The input is fed into the feature compression module to obtain the final reference remote sensing image features. Heyuan remote sensing image features ; S3. For a given reference remote sensing image, the final features Heyuan remote sensing image final features Using the camera rotation matrix corresponding to the remote sensing image Translation matrix Internal parameter matrix and depth assumptions The source remote sensing image is bent onto a parallel conical section of the reference remote sensing image through a differentiable homography transformation, forming a three-dimensional cost volume. ,in, This represents the minimum assumed depth value. This represents the maximum assumed depth value; S4. The obtained 3D cost volume COST is regularized using the 3D-Unet regularization network to obtain the probability volume P. A softmax operation is then performed on the probability volume along the depth direction to normalize the probability. The depth hypothesis value at the point of maximum probability is selected as the predicted depth value to obtain the initial depth map. ; S5. Process the initial depth map to obtain optimized residual depth values. The feature map at the second-stage scale is processed through steps S2 and S3. At this point, the range of the depth assumption value of the homography matrix in S3 becomes... After processing S4, a second depth map is obtained. The second depth map is then processed to further optimize the range of depth hypothesis values. The features at the third stage scale are then processed through steps S2, S3, and S4 to obtain a third depth map. The third depth map is then processed again to obtain an optimized range of depth hypothesis values. The features at the fourth stage scale are then processed through steps S2, S3, and S4 to obtain the final depth map. S6. Calculate the loss between the depth map and the ground truth. Train the entire network based on the loss value until a limit is reached. The resulting depth map is the final depth map. Calculate the 3D depth value of each pixel in the remote sensing image based on the depth value in the depth map to reconstruct the 3D remote sensing image.

2. The remote sensing image multi-view reconstruction method based on deformable self-attention transformer according to claim 1, characterized in that, In step S1, the specific calculation method for multi-scale feature extraction is as follows: S11. For a given multi-view remote sensing image of a building, select one view as the reference remote sensing image and the other views as the source remote sensing images to form the reference remote sensing image. and a series of source remote sensing images ; S12, Refer to remote sensing image Heyuan remote sensing images The data are fed into a ResNet network for preprocessing, and then a CNN convolutional network is used on top of the ResNet to construct multi-scale remote sensing image features. Both the reference remote sensing image and the source remote sensing image yielded feature maps at four scales. The features of the reference remote sensing image are denoted as... The features of the source remote sensing image are denoted as Specifically, the feature maps obtained from stages C3 to C5 of ResNet are taken, and these three scale feature maps are convolved with a 1×1 convolution with a stride of 1 to obtain features at three different scales. The lowest resolution feature map is obtained by convolving with a 3×3 convolution with a stride of 2 in the final C5 stage. The final feature map is then obtained. The scales, from largest to smallest, are as follows: , , , .

3. The remote sensing image multi-view reconstruction method based on deformable self-attention transformer according to claim 2, characterized in that, In step S2, the specific calculation method for self-attention processing and feature compression of the features is as follows: S21. Regarding the reference remote sensing image features obtained above... Heyuan remote sensing image features Features at the same scale are taken from the reference remote sensing image and the source remote sensing image. and ,in, Both the reference remote sensing image features and the source remote sensing image features are features, and both features are subsequently processed in the same way, using the same syntax to represent them. This indicates that the first stage will be extracted first. The scale features are input into the Swin transformer for feature self-attention, which includes the following steps: (1) For the input remote sensing image features By using linear embedding, the number of channels is changed from 256 to C, resulting in the initial input features, denoted as C. ; (2) The obtained The input is processed in a window-based transformer. After normalization and window-based deformable self-attention processing, it is added to itself to obtain... , will get After processing by a normalization layer and a multilayer perceptron, the result is added to itself, ultimately yielding a window-based output. The formula is as follows: (1) (2) Where LN represents the normalization layer, MLP represents the multilayer perceptron; W-MSDA represents window-based deformable self-attention, in which deformable self-attention calculation of formula (2) is performed within the window; formula (2) is the formula for the deformable self-attention mechanism. Represents the query vector; Indicates the position of the query vector; Features for input; Pay attention to the number of heads; and These are the weights of the key and the value, respectively. Number of channels; It is the relative position offset in position encoding; This represents the softmax function; This indicates that a linear transformation is performed on the result after attention is applied to the value to obtain the output results of different heads; (3) The obtained The input is processed in a transformer based on a displacement window. After normalization and deformational self-attention processing based on displacement windows, the result is added to itself. ,Will After processing by a normalization layer and a multilayer perceptron, the result is added to itself, ultimately yielding an output based on a displacement window. The formula is as follows: (3) Where LN represents the normalization layer, MLP represents the multilayer perceptron, and SW-MSDA represents the deformable self-attention processing based on the displacement window. Here, MSDA is calculated using formula (2). (4) Initial input After being processed by the spatial interaction module, and After addition, the sum is then processed by window-based self-attention in step (2) above to obtain the output. The final output is obtained by performing a self-attention operation based on a displacement window in step (3). , The output features obtained after steps (1), (2), (3), and (4) are the same representation of the input reference remote sensing image features and the source remote sensing image features. ; The spatial interaction module operates as follows: for a given initial input features... First, it is reshaped into... Size, all dimensions become 2 times the original. n-1 Size; The reshaped features are input into a 3×3 dilated convolutional layer with a dilation rate of 2. Average pooling is used to calculate the statistics of the feature map in both the vertical and horizontal directions along channel c. The vertical and horizontal statistics are then matrix-multiplied, followed by a 1×1 convolution and then... Adding elements together yields the features generated by the swin transformer. The calculation formula is: (4) in, Indicates the horizontal index in the direction of the channel. This represents the vertical index along the channel direction, where w is the image width and h is the image height. This indicates the characteristics of the channel space in the horizontal direction. This represents the characteristics in the vertical direction of the channel space. Represents matrix multiplication, representing Element-level addition, It is a function with batch normalization and GELU activation function; S22, The features generated above Feature compression is performed in two branches. One branch will... Obtained through three convolution operations ,in, The channel size after reshaping is calculated as follows: (5) Where conv is a 1×1 convolution, and convD3 is a 3×3 dilated convolution with a dilation factor of 2. Another branch is to use features Pixels in the pooling kernel are activated exponentially using soft pooling. For each pixel in a specific kernel neighborhood v, after soft pooling, the input is fed into the convolutional layer to obtain the output. The calculation formula is as follows: (6) Where i represents the horizontal index of the neighborhood, j represents the vertical index of the neighborhood, v is the scale of the kernel neighborhood, e represents the exponential operation, and conv is the convolution operation. The features obtained from the two branches are added together proportionally to obtain the final output features. , Represented as: (7) in, This represents element-wise addition; For input reference remote sensing image features Heyuan remote sensing image features Features of the unified representation in stage l The final output features are obtained after processing steps S21-S22. ,in, For reference remote sensing image features Heyuan remote sensing image features A unified representation.

4. The remote sensing image multi-view reconstruction method based on deformable self-attention transformer according to claim 1, characterized in that, The specific method for the differentiable homography transformation in step S3 is as follows: S31. For a given reference remote sensing image, the final features Heyuan remote sensing image final features Add the corresponding camera rotation matrix Translation matrix Internal parameter matrix and depth assumptions ,in, This represents the minimum assumed depth value. The maximum depth hypothesis value is represented by multiplying the source remote sensing image features by the differentiable homography transformation matrix to bend the source remote sensing image features onto a parallel conical cross-section of the reference remote sensing image features. The matching cost is then calculated using the following formula: (8) Where i represents the index of the camera parameters in the source remote sensing image, and the 0 subscript represents the index of the camera parameters in the reference remote sensing image. Represents the identity matrix. and Let n and n0 represent the rotation and translation matrices, respectively, which are the extrinsic parameters of the camera. Let n0 represent the normal vector of the reference camera, and T represent the transpose of the matrix. Given a depth hypothesis value d, multiply the source remote sensing image features by the given homography transformation matrix mentioned above. To obtain the source remote sensing image features at depth d from the perspective of the reference remote sensing image. ; S32. After multiplying the above source remote sensing image features by the homography matrix transformation, subtracting the reference image in the depth direction and squaring the result, the final three-dimensional cost volume is formed by aggregating multiple views. The calculation method is as follows: (9) Where N is the total number of views, and D is the total number of depth assumptions. Assuming the initial depth value, This represents the features of the i-th source remote sensing image at depth d. Indicates the features of the reference remote sensing image, This indicates the square calculation.

5. The remote sensing image multi-view reconstruction method based on deformable self-attention transformer according to claim 4, characterized in that, The specific method for cost volume regularization in step S4 is as follows: S41. After obtaining the 3D cost volume COST, the cost volume is regularized. Specifically, the 3D cost volume is input into 3D-UNet for processing. The specific structure of the network is as follows: In the first convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 1 for normalization. In the second convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 2 for normalization. In the third convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 3 for normalization. In the fourth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 4 for normalization. In the fifth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 2 and a padding value of 1. A batch normalization layer is then added after layer 5 for normalization. In the sixth convolutional layer, a convolutional kernel with a size of 3×3×3 is defined, with a stride of 1 and a padding value of 1. A batch normalization layer is then added after layer 6 for normalization. Deconvolutional layers are used in layers 7, 8, and 9, with a kernel size of 3×3×3, a stride of 2, padding of 1, and output padding of 1. A batch normalization layer is then added after these layers for normalization. S42. Softmax operation is used for probability normalization, resulting in multiple probability values, forming a probability volume. The depth hypothesis value with the highest probability value in the probability volume is used as the prediction value, ultimately yielding the initial depth map. .

6. The remote sensing image multi-view reconstruction method based on deformable self-attention transformer according to claim 5, characterized in that, The specific method for step S5 is as follows: S51. After obtaining the initial depth map, perform convolution processing on the initial depth map to obtain the residual depth hypothesis value. The reference image and the disparity values ​​corresponding to each pixel in the reference image are concatenated and then processed through three convolutional layers. The specific processing method is as follows: (10) Here, cat represents a connection operation, and conv1, conv2, and conv3 represent three convolutional layers. For the reference image, d is the depth value in the depth map; Phase Two The feature map at the scale is repeated in step S2 to obtain the source remote sensing image features and reference remote sensing image features after the second stage of feature self-attention. After step S3, the depth hypothesis value range of the homography matrix in S3 becomes... The second depth map is obtained after the S4 step operation. ; S52, Transfer the second depth map After calculation using formula (10) in S51, the depth residual value is obtained again. The range of depth assumption values ​​is optimized by applying the features of the third-stage scale through steps S2 and S3. In step S3, the range of depth assumption values ​​for the homography matrix changes. Step S4 then yields the third depth map. ; S53, the third depth map After calculation using formula (10) in S51, the depth residual value is obtained again. To optimize the range of depth assumption values, the features at the fourth-stage scale are processed through steps S2 and S3. In step S3, the range of depth assumption values ​​for the homography matrix changes. Then, step S4 is used to obtain the final refined depth map. .

7. The remote sensing image multi-view reconstruction method based on deformable self-attention transformer according to claim 6, characterized in that, Step S6 reconstructs the 3D remote sensing image, and the specific method is as follows: S61. After obtaining the refined depth map, calculate the loss between the depth map and the ground truth value. Train the entire network based on the obtained loss value until the training limit is reached. The loss function is calculated as follows: (1) First, calculate the cross-entropy loss between the final generated depth map and the ground truth, and construct the pixel-level loss. For all valid pixel sets belonging to reliable pixels For pixel p, perform the following calculations: (11) in, and These represent the actual and predicted values ​​of pixel p at the depth hypothesis d, respectively. Assuming a minimum depth, This is the assumed maximum depth value; (2) Input the reference remote sensing image into the VGG16 network to extract features and construct a feature-level loss. The features from layers 8, 15, and 22 are extracted for calculation, and the specific calculation is expressed as follows: (12) in, The i-th layer is obtained by passing through the convolutional layers in VGG16 to obtain the features of the reference remote sensing image. Let k be the source remote sensing image features obtained from the i-th convolutional layer of the VGG16 layer in the n-th view, where N is the total number of views and k is the weight value. index, This indicates the absolute value operation; (3) Adding the two loss functions together yields the final, complete loss function, expressed as follows: (13) in, This represents the pixel-level loss in (1). Indicate the feature-level loss in (2); S62. The depth map obtained after training stops is the final depth map. The depth map is also known as the disparity map. After obtaining the depth map of the reference remote sensing image, the depth value of each pixel in the reference remote sensing image is calculated using the following formula to obtain the three-dimensional structure of the image, that is, the three-dimensional remote sensing image is reconstructed. (14) Where b is the baseline, f is the focal length, d is the depth value in the depth map, and z is the depth value in three-dimensional space.

Images