Building height estimation method based on image fusion and deep learning

Abstract

The invention discloses a building height estimation method based on image fusion and deep learning, and the method comprises the steps: firstly carrying out the multi-element image fusion, and carrying out the deep fusion of the low-frequency structural features of SAR data through a Laplace pyramid hierarchical fusion strategy while maintaining the high-frequency texture details of an optical image; and making a data set. Then building a deep learning model based on a deep learning model of an encoder, a decoder and a visual transformer; deep fusion of local detail perception and global semantic modeling is realized through a deep learning model, an encoder extracts multi-scale space-spectral features from a four-band fusion image, a local branch reinforces fine-grained features such as building edges through 3 * 3 convolution, and a global branch models a large-range building group height distribution rule by using a ViT encoder. Local features and global context are dynamically coupled through a designed range attention map (A), and a pixel-level height probability distribution model is established based on unit height probability tensor, so that the problem of long-tail distribution in building height estimation is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image analysis and intelligent three-dimensional modeling, and specifically relates to a building height estimation method based on multi-source image fusion and deep learning, which is suitable for scenarios such as smart city management and urban planning. Background Art

[0002] With the acceleration of urbanization, building height estimation has important applications in smart city construction, three-dimensional map reconstruction, disaster emergency response, and urban heat island effect analysis. While traditional measurement methods (such as total stations, lidar, and manual surveying) offer high accuracy, they suffer from limitations such as high cost, low efficiency, and difficulty in large-scale coverage. In recent years, building height estimation technology based on remote sensing imagery has become a research hotspot due to its advantages such as non-contact, low cost, and wide coverage. However, existing technologies still face core bottlenecks such as insufficient multi-source data fusion, limited feature expression capabilities, and poor adaptability to complex scenarios.

[0003] Traditional measurement methods are all image-based technologies. One method is to estimate building heights through multi-view drone imagery. The principle is to use drones to take multiple images of buildings from different angles, and use computer vision technology (such as feature point matching and dense point cloud reconstruction) to process and analyze these images, extracting the building's feature points and texture information. Then, through the principles of photogrammetry, the building's three-dimensional spatial coordinates are calculated to ultimately determine the building's height. However, the method of estimating building heights through multi-view drone imagery requires drones and other flying equipment to scan and shoot, which greatly increases the manpower, material resources, and time costs required. The method relies on high-overlap images and rich surface textures, which easily leads to reconstruction holes in low-texture areas. The high computational complexity makes real-time processing difficult. Another reason is that the no-fly zones set by the government make this technology unable to meet actual needs.

[0004] The second method is to estimate height using synthetic aperture radar (SAR) image data. This method uses the interaction between the radar signal emitted by the SAR system and ground targets, and infers the distance information of the target by receiving and analyzing the time delay and phase change of the echo signal. At the same time, combined with the motion trajectory of the radar platform and synthetic aperture technology, the echo signals received multiple times are superimposed and processed to form a high-resolution radar image. Then, the height of the building is estimated through image processing and algorithm analysis. While traditional SAR images have a large coverage range and high accuracy, they generally have the problem of excessive shooting angles. Due to the overlap effect and perspective contraction phenomenon, the edges of high-rise buildings are prone to geometric distortion, and phase unwrapping errors will lead to a decrease in height inversion accuracy. This has a great impact on the geometric shape of the building and significantly reduces its reliability in dense urban scenes.

[0005] The third approach is to use end-to-end deep learning models (such as convolutional neural networks and Transformer architectures) to directly regress building height distribution from a single optical or SAR image. By training the model with a large amount of annotated data, the model exploits highly correlated features implicit in the image, such as shadows, textures, and geometric context, to achieve pixel-by-pixel height prediction. However, using deep learning for height estimation has its limitations. A single type of remote sensing image cannot provide sufficient information to accurately estimate elevation in complex terrain. While RGB images contain rich spectral information, their spatial resolution is limited when dealing with complex terrain or when affected by weather conditions. While SAR images offer all-day, all-weather observation capabilities, they contain relatively little spectral information. By fusing RGB and SAR images, the complementary information between the two can be fully utilized. Furthermore, processing these fused images with deep learning algorithms can improve the accuracy and robustness of building height estimation. The patented "Method for Predicting Farmland Building Height Based on Remote Sensing Images" utilizes this approach. However, this research does not adequately address a typical problem in deep learning: the long-tail distribution problem. Because most areas in satellite remote sensing imagery are low-rise, non-building areas, while only a few contain buildings, the long-tail problem can cause taller buildings in the image to be predicted as lower buildings, and lower buildings to be predicted as even lower non-buildings, resulting in significant errors. This is why the above solution is only suitable for predicting building heights in open farmland. From a technical perspective, its application to estimating urban building heights is very limited. Summary of the Invention

[0006] In order to estimate the height of buildings in satellite images, this paper proposes a building height estimation method based on image fusion and deep learning. A pixel-level image fusion method based on multi-scale transformation is used to fuse satellite optical images (RGB images) and SAR images. The Visual Transformer module (ViT) is used to solve the long-tail distribution problem, thereby achieving accurate estimation of building heights in satellite images.

[0007] The present invention provides a building height estimation method based on image fusion and deep learning, comprising the following steps:

[0008] Step 1: Multi-image fusion and dataset preparation.

[0009] 1) Obtain input data including SAR images, satellite optical images and corresponding nDSM models, and preprocess the SAR images.

[0010] 2) SAR image cropping based on geographic coordinates, and resampling the cropped SAR image to make its resolution consistent with the optical image.

[0011] 3) Construct Gaussian pyramids for the three-band optical image and SAR grayscale image respectively; then calculate the Laplace pyramid to obtain detail information; further, for each layer of the Laplace pyramid of the optical image and SAR image, select the significant features of the optical image and SAR image according to the principle of maximizing regional energy; finally, use the recursive formula to calculate the final three-band fusion image fusion image I Fusion .

[0012] 4) The 3-band fusion image I Fusion Compared with the original SAR single-band image I SAR Generate 4-band fusion image I by channel superposition Final , and compare it with the corresponding DSM Figure 1 The corresponding data sets are obtained and divided into training sets and test sets.

[0013] Step 2: Build a deep learning model based on the encoder-decoder and visual transformer deep learning model.

[0014] Step 3: Input the training set obtained in step 1 into the deep learning model in step 2 for end-to-end training to obtain the trained model weights.

[0015] Step 4: Use the trained deep learning model to predict the nDSM map.

[0016] The advantages of the present invention are:

[0017] 1. A pixel-level image fusion method based on multi-scale transformation in the method of the present invention, through the Laplace pyramid hierarchical fusion strategy, deeply fuses the low-frequency structural features of SAR data (such as vertical scattering characteristics) while retaining the high-frequency texture details of optical images (such as building edges and shadows), and dynamically optimizes the feature weights based on the regional energy maximization criterion, solving the problems of spatial inconsistency and information complementarity between multimodal data, significantly improving the representation ability of building height-related features, providing deep learning models with multi-band input with both geometric integrity and semantic richness, and ultimately achieving coordinated optimization of height estimation accuracy and robustness in complex scenarios.

[0018] 2. The method of the present invention achieves a deep fusion of local detail perception and global semantic modeling through the collaborative design of the encoder-decoder architecture and the Visual Transformer (ViT) through a deep learning model. The encoder (such as Unet, etc.) extracts multi-scale spatial-spectral features from the four-band fused image. The local branch strengthens fine-grained features such as building edges through 3×3 convolution, while the global branch uses the ViT encoder to model the height distribution pattern of large-scale building complexes. The designed range attention map (R) dynamically couples local features with global context, and establishes a pixel-level height probability distribution model based on the unit height probability tensor, effectively solving the long-tail distribution problem in building height estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the building height estimation method based on image fusion and deep learning of the present invention;

[0020] Figure 2 This is a block diagram of the deep learning model structure based on encoder-decoder and visual transformer (ViT);

[0021] Figure 3 A satellite image before applying the method of the present invention;

[0022] Figure 4 This is the nDSM diagram predicted by the method of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be described in further detail below with reference to the accompanying drawings.

[0024] The present invention provides a building height estimation method based on image fusion and deep learning, such as Figure 1 As shown, the specific steps are:

[0025] Step 1: Multi-image fusion and dataset preparation.

[0026] 1) Input data acquisition and preprocessing.

[0027] The input data include SAR images and satellite optical images and the corresponding nDSM models;

[0028] Among them, satellite optical images and their corresponding nDSM models (normalized Digital Surface Models (nDSM)) can directly reflect the vertical height of objects to the ground, which includes building height information and comes from the DFC2023 dataset.

[0029] SAR images were acquired via Sentinel satellites, and SAR data covering the same area and at a similar time as the optical images were downloaded.

[0030] Acquired SAR images are radiometrically calibrated to eliminate the effects of hardware factors such as sensor gain, antenna mismatch, and range attenuation on image radiometric values. Furthermore, atmospheric correction is performed on the images using an atmospheric model to correct for radiometric deviations caused by atmospheric absorption and scattering, ensuring that the radiometric values of the SAR images are consistent with the actual backscattering characteristics of the ground objects. Because SAR images are significantly affected by speckle noise, an adaptive Lee filter algorithm is employed to reduce noise, suppressing it while preserving edge and texture details. Finally, the VV polarization channel of the SAR data is converted into a single-band grayscale image, completing the data preprocessing.

[0031] 2) SAR image cropping (resolution adjustment) based on geographic coordinates.

[0032] Since the resolution of SAR images is different from that of optical images, directly using the original cropping formula will result in a mismatch in spatial range. Therefore, the cropping formula needs to be adjusted using the resolution scaling factor, as follows:

[0033] First, calculate the resolution scaling factor, assuming the resolution of the optical image is R Optical , the resolution of the SAR image is R SAR , then the resolution scaling factor α is:

[0034]

[0035] Then, the coordinate mapping formula needs to be adjusted, and the longitude and latitude of the optical image corner point is set to (lon i ,lat i ), the affine transformation parameters of the SAR image are a, b, c, d, e, f, then the pixel coordinates (x, y) in the SAR image are:

[0036]

[0037] According to the adjusted coordinate mapping formula, the pixel coordinates of the four corner points of the optical image in the SAR image are calculated (x1, y1)(x2, y2)(x3, y3)(x4, y4), and the cropping range of the SAR image is determined according to the extreme values of the coordinates:

[0038] x min =min(x1,x2,x3,x4),y min =min(y1,y2,y3,y4)

[0039] x max =max(x1,x2,x3,x4),y max =max(y1,y2,y3,y4)

[0040] Extracting rectangular regions [x min:x max ,y min :y max ] is used as the matching area with the corresponding optical image for cropping, and finally the cropped SAR image is resampled to make its resolution consistent with the optical image.

[0041] 3) Use Laplace pyramid transform for image fusion.

[0042] First, Gaussian pyramids are constructed for the optical image (RGB three-band) and the SAR grayscale image, specifically:

[0043] The original image (optical image or cropped SAR image) is decomposed into multiple scales. Each layer is Gaussian filtered and down-sampled to generate a low-resolution sub-image. A total of three downsamplings (three-layer pyramid) are performed. The formula is:

[0044] G k =down k I

[0045] Among them, G is the sampled image matrix, down k is the number of downsampling times k. I is the unsampled image matrix;

[0046] Then, the Laplace pyramid is obtained as follows:

[0047] The detail information is obtained by subtracting the convolution result of the next layer of Gaussian image after upsampling from the current layer of Gaussian image. The formula is:

[0048] (When k=3, G k+1 =0)

[0049] Among them, L k is the image matrix corresponding to the current layer of Laplace pyramid, up is the upsampling operation using bilinear interpolation, h is a Gaussian filter with a standard deviation of 1, is the convolution operation.

[0050] Furthermore, for each layer of the Laplace pyramid of optical images and SAR images, the significant features of the optical and SAR images are selected according to the principle of maximizing regional energy. The method is as follows:

[0051]

[0052] Among them, L Optical,k (i,j) and L SAR,k (i, j) are the i-row and j-column elements in the pyramid matrix of the optical image and the SAR image respectively; E Optical,k (i,j) and E SAR,k(i, j) are the local window energies of the optical image and the SAR image centered at i and j respectively; F k (i, j) is the element value at position i, j of the k-th pyramid after fusion.

[0053] Therefore, L Optical,k With L SAR,k Fusion into L' k .

[0054] Further use the recursive formula to obtain the final fused image.

[0055] The recursive formula is:

[0056]

[0057] The three-layer pyramid of the present invention is obtained in the following process:

[0058] G3'=L'3

[0059]

[0060] Finally, we get the 3-band fusion image I Fusion .

[0061] 4) Multi-band superposition is used to obtain the data set.

[0062] The 3-band fusion image I Fusion Compared with the original SAR single-band image I SAR Generate 4-band fusion image I by channel superposition Final :

[0063]

[0064] The final 4-band fusion image is compared with the corresponding DSM Figure 1 The dataset is divided into training set and test set according to the ratio of 7:3.

[0065] Step 2: Build a deep learning model based on the encoder-decoder and visual transformer (ViT) deep learning model.

[0066] Figure 2 A deep learning model based on encoder-decoder and visual transformer (ViT) is given, in which the encoder-decoder uses the Unet network (other encoder-decoders can also be used as replacements).

[0067] Using Unet network from the input image (4 band fusion image I Final ∈R 4×H×W) to generate feature maps F, which contain rich spatial and spectral information. In order to perform both local and global statistical analysis on the feature map F, a 3×3 convolutional layer is used in the local branch to process the feature map F to obtain the local feature map L;

[0068] The ViT encoder is then used in the global branch to model the global context as follows:

[0069] A convolution with a kernel size of p×p and an output channel of M is used to process the feature map F to obtain a tensor of size H / 2p×W / 2p×M, and this tensor is flattened into a flat tensor E. The tensor E is the input of the ViT encoder, and the output of the ViT encoder is the tensor G. The first-dimensional sub-tensor G1 of the tensor G is input into the multi-layer perceptron MLP to obtain the N-dimensional unit height probability tensor B (B=(b0,b1,...,b N )), then use the softmax function to normalize B to get the relative unit height probability tensor W (W=(w0,w1,...,w N ), w = softmax(MLP(G1)).

[0070] Finally, the height h of each unit can be obtained by the following formula:

[0071]

[0072] Among them, h min and h max is the minimum and maximum height in the nDSM image corresponding to the current input image; w i is the height probability of the i-th unit, w j is the height probability of the first i-1 units.

[0073] The remaining sub-tensor of the tensor G is used as the global feature map G'. G' is passed through a 1×1 convolution kernel and then cross-producted with the local feature map L to obtain the range attention map A. A 1×1 convolution operation is then performed to make the dimension of R the same as the dimension of the unit height probability tensor W obtained above. Further, the softmax function is used to calculate the probability P (P = (p0, p1, ..., p N )), the final height calculation formula for each pixel is as follows:

[0074]

[0075] Step 3: Extract features and train the dataset.

[0076] During the model training phase, the satellite imagery dataset obtained in Step 1 (the training set in Step 1) was fed into the neural network model in Step 2 for end-to-end training. The training process employed the Nadam optimizer, with an initial learning rate of 0.001 to rapidly explore the parameter space. A constant learning rate was maintained for the first 10 epochs to ensure initial model convergence. A stepwise learning rate decay strategy was then initiated: starting from the 11th epoch, the learning rate was reduced by 0.0001 per training cycle, reaching a lower bound of 0.0001. The training batch size was set to 8, with random rotations, flips, and Gaussian noise (σ = 0.01) dynamically injected into each batch to enhance data diversity. Gradient clipping (threshold 1.0) was also introduced to stabilize the training process. After each epoch, the loss function was calculated using the validation set, and a dynamic optimal model selection mechanism was implemented: if the validation loss for the current epoch was lower than the historical optimal, the model weights were updated. If the optimality was not achieved for three consecutive epochs, the model weights were regressed to the most recent optimal model weights to prevent overfitting, ultimately resulting in the trained model weights.

[0077] Step 4: Use the trained deep learning model to predict nDSM images

[0078] The four-band fusion image to be predicted (the test set in step 1) is input into the trained deep learning model. The model extracts multi-scale features from the four-band fusion image through the encoder-decoder network. Local detail perception strengthens fine-grained features such as building edges through 3×3 convolution. Global semantic modeling uses the ViT encoder to model the height distribution law of large-scale building complexes. The local features and global context are dynamically coupled through the designed range attention map A. Finally, a pixel-level height probability distribution model is established based on the unit height probability tensor, and the nDSM map is finally obtained. The optical image and the predicted nDSM map are shown in Figure 3 Figure 4 .

Claims

1. A building height estimation method based on image fusion and deep learning, characterized by: The steps are: Step 1: Multi-image fusion and dataset creation; 1) Obtain input data including SAR images, satellite optical images and corresponding nDSM models, and preprocess the SAR images; 2) SAR image cropping based on geographic coordinates, and resampling the cropped SAR image to make its resolution consistent with the optical image; 3) Construct Gaussian pyramids for the three-band optical image and SAR grayscale image respectively; then calculate the Laplace pyramid to obtain detail information; further, for each layer of the Laplace pyramid of the optical image and SAR image, select the significant features of the optical image and SAR image according to the principle of maximizing regional energy; finally, use the recursive formula to calculate the final three-band fusion image I Fusion ; 4) The 3-band fusion image I Fusion Compared with the original SAR single-band image I SAR Generate 4-band fusion image I by channel superposition Final , match it one by one with the corresponding DSM diagram to obtain the data set, and divide it into training set and test set; Step 2: Build a deep learning model based on the encoder-decoder and visual transformer deep learning model; Step 3: Input the training set obtained in step 1 into the deep learning model in step 2 for end-to-end training to obtain the trained model weights. Step 4: Use the trained deep learning model to predict the nDSM map.

2. The method for estimating building height based on image fusion and deep learning according to claim 1, wherein: In sub-step 1) of step 1, SAR image preprocessing includes: performing radiometric calibration on the acquired SAR image; performing atmospheric correction on the image in combination with an atmospheric model; further performing image noise reduction using an adaptive Lee filter algorithm; and finally converting the VV polarization channel of the SAR data into a single-band grayscale image.

3. The building height estimation method based on image fusion and deep learning as claimed in claim 1, characterized in that: In sub-step 2) of step 1, the SAR image cropping method is: The clipping formula is adjusted by the resolution scale factor: First, calculate the resolution scaling factor, assuming the resolution of the optical image is R Optical , the resolution of the SAR image is R SAR , then the resolution scaling factor α is: Then, the coordinate mapping formula needs to be adjusted, and the longitude and latitude of the optical image corner point is set to (lon i ,lat i ), the affine transformation parameters of the SAR image are a, b, c, d, e, f, then the pixel coordinates (x, y) in the SAR image are: According to the adjusted coordinate mapping formula, the pixel coordinates of the four corner points of the optical image in the SAR image are calculated (x1, y1)(x2, y2)(x3, y3)(x4, y4), and the cropping range of the SAR image is determined according to the extreme values of the coordinates: x min =min(x1,x2,x3,x4),y min =I(y1,y2,y3,y4) x max =max(x1,x2,x3,x4),y max =max(y1,y2,y3,y4) Extracting rectangular regions [x min :x max ,y min :y max ] is cropped as the matching area with the corresponding optical image.

4. The method for estimating building height based on image fusion and deep learning according to claim 1, wherein: In sub-step 3) of step 1, the image fusion method using Laplace pyramid transform is: First, construct the Laplace pyramid for the optical image and the SAR grayscale image respectively, specifically: The original image is decomposed into multiple scales, and each layer generates a low-resolution sub-image through Gaussian filtering and downsampling. A total of 3 downsampling formulas are: G k =down k I Among them, G is the sampled image matrix, down k is the number of downsampling times k; I is the unsampled image matrix; Then, the Laplace pyramid is obtained as follows: The detail information is obtained by subtracting the convolution result of the next layer of Gaussian image after upsampling from the current layer of Gaussian image. The formula is: Among them, L k is the image matrix corresponding to the current layer of Laplace pyramid, up is the upsampling operation using bilinear interpolation, h is a Gaussian filter with a standard deviation of 1, is the convolution operation; Furthermore, for each layer of the Laplace pyramid of optical images and SAR images, the significant features of the optical and SAR images are selected according to the principle of maximizing regional energy. The method is as follows: Among them, L Optical,k (i,j) and L SAR,k (i, j) are the i-row and j-column elements in the pyramid matrix of the optical image and the SAR image respectively; E Optical,k (i,j) and E SAR,k (i, j) are the local window energies of the optical image and the SAR image centered at i and j respectively; F k (i, j) is the element value at position i, j of the k-th pyramid after fusion; Therefore, L Optical,k With L SAR,k Fusion into L' k ; The recursive formula is further used to obtain the final fused image: The recursive formula is:

5. The building height estimation method based on image fusion and deep learning as claimed in claim 1, characterized in that: The specific method of step 2 is: First, the Unet network is used to extract the input image I Final Generate feature map F in the local branch, and process feature map F in the local branch to obtain local feature map L; Then, in the global branch, the ViT encoder is used to model the global context. A convolution with a kernel size of p×p and an output channel of M is used to process the feature map F to obtain a tensor of size H / 2p×W / 2p×M. This tensor is flattened into a flat tensor E, which is the input of the ViT encoder. The output of the ViT encoder is the tensor G. The first-dimensional sub-tensor G1 of the tensor G is input into the multi-layer perceptron MLP to obtain the N-dimensional unit height probability tensor B = (b0, b1, ..., b N ), then use the softmax function to normalize B to get the relative unit height probability tensor W = (w0, w1, ..., w N ), w = softmax(MLP(G1)); Finally, the height h of each unit can be obtained by the following formula: Among them, h min and h max The minimum and maximum height values in the nDSM image corresponding to the current input image; The remaining sub-tensor of the tensor G is used as the global feature map G'. G' is passed through a 1×1 convolution kernel and then cross-producted with the local feature map L to obtain the range attention map A. A 1×1 convolution operation is then performed to make the dimension of R the same as the dimension of the unit height probability tensor W obtained above. Further, the softmax function is used to calculate the probability P = (p0, p1, ..., p N ), the final height calculation formula for each pixel is as follows:

6. The method for estimating building height based on image fusion and deep learning according to claim 1, wherein: In step 4, the test set in step 1 is input into the trained deep learning model. The model extracts multi-scale features from the four-band fused image through the encoder-decoder network. Local detail perception is enhanced through 3×3 convolution to enhance the fine-grained features of building edges. Global semantic modeling uses the ViT encoder to model the height distribution pattern of large-scale building complexes. The local features and global context are dynamically coupled through the designed range attention map A. Finally, a pixel-level height probability distribution model is established based on the unit height probability tensor to obtain the nDSM map.

Images