Building height estimation method based on multi-modal image fusion

Abstract

The invention discloses a building height estimation method based on multi-modal image fusion, and relates to the technical field of computer vision, and the method specifically comprises the steps: obtaining a high-resolution satellite image and a building height data set; constructing a semantic segmentation module for extracting a building area from the optical image; carrying out pixel expansion on the extracted building footprint so as to contain a building shadow area; constructing a multi-modal image fusion module for fusing the synthetic aperture radar image and the optical image; a building height mapping module is constructed, and the building height and the building height level are predicted in combination with image embedding and building analysis areas; and training and evaluating the building height estimation model. According to the method, the complementary characteristics of the synthetic aperture radar image and the optical image are effectively fused, the problem of complex ground object background interference and the problem of building edge blurring are solved, the building height estimation precision is remarkably improved, and vertical-dimension data support is provided for urban planning.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for estimating building height based on multimodal image fusion. Background Technology

[0002] Building height plays a crucial role in urban planning. With accelerating urbanization, residents' demands for living space, commercial facilities, and industrial sites are increasing, prompting the continuous expansion of urban boundaries. However, the increase in the number of urban buildings often leads to greater energy consumption and carbon emissions, exacerbating the urban heat island effect. Therefore, the scientific planning of buildings is of great significance for mitigating climate problems and improving resident comfort. Over the past few decades, the horizontal dimensions of buildings (such as building segmentation) have been extensively studied, spawning many global products. In contrast, research on the vertical dimensions of buildings is relatively limited. Accurate building height estimation is essential for urban planning because it is closely related to factors such as land use, transportation, telecommunications, and urban climate. Furthermore, building height data is crucial for population estimation, material inventory mapping, and simulating air pollutant dispersion. While airborne lidar and high-resolution aerial imagery can accurately acquire building height information, their high cost and time-consuming acquisition processes limit their use for large-scale mapping.

[0003] Remote sensing data, due to its low acquisition cost and wide coverage, has become an important data source for building height estimation and is widely used. Meanwhile, deep learning technology, with its powerful feature representation and nonlinear fitting capabilities, has demonstrated significant performance advantages in building height estimation. However, modern deep learning-based building height estimation methods suffer from three key problems. First, insufficient multimodal data fusion: existing multimodal data fusion methods neglect the extraction of common and differential information in cross-modal data, limiting model performance improvement. Second, interference from complex background features: the presence of complex backgrounds leads to false building detection and recognition, reducing the accuracy of building height estimation. Finally, building edge blurring: existing building estimation methods often cause building edge pixels to spread outwards and become confused with pixels of other buildings or the background, forming blurred boundaries. However, current research lacks an effective deep learning method to address these key problems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a building height estimation method based on multimodal image fusion, which effectively integrates the complementary characteristics of synthetic aperture radar images and optical images, solves the interference from complex terrain backgrounds and the problem of blurred building edges, thereby significantly improving the accuracy of building height estimation.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for estimating building height based on multimodal image fusion, comprising the following steps:

[0007] Step 1: Acquire high-resolution satellite imagery and building height datasets;

[0008] Step 2: Construct a semantic segmentation module to extract building regions from optical images;

[0009] Step 3: Perform pixel expansion on the extracted building area to obtain the building analysis area containing the shadow area of ​​the building;

[0010] Step 4: Construct a multimodal image fusion module to fuse synthetic aperture radar images and optical images to obtain the fused image embedding;

[0011] Step 5: Construct a building height mapping module, combining the fused image embedding with the building analysis region to predict building height and building height level;

[0012] Step 6: Based on the dataset obtained in Step S1, train the constructed semantic segmentation module, multimodal image fusion module, and building height mapping module to obtain an estimation model for building height estimation.

[0013] Furthermore, in step 1, high-resolution satellite images and building height datasets are obtained based on open-source datasets, and the image resolutions in the datasets are normalized in size.

[0014] Furthermore, step 1 also includes setting building height labels and building height level labels based on the spatial resolution of the images in the dataset.

[0015] Furthermore, in step 1, the semantic segmentation module's processing includes:

[0016] The input optical image is converted into a sequence embedding, and the encoder of the semantic segmentation module segments the image into non-overlapping blocks of different sizes (e.g., block size is set to 2×2).

[0017] A linear embedding layer is applied to project the feature dimensions of the sequence embedding onto the set feature dimensions.

[0018] Then, hierarchical feature representations are generated through multiple cascaded Swing Transformer blocks and block fusion layers. For the output features of each Swing Transformer block, the width and height of the features are reduced by a factor of 2, while the number of channels is doubled.

[0019] The hierarchical feature representation generated by the encoder is fed into the rectangular self-correction module (RSC) to adjust the attention region, and then fed into the decoder of the semantic segmentation module;

[0020] The decoder expands the features of the last layer of the encoder's Swing Transformer block through a block expansion layer, and then fuses and decodes the decoded features from the previous layer and the encoded features from the encoder through multiple cascaded Swing Transformer blocks and block expansion layers.

[0021] The features output by the decoder are passed through a segmentation head guided by a dynamic prototype, which outputs a segmentation mask for the building.

[0022] Furthermore, the rectangular self-calibration module is used to adjust the regions in the attention map, thereby forcing the model to focus on the building areas. The rectangular self-calibration module (RSC) includes rectangular self-calibrated attention (RSA), batch normalization, and multilayer perceptron (MLP).

[0023] RCA uses horizontal and vertical pooling operations to capture the axial global context in two directions and generates two different axis vectors, then performs a broadcast addition on the two axis vectors.

[0024] To enhance the ability to distinguish between building foregrounds and complex environments, the dynamic prototype-guided segmentation head explicitly embeds generated category information into the pixel feature space by learning a prototype. Furthermore, the dynamic prototype-guided segmentation head is as follows:

[0025] The DPG header projects the features output by the decoder onto the category feature space;

[0026] The features in the category space are multiplied by the features in the pixel space to obtain a dynamic prototype, which is then compressed through a fully connected layer and a softmax layer. The dimension is used to obtain a category embedding vector, where C is the preset channel dimension;

[0027] The category embedding vector is fed into the layer normalization layer and the ReLU activation function layer, and the pixel features are weighted to obtain the classification features; the classification features are then fed into a convolutional layer to predict the building segmentation mask.

[0028] Based on the above processing, the ability to distinguish between building foreground and complex environment is improved. The dynamic prototype-guided segmentation head explicitly embeds the generated category information into the pixel feature space by learning the prototype.

[0029] Furthermore, in step 3, the extracted building regions are expanded by pixels by extending each building pixel upwards, to the left, and to the right by one pixel, thereby including the shadow areas. The resulting building analysis partitions will force the model to focus on shadow areas that are related to the building height.

[0030] Furthermore, multi-step 4 specifically includes:

[0031] Step 41: Use the Restormer-CNN dual-branch encoder structure to extract multi-level features from SAR images and optical images respectively;

[0032] Step 42: Features of the same spatial scale are then fed into the multimodal fusion layer for cross-modal information interaction and fusion;

[0033] Step 43: The multi-level fused features are fed into the decoder network, and the high-frequency components of the features are enhanced through a frequency-aware multi-scale fusion layer.

[0034] Furthermore, step 4 specifically includes:

[0035] Step 41: Employ a Restormer-CNN dual-branch encoder structure to extract multi-level features from SAR and optical images respectively; that is, use the Restormer block as a Transformer network because it models global features by implementing self-attention in the channel dimension, requiring less computation. The CNN network uses a classic residual convolutional network;

[0036] Step 42: Features of the same spatial scale are then fed into the multimodal fusion layer MMFL for cross-modal information interaction and fusion;

[0037] Step 43: The multi-level fused features are fed into the decoder network. The frequency-aware multi-scale fusion layer FMSF is used to enhance the high-frequency components of the features, thereby preserving as much high-frequency edge detail information of buildings in the input image as possible.

[0038] Furthermore, in step 41, the Restormer block is used as a Transformer network because it models global features by implementing self-attention in the channel dimension, requiring less computation. The CNN network uses a classic residual convolutional network.

[0039] Furthermore, in step 42, the specific steps of the multimodal fusion layer MMFL are as follows:

[0040] Image features from SAR and optical images generated by the backbone network are projected into token sequences. and Then, it is projected through a linear layer onto the query Q, key K, and value V matrix:

[0041]

[0042] Wherein, DIEM represents the difference information enhancement module, and SIEM represents the common information enhancement module. , and These are the output features of the DIEM, the first SIEM, and the second SIEM, respectively; that is... The query matrix is ​​first fed into the DIEM module for difference information extraction, and the updated Q matrix is ​​then fed into the SIEM module for common information extraction; finally, the Q matrix output by the first SIEM module is compared with... The K and V matrices are fed into a second SIEM module for further common information extraction.

[0043] In the DIEM module, the standard cross-attention mechanism is first used to extract common exploration information, which can be represented as:

[0044]

[0045] in, It is the scaling factor. Subsequently, common information is subtracted from the V matrix. This is to obtain the difference information between Q and V. This process can be represented as:

[0046] ;

[0047] in This represents the extracted difference information. Then, the difference information is injected into Q, represented as:

[0048]

[0049] in, Representation layer normalization, This represents a feedforward neural network. This indicates the output of the DIEM module.

[0050] Although the DIEM module enhances the difference information, its ability to model dependencies on shared information remains limited. Therefore, it is necessary to further enhance the fused features by incorporating shared information from the two images. Similar to the DIEM module, shared information is first extracted using a cross-attention mechanism, and then the Q matrix is ​​enhanced by adding the shared information to the V matrix.

[0051] Furthermore, in step S43, the frequency-aware multi-scale fusion layer FMSF can be represented as:

[0052]

[0053] in, This represents the low-pass filter predicted by the Adaptive Low-Pass Filter (ALPF) generator. Represents upsampling, This represents the high-pass filter predicted by the Adaptive High-Pass Filter (AHPF) generator. This represents the output features of the FMSF module. The ALPF generator addresses the boundary shift problem during upsampling by adaptively smoothing high-level features using a spatially varied low-pass filter, while the AHPF generator enhances the high-frequency boundary details of low-level features by using a spatially varied high-pass filter, resulting in sharper edges for the predicted building features.

[0054] To effectively generate low-pass and high-pass filters, the module's input must first be... and The components are then fused and used as input to both the ALPF and AHPF generators. This process can be represented as:

[0055]

[0056] in Represents a convolutional layer. Represents the Pixel Shuffle upsampling operation. Representing the characteristics after fusion, This represents the channel compression factor.

[0057] The ALPF generator attenuates high-frequency obfuscation information to mitigate boundary shift by predicting a series of dynamic low-pass filters. The ALPF generator then initializes the fused features. A low-pass filter that serves as input and predicts spatial variations. Fusion features. The input is first passed to a convolutional layer, and then undergoes a channel-level softmax normalization operation. This process can be represented as:

[0058]

[0059] in This represents the weights of the predicted low-pass filter. This represents the size of the filter kernel. The purpose of Softmax normalization is to ensure that all weights in the low-pass filter kernel are positive and sum to 1. Next, adjustments are made using the Pixelunshuffle method. The shape is changed so that both the height and width are halved, while the number of channels is increased fourfold. Then, the channels are divided into four groups, each with a low-pass filter, represented as... ,in Four sets of low-pass filters were used to filter the features. After convolution, four filtered features are obtained. These four features are then spatially recombined to obtain features upsampled by a factor of 2. This process can be described by a formula:

[0060]

[0061] Although the ALPF generator can effectively mitigate the building boundary shift caused by upsampling, high-frequency spatial details in low-level features still cannot be effectively recovered after downsampling. According to the Nyquist sampling theorem, frequencies above the Nyquist frequency will be permanently lost during downsampling. To compensate for this information loss, an AHPF generator is used to predict a high-pass filter for spatial variations and to optimize the low-level features. Filtering is performed to preserve as much as possible. High-frequency building edge detail information. The structure of the AHPF generator is as follows: Figure 7 As shown in (c). Similar to the ALPF generator, the AHPF generator will initialize the fused features. A high-pass filter that serves as input and predicts spatial variations. First, The filter kernel weights are predicted using a convolutional layer, followed by a channel-level softmax normalization operation and a filter inversion operation. This process can be represented as follows:

[0062]

[0063] in This represents the weights of the initial filter kernel for prediction. This represents the size of the filter kernel. To ensure that the final generated kernel is a high-pass filter kernel, channel-level softmax normalization is first used to obtain the low-pass filter kernel, and then the generated low-pass filter kernel is subtracted from the unit kernel. This is applied to low-level features... By applying a high-pass filter and adding residual connections, the enhanced features can be obtained:

[0064]

[0065] Furthermore, step 5 specifically includes the following steps:

[0066] Step 51: Use The convolutional layer projects the mask cue onto a specified spatial dimension, and then adds the image embedding and the mask cue element-wise to obtain the mask;

[0067] Step 52: Perform self-attention operation on the mask;

[0068] Step 53: Cross-attention from mask to image embedding, where the mask is used as the query matrix to obtain the updated mask;

[0069] Step 54: Pass the updated mask through a feedforward neural network, and then perform step 55 based on the mask processed by the feedforward neural network;

[0070] Step 55: Cross-attention from image embedding to mask, where the image embedding is used as the query matrix, and perform step 56 based on the resulting updated image embedding;

[0071] Step 56: The updated image embedding is passed through a convolutional layer to obtain the predicted building height map and height level map.

[0072] The technical solution provided by this invention brings at least the following beneficial effects:

[0073] This invention proposes an explicitly semantically guided multimodal fusion network to estimate building heights at a specified height resolution (e.g., 10-meter spatial resolution). The estimation model constructed in this invention includes: a semantic segmentation module (i.e., a building extraction module), a multimodal image fusion module, and a height mapping module. The building extraction module first uses a semantic segmentation model to extract semantic information of buildings from optical images, separating the buildings from complex backgrounds to reduce background interference on height estimation. The multimodal image fusion module is a multi-level network that fuses SAR and optical images to eliminate modal heterogeneity and fuse complementary information. Specifically, the multimodal image fusion module mainly includes a multimodal fusion layer (MMFL) and a frequency-aware multi-scale fusion layer (FMSF). The MMFL layer can efficiently extract difference information from multi-source images and enhance common information to improve the fusion effect. The FMSF layer performs feature enhancement from a frequency domain perspective, solving the problem of lost high-frequency edge information and thus sharpening building edges. Finally, the height mapping module combines the fused image embedding representation with analysis region cues to predict building height and building height level. The method proposed in this invention improves the accuracy and application scale of existing building height estimation methods. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A flowchart illustrating the implementation of a building height estimation method based on multimodal image fusion, as provided in this embodiment of the invention;

[0076] Figure 2 This is a schematic diagram of the framework of the building height estimation method based on multimodal image fusion according to an embodiment of the present invention;

[0077] Figure 3 This is a schematic diagram of the semantic segmentation module according to an embodiment of the present invention;

[0078] Figure 4 This is a schematic diagram of the Swing Transformer, Rectangular Self-calibration Module (RSC), and Dynamic Prototype Guidance (DPG) segmentation head according to an embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of the multimodal image fusion module according to an embodiment of the present invention;

[0080] Figure 6 This is a schematic diagram of the multimodal fusion layer, DIEM and CIEM, according to an embodiment of the present invention.

[0081] Figure 7 This is a schematic diagram of the frequency-aware multi-scale fusion layer according to an embodiment of the present invention;

[0082] Figure 8 This is a schematic diagram of the height mapping module according to an embodiment of the present invention;

[0083] Figure 9 This is a schematic diagram of the building height estimation results according to an embodiment of the present invention. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0085] like Figure 1 As shown in the figure, an embodiment of the present invention proposes a method for estimating building height based on multimodal image fusion, which includes the following steps:

[0086] S1: Based on open-source datasets, obtain high-resolution satellite images and building height datasets, and divide the datasets into training, validation and test sets;

[0087] S2: Construct a semantic segmentation module to extract building regions from optical images;

[0088] S3: Expand the extracted building footprints by pixels to include the shadow areas of the buildings;

[0089] S4: Construct a multimodal image fusion module to fuse synthetic aperture radar images and optical images;

[0090] S5: Construct a building height mapping module, which combines the fused image embedding with the building analysis region to predict building height and building height level;

[0091] S6: Train and evaluate the building height estimation model.

[0092] like Figure 2 As shown, the method proposed in this embodiment mainly comprises three parts: 1) a building extraction module; 2) a multimodal image fusion module; and 3) a building height mapping module. First, the RGB bands of the Sentinel-2 image are input into a building semantic segmentation network to obtain the building footprint. Then, the region representing the building in the building footprint image is simultaneously expanded upwards, to the left, and to the right by one pixel, thereby obtaining the analysis region for the building height. The multimodal image fusion module fuses the Sentinel-1 and Sentinel-2 images to obtain a unified image embedding representation. Finally, the analysis region is fed into the height mapping module, fused with the image embedding, and simultaneously predicts the building height and building height level.

[0093] like Figure 3 As shown, the building semantic segmentation module proposed in this embodiment of the invention includes an encoder, a decoder, a rectangular self-correction module, and a dynamic prototype-guided segmentation head. The basic unit of the encoder is a Swing Transformer block. To convert the input Sentinel-2 optical image into a sequence embedding, the encoder first segments the image into non-overlapping blocks, with a block size of [missing information]. Using this partitioning method, the feature dimensions of each block become... Then, a linear embedding layer is applied to project the feature dimension to dimension C (set value). The blocks transformed into the feature space are then passed through multiple SwinTransformer blocks and block fusion layers to generate hierarchical feature representations. At each stage, the width and height of the features are reduced by a factor of 2, while the number of channels is doubled. Specifically, the SwinTransformer blocks are used for feature representation learning, while the block fusion layers are used to reduce spatial resolution and increase feature dimension. The hierarchical features generated by the encoder are fed into a rectangular self-calibration module to adjust the attention region, and then fed into the decoder. The decoder adopts a symmetrical structure design to the encoder, and its components are SwinTransformer blocks and block expansion layers. The block expansion layers, in contrast to the block fusion layers, are specifically used to perform upsampling. Each layer of the decoder network fuses features from the previous level and features from the encoder. Finally, the features output by the decoder are passed through a dynamically prototype-guided segmentation head to output a segmentation mask for the building.

[0094] like Figure 4 As shown in (2), the RSC module includes Rectangular Self-calibrating Attention (RCA), Batch Normalization, and MLP. Rectangular Self-calibrating Attention (RCA) is the core of the RSC module. RCA first uses horizontal and vertical pooling to capture the axial global context in two directions and generates two different axis vectors. By applying broadcast addition to these two axis vectors, RCA can effectively model the rectangular region of interest in the feature map, as shown below:

[0095]

[0096] in and These represent horizontal pooling and vertical pooling, respectively.

[0097] To improve adaptability to different building shapes, RCA also utilizes two large-kernel strip convolutions to calibrate the attention maps in the horizontal and vertical directions, making the region of interest (ROI) closer to the building area. First, a horizontal strip convolution is used to calibrate the shape in the horizontal direction, adjusting the elements in each row to make the horizontal shape more closely resemble the building object. Then, a vertical strip convolution is used to further calibrate the shape. In this way, the attention region can adapt to complex building shapes. The above process can be represented as follows:

[0098]

[0099] in, and These represent horizontal strip convolution and vertical strip convolution, respectively. This represents batch normalization and the LeakyRuLU activation function. This represents the Sigmoid activation function.

[0100] like Figure 4 As shown in (3), the DPG head explicitly embeds the generated category information into the pixel feature space by learning a prototype. First, the DPG head projects the features output by the decoder into the category feature space. Then, the features in the category space are multiplied by the features in the pixel space to obtain the dynamic prototype. The process of generating the dynamic prototype can be represented as:

[0101]

[0102] in, Features representing the input, Represents linear projection. This represents the generated dynamic prototype. The prototype is then compressed by a fully connected layer and a softmax layer. The dimension is calculated to obtain a class embedding vector. This class embedding vector represents global information for each class. To project the class embedding vector onto the pixel feature space, the class embedding vector is multiplied by its transpose prototype, resulting in... The class embedding vector is then fed into a fully connected layer for class information enhancement. Through feature enhancement, this vector gains a stronger ability to distinguish between different classes. The calculation process of the class embedding vector is summarized as follows:

[0103]

[0104] in, Represents a fully connected layer. Represents matrix multiplication. This represents the enhanced category embedding vector.

[0105] at last, The pixels are fed into a layer normalization layer and a ReLU activation function layer, where they are weighted and represented as follows:

[0106]

[0107] in, Represents pixel features, Representation layer normalization, This represents the activation function. Indicates broadcast Hadamard product. This represents the output feature. Subsequently, the classification feature is fed into a convolutional layer to predict the building segmentation mask.

[0108] like Figure 5As shown, the multimodal image fusion module is a network similar to U-Net. It employs a dual-branch encoder structure to extract multi-level features from Sentinel-1 SAR images and Sentinel-2 optical images respectively, and uses a single decoder to fuse the multi-scale features, ultimately predicting a fused image embedding representation. Since the input image contains both global features (e.g., environment and background information) and local features (e.g., detailed texture information of buildings), a Transformer-CNN feature extraction module is designed to better model the input image features by utilizing their respective inductive biases. This invention uses the Restormer block as the Transformer network because it models global features by implementing self-attention in the channel dimension, requiring less computation. A residual network is used in the CNN network. Notably, the same Restormer-CNN module is used in both the backbone and the decoder.

[0109] After the Sentinel-1 and Sentinel-2 input images are projected into the feature space, they are fed in parallel into the Restormer-CNN block for global and local feature extraction. Features at the same spatial scale are then fed into the Multimodal Fusion Layer (MMFL) for cross-modal information interaction and fusion. The multi-level fused features are then fed into the decoder network. The Frequency Aware Multi-Scale Fusion Layer (FMSF) is responsible for enhancing the high-frequency components of the features, thereby preserving as much high-frequency edge detail information of buildings in the input image as possible.

[0110] like Figure 6 As shown, the specific steps of the multimodal fusion layer MMFL are as follows:

[0111] First, the Sentinel 1 and Sentinel 2 features generated by the backbone network are projected into a token sequence, and then projected onto the Q, K, and V matrices through a linear layer. The detailed formula for MMFL can be expressed as follows:

[0112]

[0113] in , and These are the outputs of the DIEM, the first SIEM, and the second SIEM, respectively. Specifically, The query matrix is ​​first fed into the DIEM module for difference information extraction, and the updated Q matrix is ​​then fed into the SIEM module for common information extraction. Finally, the Q matrix output by the first SIEM module is compared with... The K and V matrices are fed into a second SIEM module for further common information extraction.

[0114] In the DIEM module, the standard cross-attention mechanism is first used to extract common exploration information, which can be represented as:

[0115]

[0116] in This is a scaling factor that helps mitigate the issue of the softmax function converging to the region of minimum gradient as the dot product increases. Subsequently, common information is subtracted from the V matrix. This is to obtain the difference information between Q and V. This process can be represented as:

[0117]

[0118] in This represents the extracted difference information. Then, the difference information is injected into Q, represented as:

[0119]

[0120] in Representation layer normalization, This represents a feedforward neural network. This indicates the output of the DIEM module.

[0121] Although the DIEM module enhances the difference information, the model's ability to model the dependency on shared information is still limited. Therefore, it is necessary to further enhance the fused features by incorporating the shared information from the two images. Figure 6 (3) in the diagram illustrates the Common Information Enhancement Module (CIEM). Similar to the DIEM module, it first extracts common information through a cross-attention mechanism, and then enhances the Q matrix by adding the common information to the V matrix.

[0122] like Figure 7 As shown, the Frequency Aware Multi-Scale Fusion (FMSF) layer can be represented as:

[0123]

[0124] in This represents the low-pass filter predicted by the Adaptive Low-Pass Filter (ALPF) generator. Represents upsampling, This represents the high-pass filter predicted by the Adaptive High-Pass Filter (AHPF) generator. This represents the output features of the FMSF module. The ALPF generator addresses the boundary shift problem during upsampling by adaptively smoothing high-level features using a spatially varied low-pass filter, while the AHPF generator enhances the high-frequency boundary details of low-level features by using a spatially varied high-pass filter, resulting in more sharp edges in the predicted building features.

[0125] To effectively generate low-pass and high-pass filters, the module's input must first be... and The components are then fused and used as input to both the ALPF and AHPF generators. This process can be represented as:

[0126]

[0127] in Represents a convolutional layer. Represents the Pixel Shuffle upsampling operation. Representing the characteristics after fusion, H represents the channel compression factor, and H and W represent the height and width of the feature.

[0128] The ALPF generator attenuates high-frequency obfuscation information to mitigate boundary shift by predicting a series of dynamic low-pass filters. The ALPF generator then initializes the fused features. A low-pass filter that serves as input and predicts spatial variations. Fusion features. The input is first passed to a convolutional layer, and then undergoes a channel-level softmax normalization operation. This process can be represented as:

[0129]

[0130] in This represents the weights of the predicted low-pass filter. This represents the size of the filter kernel. The purpose of Softmax normalization is to ensure that all weights in the low-pass filter kernel are positive and sum to 1. Next, adjustments are made using the Pixelunshuffle method. The shape is changed so that both the height and width are halved, while the number of channels is increased fourfold. Then, the channels are divided into four groups, each with a low-pass filter, represented as... ,in Four sets of low-pass filters were used to filter the features. After convolution, four filtered features are obtained. These four features are then spatially recombined to obtain features upsampled by a factor of 2. This process can be described by a formula:

[0131]

[0132] Although the ALPF generator can effectively mitigate the building boundary shift caused by upsampling, high-frequency spatial details in low-level features still cannot be effectively recovered after downsampling. According to the Nyquist sampling theorem, frequencies above the Nyquist frequency will be permanently lost during downsampling. To compensate for this information loss, an AHPF generator is used to predict a high-pass filter for spatial variations and to optimize the low-level features. Filtering is performed to preserve as much as possible. High-frequency building edge detail information. The structure of the AHPF generator is as follows: Figure 7 As shown in (c). Similar to the ALPF generator, the AHPF generator will initialize the fused features. A high-pass filter that serves as input and predicts spatial variations. First, The filter kernel weights are predicted using a convolutional layer, followed by a channel-level softmax normalization operation and a filter inversion operation. This process can be represented as follows:

[0133]

[0134] in This represents the weights of the initial filter kernel for prediction. This represents the size of the filter kernel. To ensure that the final generated kernel is a high-pass filter kernel, channel-level softmax normalization is first used to obtain the low-pass filter kernel, and then the generated low-pass filter kernel is subtracted from the unit kernel. This is applied to low-level features... By applying a high-pass filter and adding residual connections, the enhanced features can be obtained:

[0135]

[0136] like Figure 8 As shown, the building height mapping module includes the following steps:

[0137] S51: Use The convolutional layer projects the mask cue onto 128 spatial dimensions, and then adds the image embedding and the mask cue element-wise. This process can be represented as:

[0138]

[0139] in express Convolutional layer This represents the GELU activation function. Representation layer normalization

[0140] S52: Self-attention mechanism of the mask;

[0141] S53: Cross-attention from mask to image embedding, where the mask serves as the query matrix;

[0142] S54: The updated mask is passed through a feedforward neural network;

[0143] S55: Cross-attention from image embedding to mask, where the image embedding serves as the query matrix;

[0144] S56: The updated image embedding is passed through a convolutional layer to obtain a predicted building height map or height level.

[0145] The specific experimental procedure is as follows:

[0146] 70% of the data samples were used for training, 10% for validation, and the remaining 20% ​​for testing. All experiments were based on the PyTorch framework and run on an NVIDIA A5000 graphics card with 24GB of VRAM. The Adam optimizer was used with an initial learning rate of 0.0001. The learning rate was halved every 10 epochs to accelerate model convergence. The epochs and batch size were set to 40 and 8, respectively. To improve the model's generalization ability, data augmentation techniques were used, including random horizontal flips, vertical flips, and random rotations.

[0147] Specific training loss function:

[0148] For building height prediction tasks, mean squared error loss (MSE) is used as the regression loss because it is widely used in building height prediction tasks. The MSE regression loss can be defined as:

[0149]

[0150] in Represents the predicted building height. Represents the height of a reference building. Represents the number of samples.

[0151] For the task of stratifying building heights, cross-entropy loss and Dice loss are combined to alleviate the problem of unbalanced distribution of building heights. This can be represented as follows:

[0152]

[0153] In multi-task networks, the uncertainty-aware loss function is used to automatically calculate the weights of the loss function for each prediction task. The initial weights for the two tasks are set to 1.0, representing equal weights. During training, the weights are updated based on the degree of uncertainty of the two tasks. The uncertainty-aware loss can be expressed as:

[0154]

[0155] in and These represent the learnable weights for the two tasks, respectively.

[0156] Figure 9 The diagram shows the building height estimation results of an example of this invention. It displays building heights and height level maps for six samples selected from different regions. As can be seen, the overall prediction results show high consistency with the reference map, indicating that the method of this invention has the ability to learn complex building height distributions. Furthermore, the prediction results have very clear edge details, reflecting fine-scale height changes from low-rise to high-rise buildings. Even small buildings can be accurately identified, demonstrating the model's powerful ability to extract detailed information. Moreover, although the building styles, densities, and height distributions differ among the three regions, this invention has achieved good performance in all three areas, demonstrating the generalization ability of the method.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0158] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for estimating building height based on multimodal image fusion, characterized in that, Includes the following steps: Step 1: Acquire high-resolution satellite imagery and building height datasets; Step 2: Construct a semantic segmentation module to extract building regions from optical images; Step 3: Perform pixel expansion on the extracted building area to obtain the building analysis area containing the shadow area of ​​the building; Step 4: Construct a multimodal image fusion module to fuse synthetic aperture radar images and optical images to obtain the fused image embedding; Step 5: Construct a building height mapping module, combining the fused image embedding with the building analysis region to predict building height and building height level; Step 6: Based on the dataset obtained in Step S1, train the constructed semantic segmentation module, multimodal image fusion module, and building height mapping module to obtain an estimation model for building height estimation.

2. The method as described in claim 1, characterized in that, In step 1, high-resolution satellite images and building height datasets are obtained based on open-source datasets, and the image resolutions in the datasets are normalized.

3. The method as described in claim 2, characterized in that, It also includes setting building height labels and building height level labels based on the spatial resolution of images in the dataset.

4. The method as described in claim 1, characterized in that, In step 1, the semantic segmentation module's processing includes: The input optical image is converted into a sequence embedding, and the encoder of the semantic segmentation module segments the image into non-overlapping blocks of varying sizes. A linear embedding layer is applied to project the feature dimensions of the sequence embedding onto the set feature dimensions. Then, hierarchical feature representations are generated through multiple cascaded Swing Transformer blocks and block fusion layers. For the output features of each Swing Transformer block, the width and height of the features are reduced by a factor of 2, while the number of channels is doubled. The hierarchical feature representation generated by the encoder is fed into the rectangular self-correction module (RSC) to adjust the attention region, and then fed into the decoder of the semantic segmentation module; The decoder expands the features of the last layer of the encoder's Swing Transformer block through a block expansion layer, and then fuses and decodes the decoded features from the previous layer and the encoded features from the encoder through multiple cascaded Swing Transformer blocks and block expansion layers. The features output by the decoder are passed through a segmentation head guided by a dynamic prototype, which outputs a segmentation mask for the building.

5. The method as described in claim 4, characterized in that, The rectangular self-calibration module includes rectangular self-calibration attention (RCA), batch normalization, and a multilayer perceptron. RCA uses horizontal and vertical pooling operations to capture the axial global context in two directions and generates two different axis vectors, and then performs broadcast addition on the two axis vectors.

6. The method as described in claim 4, characterized in that, The segmentation header guided by the dynamic prototype is: The DPG header projects the features output by the decoder onto the category feature space; The features in the category space are multiplied by the features in the pixel space to obtain a dynamic prototype, which is then compressed through a fully connected layer and a softmax layer. The dimension is used to obtain a category embedding vector, where C is the preset channel dimension; The category embedding vectors are fed into the layer normalization layer and the ReLU activation function layer, and the pixel features are weighted to obtain the classification features; Classification features are fed into a convolutional layer to predict building segmentation masks.

7. The method as described in claim 1, characterized in that, In step 3, the pixel expansion of the extracted building area is specifically performed by expanding the building pixels one pixel upwards, to the left, and to the right.

8. The method as described in claim 1, characterized in that, Multi-step 4 specifically includes: Step 41: Use the Restormer-CNN dual-branch encoder structure to extract multi-level features from SAR images and optical images respectively; Step 42: Features of the same spatial scale are then fed into the multimodal fusion layer for cross-modal information interaction and fusion; Step 43: The multi-level fused features are fed into the decoder network, and the high-frequency components of the features are enhanced through a frequency-aware multi-scale fusion layer.

9. The method as described in claim 1, characterized in that, Step 5 specifically includes the following steps: Step 51: Use The convolutional layer projects the mask cue onto a specified spatial dimension, and then adds the image embedding and the mask cue element-wise to obtain the mask; Step 52: Perform self-attention operation on the mask; Step 53: Cross-attention from mask to image embedding, where the mask is used as the query matrix to obtain the updated mask; Step 54: Pass the updated mask through a feedforward neural network, and then perform step 55 based on the mask processed by the feedforward neural network; Step 55: Cross-attention from image embedding to mask, where the image embedding is used as the query matrix, and perform step 56 based on the resulting updated image embedding; Step 56: The updated image embedding is passed through a convolutional layer to obtain the predicted building height map and height level map.

Images