A high-resolution remote sensing image building extraction method and system based on multi-level multi-modal feature fusion

Abstract

The application discloses a high-resolution remote sensing image building extraction method and system based on multi-level multi-modal feature fusion, and relates to the technical field of remote sensing image processing. The application adopts double modal input of RGB image and nDSM elevation data, extracts features through a double-flow independent encoder, obtains multi-level boundary priori by using an overall edge detection module, and realizes adaptive weighted feature fusion in combination with a multi-modal feature fusion module. The decoder adopts an improved lightweight residual structure, and repairs problems such as fuzzy building boundary and smooth corner through a boundary enhancement module and a forward feature replacement mechanism. Finally, a high-precision extraction result is output through post-fusion. The application also discloses a corresponding system and a storage medium. Experiments show that the application can effectively inhibit shadow and shielding interference, has clear boundaries and low calculation amount, and is superior to existing methods in IoU and F1 score on the ISPRS standard data set, and is suitable for large-scale high-resolution remote sensing image building automatic extraction.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, specifically relating to a method and system for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion. Background Technology

[0002] With the rapid development of remote sensing technology, acquiring high-resolution remote sensing imagery has become increasingly convenient. The ability to quickly and accurately extract building information from high-resolution remote sensing imagery has extremely important practical significance and application value in urban planning, map updating, disaster assessment, and 3D city modeling.

[0003] In recent years, deep learning methods, represented by convolutional neural networks (CNNs), have achieved remarkable results in the task of building extraction from remote sensing imagery. However, existing building extraction methods still face the following major technical challenges and bottlenecks:

[0004] First, most existing deep learning networks primarily rely on a single optical RGB image as input. While high-resolution RGB images contain rich texture and spectral information, the appearance, color, and roof materials of buildings are highly complex and easily affected by changes in lighting, shadows, and tree occlusion. Furthermore, a single optical image lacks elevation information, making it difficult for the network to distinguish buildings from non-elevational features with similar spectral characteristics (such as level roads, parking lots, etc.).

[0005] Secondly, to compensate for the limitations of single optical images, researchers have begun to introduce elevation data such as digital surface models (DSMs) or normalized digital surface models (nDSMs) as auxiliary modalities. However, existing multimodal data fusion methods typically only perform channel stitching at the network input or simple post-fusion at the network output. Since the acquired nDSM data is often noisy and not sufficiently clear, this simplistic fusion approach cannot adaptively explore the complementarity between the two modalities at different scales, and fails to fully utilize the clear detail features in RGB imagery to guide and compensate for the shortcomings of nDSM features, resulting in limited fusion efficiency and extraction accuracy.

[0006] Finally, in existing encoder-decoder-based building segmentation models, the network needs to perform multiple downsampling operations to extract high-level semantic features, which inevitably leads to the loss of spatial details and boundary information. Even if upsampling is used for recovery during the decoding stage, the final output building segmentation mask often suffers from problems such as blurred boundaries, irregularities, or boundary defects, making it difficult to meet the requirements of high-precision mapping.

[0007] Therefore, there is an urgent need for a high-resolution remote sensing image building extraction method that can effectively integrate the advantages of multimodal data, adaptively eliminate noise interference, and accurately repair building boundary defects. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion, thereby improving the effectiveness of building extraction.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion includes the following steps:

[0011] Step S1: Acquire RGB images of high-resolution remote sensing data. and normalized digital surface model data The data are input into the RGB encoder and the nDSM encoder respectively for multi-level feature extraction to obtain the final features. and ;

[0012] Step S2: Add an overall edge detection module to each level of the RGB encoder to extract multi-level boundary feature maps. And generate a fusion boundary feature map. ;

[0013] Step S3: Using a multimodal feature fusion module, perform mid-term multimodal fusion on the corresponding layer features of the RGB and nDSM encoders to generate mid-term fused features. ;

[0014] Step S4: Convert the final features output by the RGB encoder The input is upsampled in an RGB decoder, and the final output is a feature map containing building segmentation mask information. ;

[0015] Step S5: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input is upsampled to the nDSM decoder and then... It connects to the corresponding level decoder and finally outputs the nDSM decoding features. ;

[0016] Step S6: Use the boundary reinforcement module to... and The algorithm combines and utilizes a positive feature replacement mechanism to repair building boundary defects, outputting enhanced boundary features. ;

[0017] Step S7: [The sentence is incomplete and requires more context to be translated accurately.] and The final post-processing multimodal fusion is performed to output the predicted results of building extraction. .

[0018] Further optimization is needed. In traditional building extraction tasks, relying solely on the backbone mask segmentation of the network often faces technical shortcomings such as blurred boundaries, rounded building corners, and loss of detail. Furthermore, while the Normalized Digital Surface Model (nDSM) provides elevation information, it lacks spectral features and precise boundary details, making it difficult to directly extract clear building boundaries. To fully explore and enhance the rich physical edge structure in high-resolution RGB imagery, this application specifically introduces a global edge detection module into the RGB encoder. The core principle of this mechanism is to extract building boundaries as an independent and crucial prior feature, sharing information with the segmentation mask, thereby providing sufficient guidance for generating smooth and sharp building segment boundaries at the end of the entire network. Specifically, this includes:

[0019] Step S21: The overall edge detection module extracts the last feature map in the i-th layer of the RGB encoder, and then... The convolutional layer compresses it into a 1-channel feature map, and bilinear upsampling is used to restore the feature map size to the size of the input image, resulting in... ,in ;

[0020] Step S22: Cascade all Generate a fused boundary feature map containing sufficient information about the target boundary. Formula (1) is as follows:

[0021] (1)

[0022] in, This indicates a feature cascade operation. It refers to the number of levels that HED has. .

[0023] Further optimization is needed in multimodal remote sensing building extraction tasks. While high-resolution RGB imagery contains rich spectral and texture details, it is highly susceptible to interference from building shadows and occlusions. While nDSM data provides elevation and location contours, it lacks spectral features and is often accompanied by noise, making it difficult to extract clear boundaries independently. Traditional feature fusion methods (such as direct cascading or element-wise addition) treat these as equal contributors, and forced fusion easily leads to noise interaction between data and insufficient feature extraction. To overcome this technical deficiency, this invention constructs a multimodal feature fusion module horizontally across the data while extracting features using a dual-stream encoder. This breaks the traditional equal-weight fusion mechanism by adaptively exploring and calculating the utility weights of each channel of RGB and nDSM using global context information. High-quality features (usually RGB) dominate the prediction, while auxiliary features (nDSM) compensate for deficiencies, thereby achieving complementary advantages of multimodal features while limiting noise propagation. Specifically, the following steps are included:

[0024] Step S31: For the RGB and nDSM encoders, the first... Feature images output at each level , Global average pooling is performed on each image to obtain the global feature vector of the RGB image with C channels. and nDSM global feature vector The calculation formula (2) for global pooling is:

[0025] (2)

[0026] Among them, y c This represents the global feature vector of the c-th channel. For the height of the feature image, For the feature image width, The number of feature image channels, and These are pixel coordinates; This represents the pixel feature value at coordinates (m, n) in the c-th channel;

[0027] Step S32: Transfer the global feature vector and The input is fed into the compression-excitation block SE to explore the correlation between channels, and then activated by the Sigmoid function to obtain a weight vector between 0 and 1. Formulas (3) and (4) are as follows:

[0028] (3)

[0029] (4)

[0030] in, This represents the compression-excitation block SE. express Convolution operation, ReLU is the modified linear unit activation function; This represents the Sigmoid activation function;

[0031] Step S33: Process the feature image and Using weight vectors respectively Perform weighted summation and add element by element to generate the final result. Hierarchical mid-term fusion features Formulas (5), (6), and (7) are as follows:

[0032] (5)

[0033] (6)

[0034] (7)

[0035] in, This represents the weighted RGB feature image. This represents the weighted nDSM feature image. This indicates an element-wise multiplication operation.

[0036] Further optimization is needed. During the upsampling feature reconstruction process in the multimodal network decoding stage, in order to deeply fuse deep high-level semantic information with shallow spatial detail information, the network typically relies on improved pyramid-shaped residual connection units (RCUs) to improve the effectiveness of feature transfer. However, when processing high-resolution remote sensing images, traditional residual networks suffer from an extremely large parameter burden and huge computational cost due to the extensive use of standard convolution operations. This not only consumes a lot of GPU memory resources but also severely restricts the computational efficiency of the model in large-scale building extraction tasks. Based on this, this invention introduces a pyramid-shaped residual connection unit based on depthwise separable convolution into the decoding stream, breaking the computational barrier of traditional convolution that simultaneously extracts features in both spatial and channel dimensions. It completely decouples standard convolution into two independent operations: first, spatial feature mapping is performed on each input channel individually through depthwise convolution, and then 1×1 point convolution is used to achieve cross-layer information fusion in the channel dimension. This feature reconstruction method, which separates space and passageway, combined with a pyramid-shaped residual connection structure, greatly increases the diversity of original features, enabling key building semantic information to be preserved to the greatest extent from the deep layer and smoothly transmitted to the shallow layer.

[0037] More importantly, by introducing DSC for lightweight reconstruction, the module's parameter count and computational cost have been drastically reduced compared to traditional standard convolution, decreasing by a factor of (1 / n + 1 / k²) (where k is the kernel size of the depthwise convolution and n is the number of output channels of the pointwise convolution). This mechanism, while ensuring the decoder's ability to recover high-precision building masks, fundamentally offloads the enormous computational burden, perfectly achieving the optimal balance between model extraction performance and underlying computational efficiency.

[0038] The specific process of the RGB decoder in step S4 is as follows:

[0039] Step S41: Upsample the output features of the previous level of the RGB decoder to obtain the upsampled features of the current level. ; Receive the lateral output features of the skip connections at the corresponding levels of the RGB encoder ;Will and Perform cascading to generate cascaded features for the current level. ;

[0040] Step S42: Concatenate features The input is processed by the improved pyramid-shaped residual connection unit (RCU) and the output is the decoded feature. Formula (8) is:

[0041] (8)

[0042] in, This represents the current level of cascaded features in the input; BN(⋅) represents the batch normalization operation; DSC(⋅) represents the depthwise separable convolution operation.

[0043] Step S43: After being passed and processed layer by layer in reverse by each level of the RGB decoder, the final output layer contains features containing building segmentation mask information. .

[0044] Further optimization is needed. In a two-stream multimodal architecture, while the normalized digital surface model (nDSM) can provide crucial spatial height and contour information for building extraction, its inherent lack of rich spectral and texture features means that elevation features extracted independently from the nDSM data stream are often accompanied by noise and have unclear boundaries. If the traditional homologous skip connection (i.e., directly passing noisy nDSM encoded features across layers to the decoder) is still used in the decoding stage, it not only fails to provide clear structural priors but also easily leads to error amplification and noise accumulation. Based on this, this invention designs a "heterogeneous skip connection" mechanism in the nDSM decoding branch: utilizing complementary features output by the multimodal fusion module, dominated by high-quality RGB features. This replaces the original nDSM encoder's row-to-layer propagation, thereby precisely guiding the removal of nDSM noise features. The specific process is as follows:

[0045] Step S51: Upsample the output features of the previous level of the nDSM decoder to obtain the upsampled features of the current level. ;Will With mid-term fusion characteristics Generate multimodal cascaded features for the current level. ;

[0046] Step S52: Concatenate multimodal features The input is processed by the improved pyramid-shaped residual connection unit (RCU) and the output is the decoded feature. Formula (9) is:

[0047] (9)

[0048] Step S53: After the decoder layers pass through and process each layer in reverse, the nDSM decoding features are finally output at the last layer. .

[0049] Further optimization is needed in high-resolution remote sensing image building extraction tasks, relying solely on the segmentation mask (such as feature map) directly predicted by the backbone decoding network. While capable of locating the entire area of ​​a building, this method generally suffers from technical shortcomings such as blurred boundaries, rounded corners, and incomplete coverage of building footprints. To overcome these problems, this invention introduces a boundary enhancement module at the end of the network. Its core principle lies in using the previously independently extracted high-precision boundary features as spatial prior knowledge, and performing parallel deep interaction with coarse mask features; by constructing a merging probability map and a positive feature replacement mechanism, the defined boundaries are used to "crop" and "fill in" the blurred mask, thereby outputting a high-precision mask image of the building with smooth contours and sharp boundaries. Specifically, it includes the following steps:

[0050] Step S61: Fuse the boundary feature maps In the boundary detection subunit of the input boundary enhancement module BE, the boundary is first processed by the Sigmoid activation function to generate a 5-channel basic boundary feature x mapped between 0 and 1, as shown in formula (10):

[0051] (10)

[0052] Then, the basic boundary feature x is processed in two ways:

[0053] The first path, after dimensionality upscaling via a 1×1 convolutional layer and processing with the ReLU activation function, yields shared boundary features. It is directly passed to the segmentation mask detection subunit, and the calculation formula (11) is:

[0054] (11)

[0055] The second path will generate shared boundary features. The input is fed into the compression-activation block SE, and the output is reduced to 5 channels of attention weights. These attention weights are then multiplied element-wise with the preceding 5-channel basic boundary features x to enhance the boundary information. The enhanced features are then compressed into 1 channel through a 1×1 convolutional layer to generate the final boundary probability map. The calculation formula (12) for this process is:

[0056] (12)

[0057] Where Sigmoid(⋅) and ReLU(⋅) are activation functions, Block(⋅) represents the compression-activation block operation, and Conv1×1(⋅) represents the 1×1 convolution operation. This represents the element-wise multiplication operation;

[0058] Step S62: Segmentation mask feature map With shared boundary features In the segmentation mask detection subunit of the synchronous input boundary enhancement module (BE), a cascading operation is used to... and The features are merged, and the merged composite features are processed sequentially through a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer to finally generate a 1-channel segmentation mask probability map. The calculation formula (13) is as follows:

[0059] (13)

[0060] Where Conv3×3(⋅) represents a 3×3 convolution operation, and Conv1×1(⋅) represents a dimension reduction and compression operation to generate the final 1-channel mask probability map;

[0061] Step S63: ... and Probability maps were obtained by applying the Sigmoid activation function. and Then, they are added pixel by pixel to obtain the merged probability map. Formula (14) is as follows:

[0062] (14)

[0063] in, This represents the activation result of the boundary probability map; This represents the activation result of the mask probability map; min(⋅) represents the minimum value function;

[0064] Step S64: Obtain the feature values ​​of the original segmentation mask The positive response portion of the feature value is extracted using the ReLU activation function, and the average value of this positive response portion is calculated over the entire feature map to obtain the scaling factor. The calculation formulas (15) and (16) are as follows:

[0065] (15)

[0066] ReLU( )=max(0, (16)

[0067] in, For feature map The eigenvalues; N is the feature map. The total number of pixels; k is the label of the pixel coordinates; This represents the feature value of the k-th pixel in feature map P2;

[0068] Step S65: Strip the feature values ​​of the original segmentation mask The positive response portion is extracted, and only the negative response portion is retained; subsequently, the merged probability plot P is used. S (x) and scaling factor The proportional probability mapping is used to replace the original positive response portion, ultimately yielding the boundary-enhanced output features after boundary defect repair. The calculation formula (17) is as follows:

[0069] (17).

[0070] Further optimization reveals that, in the forward propagation of the multimodal network, although the network has achieved feature-level complementarity through the multimodal feature fusion module in the mid-stage, the RGB decoding branch and the nDSM decoding branch still retain their unique modal advantages in the final high-level semantic space. Specifically, the RGB features after boundary enhancement (BE) processing... It possesses extremely high edge sharpness and rich spectral texture information, but due to the physical limitations of optical imaging, it may still be slightly affected by the shadows of buildings themselves in extremely complex scenes; while nDSM decoding features Although lacking in texture detail, it possesses extremely strong resistance to lighting interference and absolute spatial elevation contour prior. Therefore, this embodiment introduces a decision-level post-fusion mechanism at the end of the network, aiming to further complement and spatially calibrate the "high-precision boundary features" and "highly robust elevation features" at the highest semantic dimension before the final output, thereby minimizing omissions and misclassifications in the extraction of large-area buildings. The specific process of multimodal fusion is as follows:

[0071] Step S71: ... and The input is summed pixel-by-pixel in the post-fusion unit to obtain the post-fusion features. ;

[0072] Step S72: ... The input is fed into the final classification prediction layer, and after activation by the Sigmoid activation function, the final output is a prediction result containing building probability mappings. .

[0073] A high-resolution remote sensing image building extraction system based on multi-level multimodal feature fusion includes:

[0074] The data acquisition and preprocessing module is used to acquire high-resolution remote sensing RGB images and normalized digital surface model (nDSM) data, perform sliding window cropping, spatial alignment and label reconstruction on the images, and output standardized multimodal input data.

[0075] The dual-stream coding module includes an independent RGB encoder and an nDSM encoder, which are used to extract multi-level downsampling features from RGB images and nDSM data respectively, and output RGB encoded features and nDSM encoded features.

[0076] The boundary extraction module (HED) is connected to each level of the RGB encoder. It is used to extract multi-level boundary features and perform cross-level feature concatenation to output a fused boundary feature map.

[0077] The intermediate fusion module is connected to the output of each level of the RGB encoder and the nDSM encoder, and is used to perform global pooling, SE attention weighting and adaptive feature fusion on the RGB features and nDSM features of the same level, and output the intermediate fusion features of each level.

[0078] The dual-stream decoding module includes an RGB decoder and an nDSM decoder. The RGB decoder is used to upsample, skip-connect, and improve pyramid residual unit processing of RGB encoded features to output a building segmentation mask feature map. The nDSM decoder is used to access intermediate fusion features, perform heterogeneous skip-connect and upsampled decoding, and output nDSM decoded features.

[0079] The boundary enhancement module is connected to the HED boundary extraction module and the RGB decoder output, respectively. It is used to fuse boundary features with segmentation mask features and repair building boundary defects through a positive feature replacement mechanism, and output boundary enhancement features.

[0080] The post-fusion and prediction module is used to fuse the boundary enhancement features and nDSM decoding features pixel by pixel. After processing by the classification prediction layer and activation function, the final building extraction probability map is output.

[0081] The system is used to execute the above-described method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion.

[0082] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] 1. The method described in this invention, by fusing RGB imagery with nDSM elevation data, compensates for the deficiencies of single optical images in lacking elevation information and being susceptible to interference from shadows and occlusions. It can accurately distinguish buildings from road, square and other ground features with similar spectra, reducing misclassification and omission.

[0085] 2. The method described in this invention adopts a multi-level adaptive weighted fusion mechanism to replace the traditional simple splicing or end fusion, which can explore modal complementarity at different levels, suppress noise transmission, and improve fusion efficiency and feature discriminability.

[0086] 3. The method described in this invention uses HED multi-level boundary extraction and BE module positive feature replacement mechanism to accurately repair defects such as blurred boundaries, rounded corners, and internal voids, and restore the sharp edges and complete outline of buildings to meet the requirements of high-precision mapping.

[0087] 4. The decoder described in this invention uses depth-separable convolutional improved residual units, which significantly reduces the number of parameters and computational costs while ensuring the transmission of multi-scale information, and is suitable for fast inference of large-scale high-resolution remote sensing images.

[0088] 5. The method described in this invention can output complete segmentation results without complex post-processing, and with seamless inference using a sliding window, it can be directly used in practical application scenarios such as urban planning, map updating, and disaster assessment. Attached Figure Description

[0089] Figure 1 A flowchart illustrating the overall main process of a high-resolution remote sensing image building extraction method based on multi-level multimodal feature fusion, provided in this embodiment of the invention.

[0090] Figure 2 This is a structural diagram of a multi-level, multimodal MMFNet network provided in an embodiment of the present invention;

[0091] Figure 3 A structural diagram of the overall edge detection (HED) module is provided for embodiments of the present invention;

[0092] Figure 4 A structural diagram of the multimodal feature fusion (MFF) module is provided for embodiments of the present invention;

[0093] Figure 5 A structural diagram of an RGB decoder with an improved pyramid-shaped residual connection unit (RCU) is provided for embodiments of the present invention;

[0094] Figure 6 A diagram of an nDSM decoder structure with an improved pyramid-shaped residual connection unit (RCU) is provided for embodiments of the present invention.

[0095] Figure 7 This is a structural diagram of the boundary enhancement (BE) module provided in an embodiment of the present invention;

[0096] Figure 8 A flowchart illustrating the positive feature replacement mechanism in the boundary enhancement (BE) module provided in this embodiment of the invention;

[0097] Figure 9 A schematic diagram of a large-scale, high-resolution color remote sensing image (RGB) input during the testing phase provided in an embodiment of the present invention;

[0098] Figure 10 A schematic diagram of a large-scale normalized digital surface model (nDSM) input for the testing phase provided in an embodiment of the present invention;

[0099] Figure 11 This is an output image showing the predicted distribution of large-scale buildings during the testing phase, provided in an embodiment of the present invention.

[0100] Figure 12 The results show the ablation experiments of different modules on the Vaihingen and Potsdam datasets. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are one embodiment of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0102] Example 1:

[0103] like Figure 1 As shown, a method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion specifically includes the following steps:

[0104] Step S1: Acquire multimodal image data and construct a binary classification dataset of real building labels, specifically including:

[0105] Step S1.1: Obtain a high-resolution publicly available remote sensing dataset provided by the International Society for Photogrammetry and Remote Sensing (ISPRS) committee, which includes the Vaihingen dataset and the Potsdam dataset.

[0106] In this embodiment, the Vaihingen dataset contains 33 orthophotos with a spatial resolution of 9 cm and their corresponding normalized digital surface model (nDSM) data; the Potsdam dataset contains 38 orthophotos with a spatial resolution of 5 cm (using red, green, and blue bands) and their corresponding nDSM data.

[0107] Step S1.2: Extract the ground truth labels attached to the multimodal raw data. For the six land cover categories (i.e. impervious surface, buildings, low vegetation, trees, cars and others) contained in the original labels, reconstruct them according to the task requirements. The "buildings" category in the original labels is retained, and the other five categories are uniformly merged and masked as "background", thereby transforming the six-class labels into binary true labels containing only buildings and background.

[0108] Step S1.3: Based on the reconstructed binary classification true labels of buildings and the corresponding multimodal image data, the dataset is divided; specifically, 16 images are selected from the Vaihingen dataset as the training set and the remaining images are selected as the test set; at the same time, 24 images are selected from the Potsdam dataset as the training set and the remaining tiles are selected as the test set.

[0109] Step S2: Data Preprocessing: Given the large original size of the acquired multimodal remote sensing images, direct input into a deep neural network is easily limited by GPU memory. This embodiment introduces a sliding window mechanism to dynamically collect and expand training samples. Specifically, the extraction size of the sliding window is set to 512×512 pixels. To obtain sufficient training data and ensure the continuity of slice edge features, a 30% overlap is maintained between two consecutive adjacent sliding windows. The divided training set is traversed and segmented according to the size of the sliding window and the overlap step size. During the segmentation process, the consistency of the high-resolution RGB image, normalized digital surface model (nDSM) data, and the corresponding binary classification real labels of buildings in spatial location is strictly maintained, and synchronous cropping operations are performed. Finally, a large number of uniformly sized and strictly aligned multimodal image patches and label pairs are output, which are directly used as the standard input data for subsequent construction and training of the multi-level multimodal fusion network (MMFNet).

[0110] Step S3: Construct the MMFNet building extraction network with the Multimodal Feature Fusion (MFF) module and the Boundary Enhancement (BE) module as its core.

[0111] like Figure 2 As shown, the network mainly consists of a dual-stream encoder, a global edge detection (HED) module, a multimodal feature fusion (MFF) module, a dual-stream decoder, a boundary enhancement (BE) module, and a post-multimodal fusion module.

[0112] The specific network construction and feature transfer process is as follows:

[0113] Step S3.1: Dual-stream encoder feature extraction: Extracting RGB images from high-resolution remote sensing imagery. and normalized digital surface model data The data are input separately into independent RGB encoders and nDSM encoders for multi-level feature extraction. For example... Figure 2 As shown, both encoders contain 5 levels (level 1 to level 5). After layer-by-layer downsampling and feature extraction, the deep features are finally obtained, denoted as follows: and .

[0114] Step S3.2: HED Multi-Level Boundary Extraction: In the RGB encoder, to fully extract the edge information of buildings, a High-Edge Edge Detection (HED) module is added to each level (1 to 5) of the RGB encoder. Multi-level boundary feature maps are extracted using the HED module. Subsequently, a cascading operation is used to combine the feature maps from all levels. The merging process generates a fused boundary feature map containing sufficient information about the target boundary. .

[0115] like Figure 3 As shown, the specific process is as follows:

[0116] 1) The HED module extracts the last feature map in the i-th layer of the RGB encoder, through... The convolutional layer compresses it into a 1-channel feature map, and uses bilinear upsampling to restore the feature map size to the size of the input image, denoted as . There are five layers in the network.

[0117] 2) The feature map after restoring its dimensions is obtained through cascading operations. Merge to generate a fused boundary feature map containing sufficient information about the target boundary. , H1=Concatenate(F1,F2,F3,F4,F5).

[0118] Step S3.3: While extracting features from the dual-stream encoder, a multimodal feature fusion module is constructed to perform mid-level multimodal fusion on the features output from corresponding layers of the RGB encoder and nDSM encoder. Through layer-by-layer modal information interaction, mid-level fused features for the corresponding layers are generated. .

[0119] like Figure 4 As shown, the specific implementation process of the multimodal feature fusion module is as follows:

[0120] 1) For the RGB and nDSM encoders, the first The feature images output from each level are denoted as follows: and Global average pooling is used to aggregate the global information of both, resulting in a global feature vector of RGB image with C channels. and nDSM global feature vector .

[0121] 2) Global feature vector and The input is fed into the compression-excitation block (SE) to explore the correlation between channels, and the resulting convolution output is activated using the Sigmoid function. The weight vector, restricted to 0 and 1, is calculated using formulas (3) and (4). .

[0122] 3) Feature images and Using weight vectors respectively Perform weighted summation and add element by element to generate the final result. Hierarchical mid-term fusion features The calculation formulas are as follows (5), (6), and (7).

[0123] S3.4: The final feature output by the RGB encoder The input is fed into an RGB decoder for upsampling and feature reconstruction. After layer-by-layer reverse propagation and processing, the final output layer contains a feature map containing preliminary building segmentation mask information. .

[0124] like Figure 5 As shown, the specific process of the decoder based on the improved pyramid-shaped residual connection unit (RCU) is as follows:

[0125] 1) Upsample the output features of the previous level of the RGB decoder to obtain the upsampled features of the current level, denoted as . ; Receive the lateral output features of the skip connections at the corresponding levels of the RGB encoder, denoted as ; will the and Perform cascading to generate the cascading feature of the current level, denoted as... .

[0126] 2) Cascade features The input is processed by the improved pyramid-shaped residual connection unit (RCU), and the decoded features are output using formula (8). .

[0127] 3) After being passed and processed layer by layer in reverse by the RGB decoder, the final output layer contains a feature map containing building segmentation mask information. .

[0128] Step S3.5: Convert the final features output by the nDSM encoder The input is then upsampled in the nDSM decoder. During this decoding process, intermediate fusion features are incorporated. The features are concatenated with the corresponding level of the nDSM decoder features, thereby introducing supplementary information from the RGB branch to guide the recovery of the nDSM features, and finally outputting the nDSM decoded features. .

[0129] like Figure 6 As shown, the specific implementation process is as follows:

[0130] 1) Upsample the output features of the previous level of the nDSM decoder to obtain the upsampled features of the current level, denoted as . , will the Intermediate fusion features of the corresponding level Perform cascading to generate multimodal cascaded features for the current level. .

[0131] 2) Concatenate multimodal features The input is processed in the improved pyramid-shaped residual connection unit (RCU); the decoded features are output using formula (9). .

[0132] 3) After being passed and processed layer by layer in reverse by each level of the decoder, the nDSM decoding features are finally output at the last level. .

[0133] Step S3.6: To address the common issues of blurred or incomplete boundary extraction in building extraction, a boundary enhancement (BE) module is used. This involves fusing the boundary feature maps. With segmentation mask feature map The data is synchronously input into the BE module for integration. Utilizing both mask and boundary information, along with a positive feature replacement mechanism, building boundary defects are repaired, ultimately outputting enhanced boundary features. .

[0134] like Figure 7 As shown, the specific implementation process is as follows:

[0135] 1) Fuse the cascaded 5-channel boundary feature map The input is fed into the boundary detection subunit in parallel within the BE module. First, it is processed by the Sigmoid activation function to generate a 5-channel basic boundary feature x mapped between 0 and 1.

[0136] Then, the basic boundary feature x is processed in two ways:

[0137] The first path, after dimensionality upscaling via a 1×1 convolutional layer and processing with the ReLU activation function, yields shared boundary features. It is directly passed to the segmentation mask detection subunit.

[0138] The second path will generate shared boundary features. The input is fed into the compression-activation block SE, and the output is reduced to 5 channels of attention weights. These attention weights are then multiplied element-wise with the preceding 5-channel basic boundary features x to enhance the boundary information. The enhanced features are then compressed into 1 channel through a 1×1 convolutional layer to generate the final boundary probability map. The calculation formula for this process is shown in equation (12).

[0139] 2) Segmentation mask feature map With shared boundary features The input is synchronously fed into the segmentation mask detection subunit of the BE module; through cascading operations, it is then... and The data is merged to fully utilize the dual information from the two branches (i.e., the location information of the backbone mask and the precise boundary prior information). The merged composite features are then processed sequentially through a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer to finally generate a 1-channel segmentation mask probability map. The calculation formula for this process is shown in equation (13).

[0140] 3) For example Figure 8 As shown, and Probability maps were obtained by applying the Sigmoid activation function. and The pixels are added together to obtain the merged probability map. When the sum of the probabilities exceeds 1, it is directly considered as 1.

[0141] 4) Obtain the feature values ​​of the original segmentation mask. (here) For feature map The eigenvalues ​​are used to extract the positive response portion of the eigenvalues ​​using the ReLU activation function, and the average value of the positive response portion over the entire feature map is calculated using formula (12) to obtain the scaling factor. .

[0142] 5) Remove the feature values ​​of the original segmentation mask. The positive response portion is extracted, and only the negative response portion is retained; subsequently, the merged probability plot P is used. S (x) and scaling factor The proportional probability mapping is used to replace the original positive response portion, ultimately yielding the boundary-enhanced output features after boundary defect repair. .

[0143] Step S3.7: Post-modal fusion: Enhance boundary features With nDSM decoding features The input is fed into the post-fusion unit for final post-modal fusion. The fused features are then processed by the classification prediction layer, outputting a final prediction result that includes building probability maps. The specific steps are as follows:

[0144] 1) and The input is fed into the post-fusion unit, where pixel-wise summation is used to merge the features of the two different modalities, generating the post-fusion features. .

[0145] 2) The input is fed into the final classification prediction layer, and after activation by the Sigmoid activation function, the final output is a prediction result containing building probability mappings. .

[0146] Step S4: Constructing the network prediction loss function by combining the real labels, and configuring the initial weights of the SGD optimizer and encoder / decoder, specifically includes the following steps:

[0147] Step S4.1: Perform differentiated initialization configuration of weights for the MMFNet multimodal dual-stream architecture. To accelerate network convergence and fully utilize the rich prior knowledge of low-level structures in natural images, the initial weights of the RGB encoder and nDSM encoder are configured as VGG-16 backbone network weights pre-trained on the ImageNet dataset. Simultaneously, a random initialization strategy is adopted for the weights of the decoder and various additional modules.

[0148] Step S4.2: At the end of the forward propagation of the network, obtain the final prediction result output by the post-fusion unit. A continuous probability distribution map with values ​​ranging from (0,1) is generated, representing the final predicted probability that each pixel belongs to a building. This predicted probability distribution map is then compared pixel-by-pixel with the real building segmentation mask labels provided by a high-resolution remote sensing image dataset, where building pixels are labeled as positive class 1 and background pixels as negative class 0. Based on the comparison results, a binary cross-entropy loss function is used to calculate the overall prediction loss of the network, accurately quantifying the pixel-level difference and cross-entropy error between the predicted final building probability distribution and the real binary labels.

[0149] Step S4.3: To minimize the overall prediction loss calculated in the above steps, the network is configured with a stochastic gradient descent (SGD) optimizer algorithm to drive the backpropagation and iterative update of the global weights. In the specific training hyperparameter configuration, the batch size is set to 10, the base learning rate is set to 0.01, and a momentum mechanism with a value of 0.9 is used to accelerate the descent of the gradient in the correct direction and effectively suppress training oscillations. Furthermore, a weight decay term with a value of 0.0005 is introduced for regularization to prevent overfitting when fitting complex building features. Based on the set hyperparameters, through multiple rounds of feature forward propagation and error backpropagation, the network continuously and dynamically adjusts various parameters until the overall prediction loss reaches convergence, ultimately outputting a high-precision multimodal building extraction network after training.

[0150] Step S5: Model training. The backpropagation algorithm is used to continuously optimize and update the MMFNet model parameters until convergence. Specifically:

[0151] Step S5.1: During each training round, the constructed 512×512 multimodal image patch sample set is synchronously input into the MMFNet dual-stream network architecture in batches according to the set batch size (BatchSize=10). RGB imagery and nDSM elevation data are spatially mapped layer by layer in their respective independent encoders; between corresponding feature layers, the network triggers the multimodal feature fusion (MFF) module to perform adaptive weighted intermediate feature interaction. Subsequently, the fused features are lightweighted and reconstructed through the improved pyramid residual connection unit (RCU) using cascaded reconstruction of homologous features, and then imperfections are repaired by the boundary enhancement module (BE). Finally, the network outputs a final predicted map representing the probability of building existence at the end of the network. .

[0152] Step S5.2: After obtaining the forward prediction results for the current batch, calculate the predicted probability distribution using the preset binary cross-entropy function. The global loss value between the binary classification and the true label is calculated. Subsequently, the network triggers a backpropagation algorithm, relying on the chain rule in calculus to propagate the scalar error signal from the output layer by layer along the reverse topology of the MMFNet. During this stage, the system accurately calculates the partial derivative matrix (i.e., gradient) of the overall loss function with respect to each learnable parameter in the network (including the weights of each standard convolutional kernel, the gate tensor parameters of the SE module in MFF, and the independent spatial and channel weights of the depthwise separable convolution in RCU), thereby quantifying the relative contribution of each micro-parameter in the network to the current prediction error.

[0153] Step S5.3: After obtaining the global gradient, the network's underlying parameters are iteratively updated using the configured stochastic gradient descent (SGD) optimizer. The specific mathematical mechanism for this underlying update is as follows:

[0154] Step size control: Using the set base learning rate (LearningRate=0.01), the step size of each parameter decreases along the negative gradient direction (i.e., the direction of error reduction).

[0155] Momentum acceleration and anti-oscillation: By incorporating a set momentum term (Momentum=0.9), the algorithm not only relies on the current gradient when updating the current parameters, but also retains the exponentially weighted moving average of the historical parameter update directions. This physical mechanism effectively suppresses gradient oscillations caused by differences in heterogeneous multimodal data and helps the model quickly escape local minima in the complex loss space.

[0156] L2 regularization penalty: A weight decay mechanism (WeightDecay=0.0005) is introduced into the parameter update formula to apply a slight penalty to the absolute weights with excessively large values ​​in the network during each back-up update. This underlying constraint mechanism suppresses redundant abrupt changes in model feature fitting, fundamentally preventing overfitting when fitting complex and variable building features in high-resolution remote sensing images.

[0157] Step S5.4: The MMFNet network continuously repeats the backpropagation process described above: "batch input → forward propagation → cross-entropy loss calculation → chain rule error backpropagation → SGD weight update." With multiple rounds of continuous iteration, the millions of parameters within the MMFNet adaptively change under the joint drive of multimodal data, and the predicted output continuously approaches the true mask boundary. The training process is continuously monitored until the overall cross-entropy prediction loss tends to a stable lower limit and no longer shows a significant decrease (i.e., the model reaches convergence). The iterative training ends, and the final output is a high-precision multimodal building extraction MMFNet model with optimal spatial mapping weights.

[0158] Step S6: Inference is performed using a 128-pixel short-step sliding window. The final building distribution is output by mean-based prediction of overlapping regions. Specifically:

[0159] Step S6.1: After the multi-level, multimodal fusion network training converges, the test inference stage for large-scale building extraction begins. This is because the high-resolution color remote sensing image to be predicted (e.g., Figure 9 (as shown) and normalized digital surface model elevation data (such as) Figure 10 The panoramic test data (as shown) covers an extremely wide area with huge pixel sizes, exceeding the processing limit of the video memory. Therefore, a sliding window technique must be used to dynamically crop the panoramic test data. To fundamentally eliminate the boundary fragmentation that easily occurs when stitching adjacent image blocks, the system specifically sets the sliding window span in the testing phase to 128 pixels (this step size is significantly shorter than the 512-pixel step size set in the training phase), while the size of the cropping window remains unchanged. This short-step sliding mechanism forces a large spatial overlap between adjacent cropped local image blocks.

[0160] Step S6.2 involves sequentially inputting the aforementioned local multimodal image patch pairs (i.e., RGB and corresponding nDSM data blocks) extracted through short-step slicing and containing significant spatial overlap into the trained multi-level multimodal fusion network according to their spatial coordinate sequence. The network utilizes the optimal parameters fixed at the bottom layer to sequentially perform forward propagation operations such as feature encoding, adaptive multimodal feature fusion, lightweight residual decoding and reconstruction, and boundary defect repair. Finally, the network independently outputs a corresponding local building prediction probability distribution map for each input local image patch, where each pixel value represents the confidence level of the point belonging to the building category.

[0161] Step S6.3: After obtaining the predicted probability distribution map of all local image patches, the system remaps and restores these local prediction results back to their corresponding positions in the panoramic image based on the original spatial coordinates recorded during slice cropping. For large-area spatial overlap regions generated by short-step sliding, the same geographic pixel may be included in multiple different local image patches, resulting in multiple independent predicted probability values. The system extracts all predicted probability values ​​corresponding to the same pixel and performs arithmetic averaging (i.e., multiple prediction averaging). Through this averaging mechanism based on overlapping regions, the algorithm can effectively neutralize the prediction bias caused by local receptive fields, thoroughly smoothing and eliminating boundary effects at the stitching seams. Finally, based on the averaged global smooth probability map and a preset classification threshold, binarization is performed, ultimately outputting a seamlessly stitched, highly accurate, and clearly defined large-scale building extraction distribution map, such as... Figure 11 As shown.

[0162] Experiments and verification:

[0163] 1. Comparative experiments and quantitative assessment

[0164] 1.1) Evaluation Indicators

[0165] To objectively and quantitatively evaluate the performance of the model in pixel-level semantic segmentation tasks, this experiment selected the core evaluation metrics recognized in the fields of remote sensing and computer vision, mainly including overall accuracy (OA), precision, recall, F1 score, and intersection-over-union ratio (IoU).

[0166] All the aforementioned advanced evaluation metrics are calculated based on the confusion matrix. Therefore, before defining the statistical criteria for each item, the fundamental pixel-level judgment criteria must first be established. Specific explanations are shown in Table 1.

[0167] Table 1. Criteria for Pixel Determination in the Basic Confusion Matrix (Taking "Building Extraction" as an Example)

[0168]

[0169] Based on the above four basic statistical results, the specific definitions, calculation formulas and evaluation significance of the quantitative evaluation indicators used in this experiment are shown in Table 2.

[0170] Table 2 Explanation of Quantitative Evaluation Indicators

[0171]

[0172] 1.2) Comparison experiment with the Vaihingen dataset

[0173] Table 3 shows the quantitative evaluation results of building extraction for each comparative model and the model of this invention (MMFNet) on the ISPRSVaihingen dataset.

[0174] Table 3. Comparison of extraction performance (%) of different models on the Vaihingen dataset

[0175]

[0176] 1.3) Potsdam dataset comparison experiment

[0177] To further verify the model's generalization ability in large scenes under different urban terrains, comparative experiments were conducted on the higher-resolution ISPRSPotsdam dataset. The quantitative evaluation results are shown in Table 4.

[0178] Table 4. Comparison of extraction performance (%) of different models on the Potsdam dataset

[0179]

[0180] 1.4) Comparison Results Analysis

[0181] Analysis of the quantitative evaluation results of the aforementioned dual datasets confirms that the technical solution of this invention has significant advantages over other existing technical solutions.

[0182] The MMFNet proposed in this invention achieves state-of-the-art building extraction performance on two publicly available datasets. On the Vaihingen dataset, MMFNet achieves IoU, OA, and F1 scores of 91.31%, 96.69%, and 95.07%, respectively, representing improvements of at least 0.96%, 0.48%, and 0.20% compared to the other five state-of-the-art methods. On the Potsdam dataset, which has a larger coverage area and richer details, MMFNet achieves excellent results of 93.68% (IoU), 97.90% (OA), and 96.70% (F1). Compared to comparable networks such as v-FuseNet, its IoU improvement ranges from 0.69% to 13.59%.

[0183] It is worth noting that the IoU metric of this invention outperforms other similar methods by nearly 1% or even more on both datasets. This improvement in core data demonstrates that the overall edge detection (HED) and boundary enhancement (BE) mechanisms introduced at the network end of this invention have successfully achieved the expected performance, effectively constraining the predicted contours of buildings, significantly increasing the intersection area between the predicted mask and the real mask, and overcoming the problem of blurred boundaries extracted by traditional methods.

[0184] 2. Ablation experiment of core module:

[0185] 2.1) Ablation Experiment Design

[0186] The ablation experiments followed the core principle of the controlled variable method. Based on the complete multi-level multimodal fusion network (MMFNet) proposed in this invention, the core modules were gradually stripped or replaced, resulting in the construction of four comparative network variants, as shown in Table 5.

[0187] Table 5. MMFNet Core Module Ablation Experiment Configuration Instructions

[0188]

[0189] 2.2) Ablation experiments on the Vaihingen dataset

[0190] On the Vaihingen dataset, the four network variants were trained and evaluated in a unified environment, and the changes in the objective performance indicators are shown in Table 6.

[0191] Table 6 Ablation Quantitative Assessment Results of Different Modules in the Vaihingen Dataset

[0192]

[0193] 2.3) Potsdam dataset ablation experiment

[0194] Similarly, the quantitative evaluation results of ablation validation performed on the Potsdam dataset, which has a more complex environment and higher resolution, are shown in Table 7.

[0195] Table 7 Ablation Quantitative Assessment Results of Different Modules in the Potsdam Dataset

[0196]

[0197] 2.4) Module Contribution Analysis

[0198] Combining the above quantitative data reports and the visual presentation of the extraction results, the irreplaceable and significant contributions of the two core modules of this invention can be clearly analyzed:

[0199] The technical contribution and visual validation of the boundary enhancement (BE) module demonstrate that network performance significantly degrades once the BE module (i.e., the MMFNet-BE model) is removed. Compared to MMFNet-BE, the complete MMFNet with the BE module significantly improves IoU, OA, and F1 scores by 2.28%, 1.03%, and 0.96% respectively on the Vaihingen dataset; on the Potsdam dataset, these three metrics show even greater improvements of 0.35%, 0.38%, and 0.21%, greatly validating the engineering effectiveness of this module. Combined with visual extraction analysis (such as... Figure 12 As shown in the figure, it can be clearly observed that the MMFNet_based and MMFNet-BE variants without the BE module exhibit a large number of misclassifications and irregularities at the building boundaries (such as...). Figure 12 (See boxes 2, 5, 10, and 12). This invention, through its boundary enhancement (BE) module, successfully and effectively distinguishes building edges from adjacent features, playing a decisive role in the smoothing and regularization of boundaries.

[0200] Technical Contributions and Visual Validation of the Multimodal Feature Fusion (MFF) Module: Regarding the fusion mechanism, data shows that traditional pixel-by-pixel cascading operations have significant limitations. Compared to the MMFNet-MFF variant that only uses Concatenate to merge data, this invention, by introducing the MFF module, improves IoU, OA, and F1 scores by 2.47%, 1.11%, and 1.06% respectively on the Vaihingen dataset; and shows steady increases of 0.53%, 0.67%, and 0.28% on the Potsdam dataset. From a visual validation perspective, if ordinary cascading operations (such as MMFNet_based and MMFNet-MFF) are used directly, the model will exhibit large-scale omissions when extracting low-rise buildings or buildings heavily occluded by shadows (e.g., ...). Figure 12 (Clearly visible in boxes 1-7, 9, and 11). The MFF module proposed in this invention can deeply extract complementary information between RGB images and nDSM elevation modes, effectively combating image illumination and shadow interference and inherent elevation noise points, completely solving the technical pain point of large-area omissions, and improving the overall integrity of building extraction results.

[0201] Example 2:

[0202] A high-resolution remote sensing image building extraction system based on multi-level multimodal feature fusion includes:

[0203] The data acquisition and preprocessing module is used to acquire high-resolution remote sensing RGB images and normalized digital surface model (nDSM) data, perform sliding window cropping, spatial alignment and label reconstruction on the images, and output standardized multimodal input data.

[0204] The dual-stream coding module includes an independent RGB encoder and an nDSM encoder, which are used to extract multi-level downsampling features from RGB images and nDSM data respectively, and output RGB encoded features and nDSM encoded features.

[0205] The boundary extraction module (HED) is connected to each level of the RGB encoder. It is used to extract multi-level boundary features and perform cross-level feature concatenation to output a fused boundary feature map.

[0206] The intermediate fusion module is connected to the output of each level of the RGB encoder and the nDSM encoder, and is used to perform global pooling, SE attention weighting and adaptive feature fusion on the RGB features and nDSM features of the same level, and output the intermediate fusion features of each level.

[0207] The dual-stream decoding module includes an RGB decoder and an nDSM decoder. The RGB decoder is used to upsample, skip-connect, and improve pyramid residual unit processing of RGB encoded features to output a building segmentation mask feature map. The nDSM decoder is used to access intermediate fusion features, perform heterogeneous skip-connect and upsampled decoding, and output nDSM decoded features.

[0208] The boundary enhancement module is connected to the HED boundary extraction module and the RGB decoder output, respectively. It is used to fuse boundary features with segmentation mask features and repair building boundary defects through a positive feature replacement mechanism, and output boundary enhancement features.

[0209] The post-fusion and prediction module is used to fuse the boundary enhancement features and nDSM decoding features pixel by pixel. After processing by the classification prediction layer and activation function, the final building extraction probability map is output.

[0210] The system is used to execute the above-described method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion.

[0211] Example 3:

[0212] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion.

Claims

1. A method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion, characterized in that, Includes the following steps: Step S1: Acquire RGB images of high-resolution remote sensing data. and normalized digital surface model data The data are input into the RGB encoder and the nDSM encoder respectively for multi-level feature extraction to obtain the final features. and ; Step S2: Add an overall edge detection module to each level of the RGB encoder to extract multi-level boundary feature maps. And generate a fusion boundary feature map. ; Step S3: Using a multimodal feature fusion module, perform mid-term multimodal fusion on the corresponding layer features of the RGB and nDSM encoders to generate mid-term fused features. ; Step S4: Convert the final features output by the RGB encoder The input is upsampled in an RGB decoder, and the final output is a feature map containing building segmentation mask information. ; Step S5: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input is upsampled to the nDSM decoder and then... It connects to the corresponding level decoder and finally outputs the nDSM decoding features. ; Step S6: Use the boundary reinforcement module to... and The algorithm combines and utilizes a positive feature replacement mechanism to repair building boundary defects, outputting enhanced boundary features. ; Step S7: [The sentence is incomplete and requires more context to be translated accurately.] and The final post-processing multimodal fusion is performed to output the predicted results of building extraction. .

2. The method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion according to claim 1, characterized in that, The specific steps in step S2 include: Step S21: The overall edge detection module extracts the last feature map in the i-th layer of the RGB encoder, and then... The convolutional layer compresses it into a 1-channel feature map, and bilinear upsampling is used to restore the feature map size to the size of the input image, resulting in... ,in ; Step S22: Cascade all Generate a fused boundary feature map containing sufficient information about the target boundary. The formula is as follows: ； in, This indicates a feature cascade operation. It refers to the number of levels that HED has. .

3. The method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion according to claim 2, characterized in that, The specific process of step S3 is as follows: Step S31: For the RGB and nDSM encoders, the first... Feature images output at each level , Global average pooling is performed on each image to obtain the global feature vector of the RGB image with C channels. and nDSM global feature vector The formula for calculating global pooling is: ； Among them, y c This represents the global feature vector of the c-th channel. For the height of the feature image, For the feature image width, The number of feature image channels, and These are pixel coordinates; This represents the pixel feature value at coordinates (m, n) in the c-th channel; Step S32: Transfer the global feature vector and The input is fed into the compression-excitation block SE to explore the correlation between channels, and then activated by the Sigmoid function to obtain a weight vector between 0 and 1. The formula is as follows: ； ； in, This represents the compression-excitation block SE. express Convolution operation, ReLU is the modified linear unit activation function; This represents the Sigmoid activation function; Step S33: Process the feature image and Using weight vectors respectively Perform weighted summation and add element by element to generate the final result. Hierarchical mid-term fusion features The formula is as follows: ； ； ； in, This represents the weighted RGB feature image. This represents the weighted nDSM feature image. This indicates an element-wise multiplication operation.

4. The method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion according to claim 3, characterized in that, The specific process of the RGB decoder in step S4 is as follows: Step S41: Upsample the output features of the previous level of the RGB decoder to obtain the upsampled features of the current level. ; Receive the lateral output features of the skip connections at the corresponding levels of the RGB encoder ;Will and Perform cascading to generate cascaded features for the current level. ; Step S42: Concatenate features The input is processed by the improved pyramid-shaped residual connection unit (RCU) and the output is the decoded feature. The formula is: ； in, This represents the current level of cascaded features in the input; BN(⋅) represents batch normalization; DSC(⋅) represents depthwise separable convolution. Step S43: After being passed and processed layer by layer in reverse by each level of the RGB decoder, the final output layer contains features containing building segmentation mask information. .

5. The method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion according to claim 4, characterized in that, The specific process of the nDSM decoder in step S5 is as follows: Step S51: Upsample the output features of the previous level of the nDSM decoder to obtain the upsampled features of the current level. ;Will With mid-term fusion characteristics Generate multimodal cascaded features for the current level. ; Step S52: Concatenate multimodal features The input is processed by the improved pyramid-shaped residual connection unit (RCU) and the output is the decoded feature. The formula is: ； Step S53: After the decoder layers pass through and process each layer in reverse, the nDSM decoding features are finally output at the last layer. .

6. The method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion according to claim 5, characterized in that, The specific process of step S6 is as follows: Step S61: Fuse the boundary feature maps In the boundary detection subunit of the input boundary enhancement module, the boundary features are first processed by the Sigmoid activation function to generate a 5-channel basic boundary feature x mapped between 0 and 1, as shown in the formula: ； Then, the basic boundary feature x is processed in two ways: The first path, after dimensionality upscaling via a 1×1 convolutional layer and processing with the ReLU activation function, yields shared boundary features. It is directly passed to the segmentation mask detection subunit, and the calculation formula is: ； The second path will generate shared boundary features. The input is fed into the compression-activation block SE, and the output is reduced to 5 channels of attention weights. These attention weights are then multiplied element-wise with the preceding 5-channel basic boundary features x to enhance the boundary information. The enhanced features are then compressed into 1 channel through a 1×1 convolutional layer to generate the final boundary probability map. The calculation formula for this process is: ； Where Sigmoid(⋅) and ReLU(⋅) are activation functions, Block(⋅) represents the compression-activation block operation, and Conv1×1(⋅) represents the 1×1 convolution operation. This represents the element-wise multiplication operation; Step S62: Segmentation mask feature map With shared boundary features In the segmentation mask detection subunit of the synchronous input boundary enhancement module (BE), a cascading operation is used to... and The features are merged, and the merged composite features are processed sequentially through a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer to finally generate a 1-channel segmentation mask probability map. The calculation formula is: ； Where Conv3×3(⋅) represents a 3×3 convolution operation, and Conv1×1(⋅) represents a dimension reduction and compression operation to generate the final 1-channel mask probability map; Step S63: ... and Probability maps were obtained by applying the Sigmoid activation function. and Then, they are added pixel by pixel to obtain the merged probability map. The formula is as follows: ； in, This represents the activation result of the boundary probability map; This represents the activation result of the mask probability map; min(⋅) represents the minimum value function; Step S64: Obtain the feature values ​​of the original segmentation mask The positive response portion of the feature value is extracted using the ReLU activation function, and the average value of this positive response portion is calculated over the entire feature map to obtain the scaling factor. The calculation formula is as follows: ； ReLU( )=max(0, ); in, For feature map The eigenvalues; N is the feature map. The total number of pixels; k is the label of the pixel coordinates; This represents the feature value of the k-th pixel in feature map P2; Step S65: Strip the feature values ​​of the original segmentation mask The positive response portion is extracted, and only the negative response portion is retained; subsequently, the merged probability plot P is used. S (x) and scaling factor The proportional probability mapping is used to replace the original positive response portion, ultimately yielding the boundary-enhanced output features after boundary defect repair. The calculation formula is as follows: 。 7. The method for extracting buildings from high-resolution remote sensing images based on multi-level multimodal feature fusion according to claim 1, characterized in that, In step S7, the specific process of multimodal fusion is as follows: Step S71: ... and The input is summed pixel-by-pixel in the post-fusion unit to obtain the post-fusion features. ; Step S72: ... The input is fed into the final classification prediction layer, and after activation by the Sigmoid activation function, the final output is a prediction result containing building probability mappings. .

8. A high-resolution remote sensing image building extraction system based on multi-level multimodal feature fusion, characterized in that, include: The data acquisition and preprocessing module is used to acquire high-resolution remote sensing RGB images and normalized digital surface model (nDSM) data, perform sliding window cropping, spatial alignment and label reconstruction on the images, and output standardized multimodal input data. The dual-stream coding module includes an independent RGB encoder and an nDSM encoder, which are used to extract multi-level downsampling features from RGB images and nDSM data respectively, and output RGB encoded features and nDSM encoded features. The boundary extraction module is connected to each level of the RGB encoder. It is used to extract multi-level boundary features and perform cross-level feature concatenation to output a fused boundary feature map. The intermediate fusion module is connected to the output of each level of the RGB encoder and the nDSM encoder, and is used to perform global pooling, SE attention weighting and adaptive feature fusion on the RGB features and nDSM features of the same level, and output the intermediate fusion features of each level. The dual-stream decoding module includes an RGB decoder and an nDSM decoder. The RGB decoder is used to upsample, skip-connect, and improve pyramid residual unit processing of RGB encoded features to output a building segmentation mask feature map. The nDSM decoder is used to access intermediate fusion features, perform heterogeneous skip-connect and upsampled decoding, and output nDSM decoded features. The boundary enhancement module is connected to the HED boundary extraction module and the RGB decoder output, respectively. It is used to fuse boundary features with segmentation mask features and repair building boundary defects through a positive feature replacement mechanism, and output boundary enhancement features. The post-fusion and prediction module is used to fuse the boundary enhancement features and nDSM decoding features pixel by pixel. After processing by the classification prediction layer and activation function, the final building extraction probability map is output. The system is used to perform the high-resolution remote sensing image building extraction method based on multi-level multimodal feature fusion as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the high-resolution remote sensing image building extraction method based on multi-level multimodal feature fusion as described in any one of claims 1-7.

Images