Building extraction method and system driven by cross-modal semantic collaboration and frequency domain decoupling

Abstract

The application discloses a building extraction method and system driven by cross-modal semantic cooperation and frequency domain decoupling, and belongs to the technical field of remote sensing image intelligent interpretation and deep learning. The application constructs a MSSmamba multi-modal extraction model: firstly, a double-branch backbone network based on a visual state space VSS is designed to realize efficient long-distance context modeling with linear computational complexity; secondly, a sparse semantic fusion module SSFM is proposed, wherein a priori guided sparse capture mechanism is used to reduce feature redundancy, and a self-attention cross module is used to establish fine-grained cross-modal dependence; finally, a semantic frequency domain division module SFPM is designed, wherein high-frequency details and low-frequency semantics are decoupled through frequency domain decomposition, and multi-modal noise accumulation is inhibited. The application has excellent extraction precision, generalization and engineering landing potential, and can be widely applied to intelligent city planning, disaster emergency assessment and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of intelligent interpretation of remote sensing images and deep learning technology, and in particular to a method and system for building extraction driven by cross-modal semantic collaboration and frequency domain decoupling. Background Technology

[0002] Building extraction is one of the core tasks of remote sensing image analysis, and its results are widely used in various fields such as smart city planning, land use surveys, disaster emergency assessment, and digital city construction. Traditional building extraction methods based on single-modal optical remote sensing images are easily affected by factors such as shadows, vegetation occlusion, changes in illumination, and different spectra of the same object versus different objects of the same spectra. These methods have insufficient accuracy in extracting building edges and cannot meet the practical application needs in complex urban scenarios.

[0003] In recent years, deep learning technology has provided a new technical approach for remote sensing building extraction. Convolutional Neural Network (CNN)-based methods, such as U-Net, DeepLab, and HRNet, excel at capturing local spatial features, but are limited by their local receptive field, making it difficult to model the long-range spatial dependencies required for large-scale buildings and occluded areas. This leads to problems such as voids and blurred edges within large buildings. Transformer-based methods, such as ViT and SwinTransformer, overcome the local constraints of CNNs and possess excellent long-range modeling capabilities; however, the secondary computational complexity of the self-attention mechanism... This makes it too computationally expensive and memory-intensive when processing high-resolution remote sensing images, making it difficult to implement in engineering.

[0004] Multimodal fusion methods, which combine optical imagery with multi-source data such as LiDAR, leverage the semantic texture advantages of optical data and the geometric elevation advantages of LiDAR data to improve extraction accuracy, and are currently the mainstream research in this field. However, existing multimodal remote sensing building extraction methods still have many technical bottlenecks: First, information conflicts, feature redundancy, and modal imbalances easily occur during cross-modal fusion. Traditional pixel-level and decision-level fusion ignores the nonlinear relationships and semantic hierarchical differences between modalities, and feature-level fusion is prone to problems such as single-modal dominance and suppression of contributions from other modalities. Second, existing fusion methods are difficult to establish fine-grained cross-modal dependencies and cannot fully utilize the synergistic complementarity of different modal data, resulting in limited improvement in the accuracy of building edge extraction. Third, noise accumulation and amplification are prone to occur during multimodal fusion. Background noise in complex urban scenes can seriously affect the discriminative power of building features, leading to false positives, false negatives, and artifacts in the extraction results.

[0005] Furthermore, the emergence of Mamba-like state-space models provides linear complexity for modeling long sequences. While existing Mamba-based visual models have shown certain advantages in single-modal remote sensing image processing, there is still a gap in the design for adaptability to multimodal remote sensing data fusion, which cannot directly solve the core pain points in multimodal building extraction. Summary of the Invention

[0006] The purpose of this invention is to propose a building extraction method and system driven by cross-modal semantic collaboration and frequency domain decoupling, which solves the problems of cross-modal information conflict, noise accumulation, and difficulty in balancing long-distance modeling and computational efficiency in existing multimodal remote sensing building extraction methods, and cannot simultaneously meet the needs of high-precision extraction and engineering implementation.

[0007] To achieve the above objectives, this invention provides a building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling, comprising the following steps: Step S1: Acquire the high-resolution optical remote sensing image to be processed and the corresponding LiDAR data, and perform registration, cropping and normalization preprocessing on the optical remote sensing image and the corresponding LiDAR data to obtain spatially and pixelally matched multimodal input data. Step S2: Construct a dual-branch backbone network based on the visual state space (VSS) to perform multi-stage hierarchical encoding on the preprocessed optical remote sensing images and the corresponding LiDAR data, and extract multiple sets of single-modal depth features at corresponding scales; the VSS backbone network achieves long-distance context modeling with linear computational complexity through a cross-scanning mechanism and the state space model (SSM), while dynamically capturing the spatial-spectral continuity of building outlines in the remote sensing images; Step S3: Input optical features and LiDAR features of the same scale into the sparse semantic fusion module SSFM to complete the cross-modal geometric-semantic feature collaborative fusion; the sparse semantic fusion module SSFM first reduces feature redundancy through the prior guided sparse capture mechanism PG-SCM, and then establishes fine-grained cross-modal dependencies through the self-attention cross module SACM, and outputs preliminary fused features. Step S4: Input the preliminary fused features into the semantic frequency domain partitioning module SFPM. Through frequency domain decomposition, the fused features are decoupled into high-frequency detail components and low-frequency semantic components. The two types of components are enhanced and coupled separately to obtain the noise-reduced and optimized multimodal fused features. Step S5: Upsample and stitch together the optimized fusion features at several levels, and extract the building by outputting the semantic segmentation results of the building with the same size as the input image through the prediction layer.

[0008] Preferably, in step S2, the VSS backbone network comprises several cascaded VSS modules, each corresponding to a different downsampling rate, outputting multiple sets of single-modal features at different scales; each VSS module achieves feature mapping through discretized state-space equations, and the continuous-form state-space model expression is: ; ; in, Here is the state transition matrix. and For learnable weighted parameters, Given the input feature sequence, In hidden state, For output features; The VSS module adapts 2D image data through the Cross Scan Module (CSM), specifically by: inputting feature maps... The sequence is unfolded into a one-dimensional sequence along four different scanning directions: top left-bottom right, top right-bottom left, bottom right-top left, and bottom left-top right. Each sequence is processed by an independent SSM discretization unit and then reconstructed into a two-dimensional feature map through inverse folding and weighted aggregation, as expressed in the following formula: ; in, For along the first Serialization operation of scan path, This is the inverse transformation operation. This is for outputting feature maps.

[0009] Preferably, in step S3, the specific processing steps of the priori guided sparse capture mechanism PG-SCM are as follows: S311: Input feature map A single-channel prior guided weight map is generated through two layers of convolution and activation functions. The expression is: ; in, , For learnable convolutional kernels, It is the sigmoid activation function. It is a linear rectified activation function. These are bias parameters; S312: Expand the prior guided weight map along the channel dimension to the same number of channels as the input feature map, and multiply it element-wise with the input feature map to obtain the spatial prior feature map. The expression is: ; in, This is the prior weight graph after channel dimension expansion. This is an element-wise multiplication operation; S313: Based on preset sparsity The spatial prior feature map is divided into non-overlapping blocks, and each block is flattened into a token vector to generate a compact token sequence, as expressed by: ; ; in, For token vectors, This represents the number of tokens.

[0010] Preferably, in step S3, the specific processing steps of the Self-Attention Cross Module (SACM) are as follows: S321: Optical characteristics of the input With LiDAR features Channel compression is performed using convolutional layers of preset sizes to generate dimensionality-reduced features with the same number of channels. and Maintaining the feature space resolution unchanged; S322: The dimensionality-reduced bimodal features are transformed by separable convolutions of preset size and depth, and then a key matrix is ​​generated by a reshape operation. AND-value matrix The expression is: ; ; in, For modal identification, , These are the key matrix and the value matrix, respectively. This is a depthwise separable convolution operation; S323: The dimensionality-reduced optical features are concatenated with LiDAR features along the channel dimension. Convolution and reshape operations can be separated by a preset size and depth to generate a shared query matrix. The expression is: ; in, For feature splicing operations, For query matrix; S324: Calculate the query matrix AND key matrix The dot product of the values ​​is normalized using the Softmax function to obtain the cross-modal attention weight matrix, which is then combined with the value matrix. Weighted fusion is performed, and the original bimodal features are injected through residual connections to output the final cross-modal fused features.

[0011] Preferably, in step S4, the specific processing steps of the semantic frequency domain partitioning module (SFPM) are as follows: S41: Fusion features on input Low-resolution feature prototypes are obtained by downsampling through a pre-defined depthwise convolution with a pre-defined stride. ; S42: Perform a two-dimensional wavelet decomposition of the feature prototype with a preset wavelet basis at a preset number of levels to obtain at least one low-frequency component. High-frequency components in three directions The expression is: ; in, This is a two-dimensional wavelet decomposition operation based on a preset wavelet basis. Contains main semantic information, Each contains edge texture detail information in different directions; S43: After summing multiple high-frequency components, bilinear upsampling is performed to restore the size of the input features, which is then coupled with the original input features to obtain high-frequency detail information. The low-frequency components are upsampled bilinearly at the same rate and coupled with the original input features to obtain low-frequency semantic information. The expression is: ; ; in, This is a bilinear upsampling operation; S44: High-frequency detail information and low-frequency semantic information are concatenated in the channel dimension, and channel fusion and channel number restoration are performed through convolution of preset size to output the fused features after noise reduction and enhancement.

[0012] Preferably, in step S1, the preprocessing specifically includes: Ensure that the geographic coordinates and pixel dimensions of the optical remote sensing imagery and LiDAR data are perfectly matched; The original data is uniformly cropped into image blocks of a preset size, and zero-filling is applied to areas with insufficient edge size. The RGB channels of optical images are normalized using the mean and standard deviation of a preset dataset, and the nDSM data of LiDAR are normalized to a preset value range using min-max normalization. The labeled data is mapped to building categories as a first preset value, and the remaining categories are mapped to second preset values. An online data augmentation strategy involving random horizontal flipping and random rotation is employed for the training set data.

[0013] Preferably, the AdamW optimizer is used during the model training phase, with preset base learning rate, preset number of training iterations and preset batch size; the cross-entropy loss function is used as the loss function for model training; and synchronous batch normalization is used to ensure training stability during multi-GPU training.

[0014] A building extraction system driven by cross-modal semantic collaboration and frequency domain decoupling includes: The data preprocessing module is used to acquire the optical remote sensing images to be processed and the corresponding LiDAR data, complete the registration, cropping, normalization and data augmentation operations, and output multimodal input data that matches space and pixels. The dual-branch feature encoding module has a built-in dual-branch backbone network based on visual state space (VSS) for parallel multi-stage hierarchical encoding of optical remote sensing images and corresponding LiDAR data, and outputs multiple sets of single-modal depth features at different scales. The sparse semantic fusion module has a built-in prior-guided sparse capture unit PG-SCM and a self-attention cross-unit SACM, which are used to perform sparsification processing and fine-grained cross-modal attention fusion on bimodal features of the same scale, and output preliminary fused features. The frequency domain decoupling enhancement module has a built-in semantic frequency domain partitioning unit (SFPM) for frequency domain decomposition, component enhancement and noise reduction coupling of the initial fused features, and outputs optimized multimodal fused features. The decoding and prediction module is used to perform stepwise upsampling, cross-scale stitching, and pixel-level classification on multi-scale fused features, and output semantic segmentation results of buildings with the same size as the input image.

[0015] Therefore, the building extraction method and system driven by the cross-modal semantic collaboration and frequency domain decoupling described above have the following advantages: (1) This invention is the first to apply the Visual State Space Model (VSS) to the field of multimodal remote sensing building extraction. It achieves long-distance context modeling with linear computational complexity through a dual-branch VSS backbone network. Compared with Transformer-type methods, the computational efficiency is significantly improved, and it has excellent potential for engineering application. (2) The sparse semantic fusion module SSFM proposed in this invention significantly reduces feature redundancy through prior-guided sparsification and establishes fine-grained cross-modal dependency through the self-attention cross mechanism of shared query, effectively solving the problems of information conflict and modal imbalance in multimodal fusion. (3) The semantic frequency domain partitioning module SFPM designed in this invention achieves the separation and enhancement of high-frequency details and low-frequency semantics through frequency domain decoupling, which specifically suppresses noise accumulation in the multimodal fusion process and significantly improves the extraction accuracy of building edges.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a diagram of the MSSmamba framework in the building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling in this embodiment of the invention. Figure 2 This is a diagram of the VSS core components in an embodiment of the present invention; Figure 3 This is a flowchart of the SSFM process in an embodiment of the present invention; Figure 4 This is a flowchart of the PG-SCM embodiment of the present invention; Figure 5 This is a flowchart of the SACM process in an embodiment of the present invention; Figure 6 This is a visual comparison of the Vaihingen dataset in an embodiment of the present invention; Figure 7 This is a visual comparison of the Potsdam dataset in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] Example like Figure 1-7 As shown, this embodiment proposes a building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling. This embodiment details the core module design of the MSSmamba model, and the MSSmamba framework is as follows: Figure 1As shown, all modules are implemented based on the PyTorch 2.1.1 framework and are compatible with NVIDIA GPU CUDA acceleration. The specific implementation steps are as follows: Step 1: Data Preprocessing To acquire the high-resolution optical remote sensing image to be processed and the corresponding LiDAR nDSM data, georegistration is first performed to ensure that the geographic coordinates and pixel sizes of the two types of data are perfectly matched. The original data is then uniformly cropped into image blocks of a preset size. In this embodiment, the preset size is 512×512 pixels. A non-overlapping cropping method is used, and zero-padding is applied to areas with edges smaller than 512×512 pixels.

[0021] For the RGB channels of the optical image, normalization is performed using the mean and standard deviation of a preset dataset. In this embodiment, the preset dataset is ImageNet, with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225]. For the LiDAR nDSM data, min-max normalization is performed to a preset numerical range, which in this embodiment is [0, 1]. The labeled data is binarized, mapping building categories to a first preset value (1 in this embodiment) and all other categories to a second preset value (0 in this embodiment).

[0022] Online data augmentation strategies were employed on the training set data, including random horizontal flipping (probability 0.5) and random rotation (angle range -15° to +15°), to improve the model's generalization ability.

[0023] Step 2: Coding of the dual-branch VSS backbone network: A dual-branch backbone network based on Visual State Space (VSS) is constructed to encode the preprocessed optical remote sensing images and LiDAR data in parallel. Each branch contains multiple cascaded VSS modules, four in this embodiment, corresponding to downsampling ratios of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively, outputting four sets of single-modal depth features at different scales.

[0024] like Figure 2 As shown, the core of each VSS Block is the State-Space Model (SSM) and the Cross-Scan Module (CSM), where the continuous state-space equation of the SSM is: ; ; in, This is the state transition matrix, used to model the long-range dependencies of sequences; and These are learnable weighted parameters; The input feature sequence; This is a hidden state used to store historical information about the sequence; For output features.

[0025] For adaptation to 2D remote sensing imagery, the CSM module will input feature maps. The sequence is unfolded into a one-dimensional sequence along multiple different scanning directions. In this embodiment, there are four scanning directions: top left-bottom right, top right-bottom left, bottom right-top left, and bottom left-top right. Each sequence is processed by an independent S6 unit (an efficient discretization of SSM), and then the two-dimensional structure is restored through a de-folding operation. Finally, the outputs from the four directions are weighted and aggregated to obtain the final output feature, expressed as: ; in, For along the first Serialization operation of scan path, This is the inverse transformation operation. This design allows the model to maintain linear computational complexity. At the same time, it achieves the aggregation of global context information, which can effectively capture the overall structural and outline continuity of large-sized buildings.

[0026] In this embodiment, the VSS backbone adopts the VSSM-Small configuration, and the batch normalization layer uses synchronous batch normalization during multi-GPU training to ensure training stability.

[0027] Step 3: Sparse semantic fusion: Optical features and LiDAR features of the same scale are input into the Sparse Semantic Fusion (SSFM) module to achieve cross-modal geometric-semantic feature collaborative fusion. The SSFM module workflow is as follows: Figure 3 As shown, the core contains two units: PG-SCM and SACM.

[0028] The processing flow of the priori-guided sparse capture unit (PG-SCM) is as follows: Figure 4 As shown, the specific processing steps are as follows: (1) Input feature map A single-channel prior guided weight map is generated by sequentially applying 1×1 convolution, ReLU activation function, 1×1 convolution, and sigmoid activation function. The expression is: ; Prior guidance weight maps can automatically identify information-rich building areas and background areas in feature maps, enhance the feature response of building areas, and suppress redundant information in background areas.

[0029] (2) Broadcast the prior guiding weight map along the channel dimension to the same number of channels as the input feature map, and multiply it element-wise with the input feature map to obtain the spatial prior feature map. : .

[0030] (3) Based on the preset sparsity The spatial prior feature map is divided into non-overlapping blocks. In this embodiment, the sparsity rate is preset to 2, that is, 2×2 non-overlapping blocks are performed. Each block is flattened into a token vector, generating a compact token sequence. At this time, the number of tokens is reduced to 1 / 4 of the original features, which significantly reduces the complexity of subsequent attention calculations, while retaining the core features of the building area.

[0031] The architecture of the self-attention cross unit (SACM) is as follows: Figure 5 As shown, the specific processing steps are as follows: (1) Optical characteristics of the input With LiDAR features Channel compression is performed using convolutional layers of a preset size (1×1 in this embodiment) to generate dimensionality-reduced features with the same number of channels. and This maintains the spatial resolution of the features.

[0032] (2) For the dimensionality-reduced bimodal features, feature transformation is performed by depthwise separable convolution with a preset size. In this embodiment, the preset size is 3×3. Then, a key matrix is ​​generated by reshape operation. AND-value matrix : ; ; in, For modal identification, .

[0033] (3) The dimensionality-reduced optical features and LiDAR features are concatenated along the channel dimension. The convolution and reshape operations can be separated by a preset size and depth. In this embodiment, the preset size is 3×3, generating a shared query matrix. : ; The design of a shared query matrix enables the model to focus on key information from both optical and LiDAR features, effectively mitigating the modal imbalance problem.

[0034] (4) Calculate the query matrix AND key matrix The dot product of the values ​​is normalized using the Softmax function to obtain the cross-modal attention weight matrix, which is then combined with the value matrix. Weighted fusion is performed to obtain attention-enhanced features. Simultaneously, residual connections are used to inject the original bimodal features into the fusion result, avoiding the gradient vanishing problem, and finally outputting cross-modal fused features.

[0035] Step 4: Frequency Domain Decoupling Enhancement: The initial fused features are input into the semantic frequency domain partitioning module (SFPM) for frequency domain decoupling and noise reduction enhancement. The specific steps are as follows: (1) Input fusion features Downsampling is performed using a depthwise convolution with a preset stride and a preset size. In this embodiment, the preset stride is 2 and the preset size is 3×3, resulting in a low-resolution feature prototype. Capture spatially invariant semantic patterns.

[0036] (2) The feature prototype is decomposed into two-dimensional wavelet decomposition with a preset number of layers using a preset wavelet basis. In this embodiment, the preset wavelet basis is Daubechies-1 (db1) and the preset number of layers is 1, resulting in one low-frequency component. High-frequency components in three directions : ; Among them, low-frequency components Contains semantic information of the subject with fused features, high-frequency components It contains edge texture details in the horizontal, vertical, and diagonal directions respectively.

[0037] (3) After summing the three high-frequency components, perform bilinear upsampling to restore the size of the input features, and couple it with the original input features to obtain high-frequency detail information. The low-frequency components are upsampled bilinearly at the same rate and coupled with the original input features to obtain low-frequency semantic information. : ; ; in, For bilinear upsampling operation (4) High-frequency detail information and low-frequency semantic information are spliced ​​together in the channel dimension, and channel fusion and channel number recovery are performed by convolution with a preset size. In this embodiment, the preset size is 1×1, and the fused features after noise reduction and enhancement are output.

[0038] In this embodiment, the SFPM module is deployed after the SSFM module. This deployment method can specifically suppress noise and artifacts generated during the fusion process, while not destroying the established cross-modal dependencies, thus achieving optimal performance.

[0039] Step 5: Decoding and Prediction The decoding part employs a progressive upsampling strategy, progressively upsampling the deepest fused features by a factor of 2, then concatenating them with the fused features from the previous level, followed by feature optimization using a 3×3 convolution. This process is repeated until the feature size is restored to 1 / 4 of the input image. Finally, a 2-channel semantic segmentation result is output through a 2x upsampling and a 1×1 convolutional prediction head. After Softmax activation, pixel-level building classification results are obtained, completing building extraction.

[0040] Model training and performance validation: This embodiment uses the internationally recognized ISPRS Vaihingen and Potsdam datasets for model training and validation: The Vaihingen dataset has a ground sampling distance of 9cm (GSD) and contains 33 orthophotos labeled with six types of land features: buildings, impervious surfaces, low vegetation, trees, vehicles, and background. This example performs binarization processing for the building extraction task.

[0041] The Potsdam dataset has a GSD of 5cm and contains 38 orthophoto tiles of 6000×6000 pixels. The labeled categories are the same as those in the Vaihingen dataset, and the standard partitioning method of 24 training images and 14 test images is used.

[0042] The training environment consisted of an NVIDIA GeForce RTX 3060 12GB GPU, an Intel i7 processor, 32GB of RAM, an Ubuntu 20.04 operating system, PyTorch 2.1.1 deep learning framework, and CUDA version 11.8.

[0043] Training configuration: AdamW optimizer is used, with a preset base learning rate of [missing information]. The weight decay coefficient is The preset training iteration count is 40,000, the preset batch size is 2, and the cross-entropy loss function is used as the loss function for model training.

[0044] This embodiment verifies the effectiveness of the invention through comparative experiments, ablation experiments, and generalization experiments. The evaluation indicators used include: overall precision (OA), precision (Pre.), recall (Rec.), F1 score, and mean intersection-over-union ratio (mIoU). The calculation formulas for each indicator are as follows: ; ; ; ; ; Wherein, TP represents the number of true positive samples, TN represents the number of true negative samples, FP represents the number of false positive samples, and FN represents the number of false negative samples.

[0045] Experimental results show that, on the Vaihingen and Potsdam datasets, the MSSmamba model of this invention is compared with existing mainstream methods, including single-modal methods MANet, ST-UNet, CMTFNet, Vmamba, and multi-modal methods EndNet, AMSSENet, MultiSenseSeg, FTransUNet, and AMSNet. Figure 6 and Figure 7 The segmentation capability was demonstrated. Experimental results show that the MSSmamba model of this invention achieves state-of-the-art performance on both datasets. On the Vaihingen dataset, the mIoU reaches 89.29%, which is 1.29 percentage points higher than the existing best multimodal method, and the recall rate is improved by 1.37 percentage points. On the Potsdam dataset, the mIoU reaches 90.53%, which is 1.29 percentage points higher than the existing best method, and the OA exceeds 97% for the first time, fully verifying the effectiveness of this invention. The experimental results are shown in Tables 1 and 2 below: Table 1. Comparison Experiment Results of Vaihingen Dataset

[0046] Table 2 Comparison Experiment Results of Potsdam Dataset

[0047] This invention also provides a building extraction system driven by cross-modal semantic collaboration and frequency domain decoupling, including a data preprocessing module, a dual-branch feature encoding module, a sparse semantic fusion module, a frequency domain decoupling enhancement module, and a decoding prediction module. Each module is connected in sequence and executes the corresponding steps in the above method to collaboratively complete the multimodal remote sensing building extraction task.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling, characterized in that, Includes the following steps: Step S1: Acquire the high-resolution optical remote sensing image to be processed and the corresponding LiDAR data, and perform registration, cropping and normalization preprocessing on the optical remote sensing image and the corresponding LiDAR data to obtain spatially and pixelally matched multimodal input data. Step S2: Construct a dual-branch backbone network based on the visual state space (VSS) to perform multi-stage hierarchical encoding on the preprocessed optical remote sensing images and the corresponding LiDAR data, and extract multiple sets of single-modal depth features at corresponding scales; the VSS backbone network achieves long-distance context modeling with linear computational complexity through a cross-scanning mechanism and the state space model (SSM), while dynamically capturing the spatial-spectral continuity of building outlines in the remote sensing images; Step S3: Input optical features and LiDAR features of the same scale into the sparse semantic fusion module SSFM to complete the cross-modal geometric-semantic feature collaborative fusion; The Sparse Semantic Fusion Module (SSFM) first reduces feature redundancy through the Prior Guided Sparse Capture Mechanism (PG-SCM), and then establishes fine-grained cross-modal dependencies through the Self Attention Cross Module (SACM), outputting preliminary fused features. Step S4: Input the preliminary fused features into the semantic frequency domain partitioning module SFPM. Through frequency domain decomposition, the fused features are decoupled into high-frequency detail components and low-frequency semantic components. The two types of components are enhanced and coupled separately to obtain the noise-reduced and optimized multimodal fused features. Step S5: Upsample and stitch together the optimized fusion features at several levels, and extract the building by outputting the semantic segmentation results of the building with the same size as the input image through the prediction layer.

2. The building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling according to claim 1, characterized in that: In step S2, the VSS backbone network contains several cascaded VSS modules, each corresponding to a different downsampling rate, outputting multiple sets of single-modal features at different scales. Each VSS module achieves feature mapping through discretized state-space equations, and the continuous-form state-space model expression is as follows: ； ； in, Here is the state transition matrix. and For learnable weighted parameters, Given the input feature sequence, In hidden state, For output features; The VSS module adapts 2D image data through the Cross Scan Module (CSM), specifically by: inputting feature maps... The sequence is unfolded into a one-dimensional sequence along four different scanning directions: top left-bottom right, top right-bottom left, bottom right-top left, and bottom left-top right. Each sequence is processed by an independent SSM discretization unit and then reconstructed into a two-dimensional feature map through inverse folding and weighted aggregation, as expressed in the following formula: ； in, For along the first Serialization operation of scan path, This is the inverse transformation operation. This is for outputting feature maps.

3. The building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling according to claim 1, characterized in that: In step S3, the specific processing steps of the priori guided sparse capture mechanism PG-SCM are as follows: S311: Input feature map A single-channel prior guided weight map is generated through two layers of convolution and activation functions. The expression is: ； in, , For learnable convolutional kernels, It is the sigmoid activation function. It is a linear rectified activation function. These are bias parameters; S312: Expand the prior guided weight map along the channel dimension to the same number of channels as the input feature map, and multiply it element-wise with the input feature map to obtain the spatial prior feature map. The expression is: ； in, This is the prior weight graph after channel dimension expansion. This is an element-wise multiplication operation; S313: Based on preset sparsity The spatial prior feature map is divided into non-overlapping blocks, and each block is flattened into a token vector to generate a compact token sequence, as expressed by: ； ； in, For token vectors, This represents the number of tokens.

4. The building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling according to claim 1, characterized in that: In step S3, the specific processing steps of the Self-Attention Cross Module (SACM) are as follows: S321: Optical characteristics of the input With LiDAR features Channel compression is performed using convolutional layers of preset sizes to generate dimensionality-reduced features with the same number of channels. and Maintaining the feature space resolution unchanged; S322: The dimensionality-reduced bimodal features are transformed by separable convolutions of preset size and depth, and then a key matrix is ​​generated by a reshape operation. AND-value matrix The expression is: ； ； in, For modal identification, , These are the key matrix and the value matrix, respectively. This is a depthwise separable convolution operation; S323: The dimensionality-reduced optical features are concatenated with LiDAR features along the channel dimension. Convolution and reshape operations can be separated by a preset size and depth to generate a shared query matrix. The expression is: ； in, For feature splicing operations, For query matrix; S324: Calculate the query matrix AND key matrix The dot product of the values ​​is normalized using the Softmax function to obtain the cross-modal attention weight matrix, which is then combined with the value matrix. Weighted fusion is performed, and the original bimodal features are injected through residual connections to output the final cross-modal fused features.

5. The building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling according to claim 1, characterized in that: In step S4, the specific processing steps of the semantic frequency domain partitioning module (SFPM) are as follows: S41: Fusion features on input Low-resolution feature prototypes are obtained by downsampling through a pre-defined depthwise convolution with a pre-defined stride. ; S42: Perform a two-dimensional wavelet decomposition of the feature prototype with a preset wavelet basis at a preset number of levels to obtain at least one low-frequency component. High-frequency components in three directions The expression is: ； in, This is a two-dimensional wavelet decomposition operation based on a preset wavelet basis. Contains main semantic information, Each contains edge texture detail information in different directions; S43: After summing multiple high-frequency components, bilinear upsampling is performed to restore the size of the input features, which is then coupled with the original input features to obtain high-frequency detail information. The low-frequency components are upsampled bilinearly at the same rate and coupled with the original input features to obtain low-frequency semantic information. The expression is: ； ； in, This is a bilinear upsampling operation; S44: High-frequency detail information and low-frequency semantic information are concatenated in the channel dimension, and channel fusion and channel number restoration are performed through convolution of preset size to output the fused features after noise reduction and enhancement.

6. The building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling according to claim 1, characterized in that: In step S1, the preprocessing specifically includes: Ensure that the geographic coordinates and pixel dimensions of the optical remote sensing imagery and LiDAR data are perfectly matched; The original data is uniformly cropped into image blocks of a preset size, and zero-filling is applied to areas with insufficient edge size. The RGB channels of optical images are normalized using the mean and standard deviation of a preset dataset, and the nDSM data of LiDAR are normalized to a preset value range using min-max normalization. The labeled data is mapped to building categories as a first preset value, and the remaining categories are mapped to second preset values. An online data augmentation strategy involving random horizontal flipping and random rotation is employed for the training set data.

7. The building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling according to claim 1, characterized in that: During the model training phase, the AdamW optimizer is used, with preset base learning rate, preset number of training iterations, and preset batch size. The cross-entropy loss function is used as the loss function for model training. Synchronous batch normalization is used to ensure training stability during multi-GPU training.

8. A building extraction system driven by cross-modal semantic collaboration and frequency domain decoupling, used to implement the building extraction method driven by cross-modal semantic collaboration and frequency domain decoupling as described in any one of claims 1-7, characterized in that, include: The data preprocessing module is used to acquire the optical remote sensing images to be processed and the corresponding LiDAR data, complete the registration, cropping, normalization and data augmentation operations, and output multimodal input data that matches space and pixels. The dual-branch feature encoding module has a built-in dual-branch backbone network based on visual state space (VSS) for parallel multi-stage hierarchical encoding of optical remote sensing images and corresponding LiDAR data, and outputs multiple sets of single-modal depth features at different scales. The sparse semantic fusion module has a built-in prior-guided sparse capture unit PG-SCM and a self-attention cross-unit SACM, which are used to perform sparsification processing and fine-grained cross-modal attention fusion on bimodal features of the same scale, and output preliminary fused features. The frequency domain decoupling enhancement module has a built-in semantic frequency domain partitioning unit (SFPM) for frequency domain decomposition, component enhancement and noise reduction coupling of the initial fused features, and outputs optimized multimodal fused features. The decoding and prediction module is used to perform stepwise upsampling, cross-scale stitching, and pixel-level classification on multi-scale fused features, and output semantic segmentation results of buildings with the same size as the input image.

Images