Building detection method based on optical and SAR image fusion technology
By employing techniques such as twin networks, CBAM attention modules, wavelet transform, and cross-frequency fusion modules (CFFM), the domain gap between optical and SAR images in building detection was resolved, enabling the fusion of complementary information from optical and SAR images and improving the robustness and accuracy of building detection.
Patent Information
- Application Number
- CN202511764077.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-02-24
AI Technical Summary
In existing technologies, there is a domain gap between optical and SAR images in building detection, making it difficult to fully utilize complementary information. Furthermore, optical images are easily obscured by clouds and fog, while SAR images are affected by speckle noise, resulting in incomplete building edge information and poor robustness.
A twin network is used to extract spatial domain features from optical and SAR images, and a CBAM attention module and wavelet transform are used to extract frequency domain features. Spatial and frequency domain features are then fused through a cross-frequency fusion module CFFM and multi-layer dilated convolution to achieve complementary information fusion between optical and SAR images.
It effectively reduces false alarms, ensures the integrity of building outlines, achieves all-weather imaging and rich texture information fusion, and improves the robustness and accuracy of building detection.
Smart Images
Figure CN121564547A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of building detection technology, and in particular to a building detection method based on optical and SAR image fusion technology. Background Technology
[0002] Building detection plays a vital role in everyday human applications such as urban planning, disaster assessment, and emergency rescue. Optical and SAR images contain complementary information about the same ground feature, and joint processing of multimodal information from optical and SAR images can improve building detection performance. However, optical images are susceptible to factors such as cloud cover and lighting conditions, posing challenges to interpretation; SAR images, limited by their imaging mechanism, suffer from speckle noise interference, and the edge information of buildings is less complete, posing challenges to the robustness of algorithms.
[0003] Meanwhile, optical and SAR images, as images generated by different sensors, have a large semantic gap and are difficult to process directly together. Existing methods often only rely on deep learning methods to achieve optical and SAR fusion processing from the image domain perspective, which does not make full use of complementary information and has poor performance in representing the domain-inherent features of each modality. Summary of the Invention
[0004] The purpose of this invention is to provide a building detection method based on optical and SAR image fusion technology, which effectively solves the domain gap problem between optical and SAR images from the perspective of frequency domain, realizes the fusion of complementary information between the two, reduces false alarms, and ensures the integrity of building outlines.
[0005] To achieve the above objectives, the present invention provides a building detection method based on optical and SAR image fusion technology, comprising the following steps: S1. Extract spatial domain features from optical and SAR images using twin networks; S2. Spatial domain features of optical and SAR images are extracted from the fusion of channel dimension and spatial dimension using the CBAM attention module. S3. Extract frequency domain features of optical and SAR images using wavelet transform; S4. Frequency characteristics of optical and SAR images fused based on the cross-frequency fusion module CFFM; S5. Utilize multi-layered dilated convolution to fuse the spatial and frequency domain features of optical and SAR images.
[0006] Preferably, in S1, paired optical and SAR images are selected as inputs to the network, and image features of the optical and SAR images are extracted using a twin network, with ResNet50 used as the backbone network for feature extraction.
[0007] Preferably, in S2, in the spatial domain branch, deep features of optical and SAR images are integrated through channel attention and spatial attention mechanisms. Simultaneously, shallow optical features with rich detail are fused. The fusion process is specifically represented by the following formula: ; in, Represents channel cascading, These represent the deep optical and SAR image features extracted by the backbone network, respectively.
[0008] Preferably, in S2, the fused deep information is upsampled and made to the same size as the shallow features of the optical image, and then fused with the shallow features to capture fine-grained information, specifically: ; in, Represents the multimodal features of optical and SAR images in the final spatial domain fusion. This represents the shallow optical features extracted by the backbone network. This represents an upsampling operation.
[0009] Preferably, in S3, in the frequency domain processing section, wavelet transform is used to extract the frequency domain features of optical and SAR images. The wavelet transform is expressed by the following formula. ; in, For the input image features, It corresponds to the low-frequency component. These are the corresponding high-frequency components in the horizontal, vertical, and diagonal directions.
[0010] Preferably, in S3, the change domain operation is performed continuously to decompose the frequency information of the features, as shown in the following formula: ; in, , i This represents the current stage or level.
[0011] Preferably, in S4, frequency information fusion is performed based on the cross-frequency fusion module CFFM, as shown in the following equation: ; ; ; in, Representative at the i The stage utilizes frequency information from optical and SAR images fused by CFFM. and Representing the first iFrequency domain information of optical and SAR images at each stage; , , and Representing the first i The low-frequency information and the corresponding three high-frequency features in the optical image of the stage; , , and Representing the first i The low-frequency information and the corresponding three high-frequency features in the frequency information of the SAR image at this stage.
[0012] Preferably, in S4, during image feature domain transformation, the main features of the building are in the low-frequency feature map, while the edge features are mainly reflected in the high-frequency feature map. When performing cross-modal frequency information fusion, because the SAR image fully reflects the main building but lacks sufficient edge detail, its low-frequency information is considered during fusion. This process is represented by the following formula: ; ; in, , and Representing the first i Low-frequency information from staged optical images, SAR images, and fused low-frequency information. Representing the first i Three sets of high-frequency information from the phased optical image. These represent the high-frequency information after CFFM processing.
[0013] Preferably, in S4, the frequency characteristics are obtained using inverse wavelet transform: ; in, Represents inverse wavelet transform. It is the first i Phase CFFM achieves a joint representation that integrates multimodal low-frequency features representing stable building bodies and multimodal high-frequency information representing fine building edges; ; This represents the frequency information of the fusion of multi-scale optical and SAR images.
[0014] Preferably, in S5, in the spatial-frequency domain feature fusion part, multi-layer dilated convolution is used to fuse the spatial and frequency domain features of optical and SAR images starting from different receptive fields, and finally the model parameters are optimized using the cross-entropy loss function.
[0015] Therefore, the building detection method based on optical and SAR image fusion technology described above has the following advantages: (1) This invention addresses the domain gap between optical and SAR images in building extraction tasks from the perspective of the frequency domain, effectively realizing the fusion of complementary information between optical and SAR images, reducing false alarms while further ensuring the integrity of the extracted building outline.
[0016] (2) The cross-frequency fusion module proposed in this invention combines the characteristics of SAR images, which are not affected by weather and other conditions and can be imaged all day long, with the characteristics of optical images, which have rich texture information and rich detail information. It better integrates the frequency information of optical and SAR images and integrates optical and SAR image features at different scales, ensuring the integrity and consistency of global-local information.
[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0018] Figure 1 This is a general architecture diagram of an embodiment of a building detection method based on optical and SAR image fusion technology according to the present invention; Figure 2 This is a schematic diagram of the optical and SAR image frequency information fusion process in an embodiment of a building detection method based on optical and SAR image fusion technology according to the present invention. Detailed Implementation
[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0020] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
[0021] This invention provides a dual-branch optical and SAR joint processing architecture that combines image spatial domain and frequency domain to achieve accurate building extraction.
[0022] like Figure 1 As shown, a building detection method based on optical and SAR image fusion technology is proposed. This method extracts and fuses features from optical and SAR images in both the spatial and frequency domains, and then fuses the spatial and frequency features through a spatial-frequency fusion component. Specifically, the method includes the following steps: S1. Spatial domain features of optical and SAR images are extracted using twin networks.
[0023] Pairs of optical and SAR images are selected as inputs to the network. The Siamese network is used to extract spectral and spatial image features in the spatial domain of the optical and SAR images. The ResNet50, a representative multi-layer convolutional network architecture, is used as the backbone network for feature extraction.
[0024] S2. Using the CBAM attention module, spatial domain features of optical and SAR images are extracted from the fusion of channel and spatial dimensions.
[0025] In the spatial domain branch, deep features from optical and SAR images are integrated through channel attention and spatial attention mechanisms. Simultaneously, shallow optical features rich in detail are fused to achieve efficient global-local information fusion. The fusion process can be specifically represented by the following equation: ; in, Represents channel cascading, These represent the deep optical and SAR image features extracted by the backbone network, respectively.
[0026] Compared to SAR images, optical images contain richer texture characteristics, and their shallow features contain more detailed information. To better capture the edge information of buildings, this implementation upsamples the fused deep information to the same size as the shallow features of the optical image, and then fuses it with the shallow features to capture fine-grained information. Specifically: ; in, Represents the multimodal features of optical and SAR images in the final spatial domain fusion. This represents the shallow optical features extracted by the backbone network. This represents an upsampling operation.
[0027] S3. Extract frequency domain features of optical and SAR images using wavelet transform.
[0028] In the frequency domain processing section, a multi-layer recursive network is designed using the concept of wavelet transform to obtain frequency representations of different receptive fields and shallow / deep layer features. This achieves the conversion from image domain features to frequency domain features. Wavelet transform is a multi-scale analysis method that can decompose an image into different frequency components, and these frequency components have certain time-domain characteristics, providing both frequency domain and location information simultaneously. Wavelet transform can be expressed by the following equation: ; in, For the input image features, It corresponds to the low-frequency component. These represent the high-frequency components in the corresponding horizontal, vertical, and diagonal directions. It is worth noting that the spatial resolution of each frequency component generated by the wavelet transform is a feature of the input. Half of the wavelet transform is used in this embodiment, which uses Haar as the basic transformation of the wavelet transform.
[0029] To fully obtain the multi-scale frequency information of the image, continuous change domain operations are performed to decompose the frequency information of the features, as shown in the following equation: ; in, , i This represents the current stage or level.
[0030] S4, such as Figure 2 As shown, based on the cross-frequency fusion module CFFM, the full fusion of optical and SAR frequency domain features is achieved, enhancing the network's ability to jointly represent multimodal features. The process is illustrated in the following equation: ; ; ; in, Representative at the i The stage utilizes frequency information from optical and SAR images fused by CFFM. and Representing the first i Frequency domain information of the optical and SAR images of the phase, where, , , and Representing the first i The low-frequency information and the corresponding three high-frequency features in the optical image of the stage; , , and Representing the first i The low-frequency information and the corresponding three high-frequency features in the frequency information of the SAR image at this stage.
[0031] When performing domain transformation on image features, the more stable features of the main building body are mainly reflected in the low-frequency image, while the features representing the finer building edges are mainly stored in the three sets of high-frequency images. When performing cross-modal frequency information fusion, considering that SAR images are not affected by clouds, fog, or weather conditions and thus provide a relatively complete representation of the main building body, but are constrained by imaging mechanisms such as speckle noise, the fine detail in depicting the fine building edges is insufficient, therefore, the fusion process mainly considers the low-frequency information. This process can be represented by the following formula: ; ; in, , and Representing the first i Low-frequency information from staged optical images, SAR images, and fused low-frequency information. Representing the first i Three sets of high-frequency information from the phased optical image. These represent the high-frequency information after CFFM processing.
[0032] Therefore, the multimodal fusion features of optical and SAR images in the frequency domain can be obtained.
[0033] Then, the frequency features obtained by inverse wavelet transform are processed: ; in, Represents inverse wavelet transform. The i-th stage of CFFM is a joint representation that integrates multimodal low-frequency features representing the stable building body and multimodal high-frequency information representing the fine building edge, taking into account the fine edge characteristics while ensuring the stability of the deep structure.
[0034] ; Frequency information from multi-scale optical and SAR image fusion.
[0035] S5, fusion of spatial and frequency domain features of optical and SAR images.
[0036] In the spatial-frequency domain feature fusion part, multi-layer dilated convolution is used to start from different receptive fields and fully combine global and local information to better fuse multimodal spatial information and complementary frequency information. Finally, the cross-entropy loss function is used to optimize the model parameters.
[0037] This invention utilizes the publicly available SpaceNet6 dataset for experiments and verifies the methods using mainstream approaches such as Deeplabv3+, UNet, and MCANet. Specifically, Deeplabv3+ (OPT) represents a method whose input is only optical images, Deeplabv3+ (SAR) refers to a method whose input is only SAR images, Deeplabv3+ (OS), and UNet (OS) represent methods that use a simple cascade of optical and SAR image channels as multimodal inputs. MCANet is a mainstream multimodal network.
[0038] The specific experimental results are shown in the table below. This invention uses multiple metrics such as OA, Precision, Recall, F1, and IoU to calculate the results: Table 1 Comparative experimental results on the MSAW dataset
[0039] As can be seen from the table above, the method provided by this invention outperforms the comparative experimental methods in all five indicators. Specifically, the comparison with Deeplabv3+ (SAR) and Deeplabv3+ (OPT), which use only single-sensor images, demonstrates the effectiveness of fully utilizing optical and SAR multi-source image data. The comparison with Deeplabv3+ (OS) and Unet (OS), which simply cascade optical and SAR image features for interpretation, demonstrates the effectiveness of the multi-branch network based on Siamese networks in this invention. The comparison results with other advanced multimodal feature fusion methods, such as MCANet, show the effectiveness of the space-frequency structure proposed in this invention, fully demonstrating the superiority of the method proposed in this invention.
[0040] Therefore, this invention employs a building detection method based on optical and SAR image fusion technology, addressing domain discrepancies in the frequency domain and fusing complementary information to reduce false alarms while ensuring the integrity of building outlines. The cross-frequency fusion module combines the advantages of both methods, fusing features at multiple scales to ensure complete and consistent information. Experiments show that this method outperforms mainstream methods in several aspects, resulting in superior detection performance.
[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A building detection method based on optical and SAR image fusion technology, characterized in that, Includes the following steps: S1. Extract spatial domain features from optical and SAR images using twin networks; S2. Spatial domain features of optical and SAR images are extracted from the fusion of channel dimension and spatial dimension using the CBAM attention module. S3. Extract frequency domain features of optical and SAR images using wavelet transform; S4. Frequency characteristics of optical and SAR images fused based on the cross-frequency fusion module CFFM; S5. Utilize multi-layered dilated convolution to fuse the spatial and frequency domain features of optical and SAR images.
2. The building detection method based on optical and SAR image fusion technology according to claim 1, characterized in that, In S1, paired optical and SAR images are selected as inputs to the network. The image features of the optical and SAR images are extracted using a Siamese network, and ResNet50 is used as the backbone network for feature extraction.
3. The building detection method based on optical and SAR image fusion technology according to claim 2, characterized in that, In S2, in the spatial domain branch, deep features of optical and SAR images are integrated through channel attention and spatial attention mechanisms. Simultaneously, shallow optical features with rich detail are fused. The fusion process is specifically represented by the following equation: ； in, Represents channel cascading. These represent the deep optical and SAR image features extracted by the backbone network, respectively.
4. The building detection method based on optical and SAR image fusion technology according to claim 3, characterized in that, In S2, the fused deep information is upsampled and made to the same size as the shallow features of the optical image, then fused with the shallow features to capture fine-grained information. Specifically: ； in, Represents the multimodal features of optical and SAR images in the final spatial domain fusion. This represents the shallow optical features extracted by the backbone network. This represents an upsampling operation.
5. The building detection method based on optical and SAR image fusion technology according to claim 4, characterized in that, In S3, the frequency domain processing section utilizes wavelet transform to extract frequency domain features from optical and SAR images. The wavelet transform is expressed by the following equation. ； in, For the input image features, It corresponds to the low-frequency component. These are the corresponding high-frequency components in the horizontal, vertical, and diagonal directions.
6. The building detection method based on optical and SAR image fusion technology according to claim 5, characterized in that, In S3, the change domain operation is performed continuously to decompose the frequency information of the features, as shown in the following equation: ； in, , i This represents the current stage or level.
7. A building detection method based on optical and SAR image fusion technology according to claim 6, characterized in that, In S4, frequency information fusion is performed based on the cross-frequency fusion module CFFM, as shown in the following equation: ； ； ； in, Representative at the i The stage utilizes frequency information from optical and SAR images fused by CFFM. and Representing the first i Frequency domain information of the optical and SAR images at each stage; , , and Representing the first i The low-frequency information and the corresponding three high-frequency features in the optical image of the stage; , , and Representing the first i The low-frequency information and the corresponding three high-frequency features in the frequency information of the SAR image at this stage.
8. The building detection method based on optical and SAR image fusion technology according to claim 7, characterized in that, In S4, during image feature domain transformation, the main features of the building are in the low-frequency feature map, while the edge features are mainly reflected in the high-frequency feature map. When performing cross-modal frequency information fusion, because the SAR image fully reflects the main building but lacks sufficient edge detail, its low-frequency information is considered during fusion. This process is represented by the following formula: ； ； in, , and Representing the first i Low-frequency information from staged optical images, SAR images, and fused low-frequency information. Representing the first i Three sets of high-frequency information from the phased optical image. These represent the high-frequency information after CFFM processing.
9. A building detection method based on optical and SAR image fusion technology according to claim 8, characterized in that, In S4, the frequency characteristics are obtained by using inverse wavelet transform: ； in, Represents inverse wavelet transform. It is the first i Phase CFFM achieves a joint representation that integrates multimodal low-frequency features representing stable building bodies and multimodal high-frequency information representing fine building edges; ； This represents the frequency information of the fusion of multi-scale optical and SAR images.
10. A building detection method based on optical and SAR image fusion technology according to claim 9, characterized in that, In S5, in the spatial-frequency domain feature fusion part, multi-layer dilated convolution is used to fuse spatial and frequency domain features of optical and SAR images starting from different receptive fields, and finally the cross-entropy loss function is used to optimize the model parameters.