SAR (Synthetic Aperture Radar) image building area extraction method based on frequency domain attention and multi-level context perception

Abstract

The invention discloses an SAR image building area extraction method based on frequency domain attention and multi-level context perception, and belongs to the technical field of image segmentation. The method comprises the following steps: constructing an FHRNet-MLCA deep learning image segmentation model; and inputting the data set into the neural network model for training to obtain model parameters. And inputting a to-be-predicted SAR image into the trained model to obtain a building area extraction result. By improving a network structure, adding a frequency domain attention module at a coding end, adding a multi-scale context sensing module at a decoding end, and improving a feature fusion mode, the extraction capability of the network on a small target building area is effectively enhanced, and the problem that the edges of a complex building area are difficult to distinguish is improved; and the extraction precision of the building area of the SAR image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for extracting building areas from SAR images based on frequency domain attention and multi-level context perception. Background Art

[0002] Remote sensing has long been humanity's most effective means of observing the Earth and understanding its features. Numerous remote sensing data products provide a rich and reliable data source for surface cover monitoring, enabling continuous advancements in remote sensing image processing technology and playing an irreplaceable role in achieving sustainable development strategic goals. Traditional remote sensing building information extraction relies heavily on texture and spatial geometry, requiring manual feature design, which is complex and inefficient. With the advent of the era of remote sensing big data, massive amounts of multi-source, heterogeneous remote sensing data are placing higher demands on the computing power and processing methods required for image processing. Traditional extraction methods are no longer able to meet the precision and speed requirements of remote sensing big data.

[0003] Deep learning is one of the major breakthroughs in the field of artificial intelligence in recent years and has been widely applied in fields such as natural language processing, computer vision, and multimedia. The key difference between deep learning and traditional pattern recognition methods is that it does not rely on manually designed features, but instead uses network models to automatically learn features from big data. The resulting models are not only highly efficient but also possess stronger representational capabilities.

[0004] Most current SAR image feature segmentation methods are based on deep learning methods. However, when extracting built-up areas, effective distinction is difficult due to the widespread scale non-uniformity, the presence of a large number of small target areas, and the complex edges of built-up areas. This paper combines a deep learning semantic segmentation network with a multi-scale context-aware module and introduces a channel attention mechanism in the frequency domain. This focuses the segmentation focus more on small target areas and refines the edges of built-up areas, forming a model structure with stronger expressiveness and generalization capabilities. This model has stronger overall performance and improved recognition accuracy. Summary of the Invention

[0005] The present invention aims to solve the problems existing in the prior art, such as the insufficient fusion of multi-scale features and the difficulty in distinguishing small target areas and segmentation edges, and to provide a SAR image building that can identify small target areas and improve the accuracy of image segmentation edges.

[0006] The present invention adopts an improved high-resolution feature extraction network HRNet as the encoder, adopts a frequency-domain attention mechanism to focus on the high-frequency components in the network, and outputs a four-layer multi-scale feature image to produce a rich multi-resolution representation; a multi-scale context-aware module is used to enhance the extraction of effective feature information at each scale; the decoder uses a self-attention mechanism to model complex global context and introduces pixel-level features to further improve the accuracy of prediction.

[0007] In order to achieve the above object, the present invention provides a method for extracting building areas from SAR images based on frequency domain attention and multi-level context perception, comprising the following steps:

[0008] Step S1: Construct the FHRNet-MLCA semantic segmentation network model, which consists of three modules: encoder, decoder, and multi-level context perception module. The specific steps include:

[0009] In the encoder module, an image with an input resolution of H×W×3 is first processed through two downsampling layers and four residual modules with a bottleneck structure, reducing the resolution to H / 4×W / 4×C. Pooling then occurs, with the two parallel branches before and after pooling passing through four residual modules respectively. A frequency attention module then extracts channel-wise attention. The two branches then exchange information through pooling and upsampling. The branch with the smallest dimension is then further downsampled. This process is repeated three times, with the number of information exchanges occurring 1, 4, and 3 times, respectively. This results in four layers of multi-scale feature maps with resolutions of H / 4×W / 4×C, H / 8×W / 8×2C, H / 16×W / 16×4C, and H / 32×W / 32×8C. These four feature maps serve as the encoder output and are input to the decoder module and the multi-level context-aware module.

[0010] The above frequency domain attention module: cuts the channel of the feature map into 2 N Components, DCT transform is performed on each component to obtain the frequency component and spliced ​​on the channel, and then output as the channel weight coefficient through FC and Sigmoid in turn, and finally multiplied with the original feature map on the channel to obtain the weighted feature image.

[0011] The multi-level context-aware module (MLCA) upsamples the input four-layer multi-scale resolution feature map step by step. The specific method is to concat the low-dimensional feature map with the high-dimensional feature map after upsampling, and then adjust the number of channels back to the original to obtain a new high-dimensional feature map. Repeat the above steps to finally obtain four new feature maps with different resolutions. The four feature maps are adjusted to have a channel number of 2 through 1×1 convolution. M , get pixel features of different resolutions, and then expand them into four two-dimensional data of different lengths, and adjust the number of channels to 2 M-1, counted as key value Q, the sizes of the four Qs are

[0012] The input of the decoder module is the output of the encoder module and the MLCA module, and the output is the final result of the model. The decoder module contains the ORR module and the MSA module. The ORR module upsamples the input four-layer multi-scale resolution feature map to make the dimensions consistent, and then concats to obtain a feature map of size H / 4×W / 4×15C, calculates the object area representation of the feature map, and outputs the key values ​​K and V; calculates the pixel representation of the feature map, and outputs the pixel features. The MSA module calculates the self-attention of the key values ​​K and V output by the ORR module and the key value Q output by the context perception module, respectively, to obtain four sets of feature vectors, and then flattens them along the spatial dimension to restore them to the feature map of the original dimension size. These four sets of feature maps are concat with the pixel features output by the ORR module, and then the dimension size and number of channels are adjusted to match the true value. Figure 1 The final output of the network is obtained.

[0013] The calculation formula for self-attention is as follows:

[0014]

[0015] Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively. k is the dimension of each column feature in the key matrix.

[0016] The model defines a multi-scale composite loss (Mco_loss) as the model's loss. Mco_loss consists of four losses. The first loss is the primary loss. The ground truth image is downsampled and then compared with the decoder module's output, calculated as loss1. This loss is calculated as a weighted sum of the Dice loss and the cross-entropy loss. The cross-entropy loss effectively prevents network oscillations during early training and helps convergence, while the Dice loss is sensitive to extreme pixels and helps identify small object areas. The second, third, and fourth losses are auxiliary losses. In the multi-level context-aware module, four new feature images are generated through upsampling. The feature images in the second, third, and fourth layers are resized to match the number of categories through 1×1 convolutions to form a coarse segmentation result. The ground truth image is downsampled to match the dimensions of the coarse segmentation result, resulting in three auxiliary losses: loss2, loss3, and loss4. The auxiliary losses are calculated using the cross-entropy loss function. The final total loss is the weighted sum of these four losses, calculated as loss = λ1loss1+λ2loss2+λ3loss3+λ4loss4. The loss function calculation formula for the main losses described in (4) is as follows:

[0017]

[0018]

[0019]

[0020] Where α is a constant between 0 and 1, N is the number of samples, and y i is the true label, is the model prediction value, and ∈ is a small constant used to avoid the denominator being zero.

[0021] Step S2: Divide the SAR dataset into training, validation, and test sets proportionally. Input the training set into the FHRNet-MLCA network constructed in S1 for parameter training. The loss function is Mco_loss defined in S1, and the optimizer uses the adaptive optimization algorithm Adam. Set the initial learning rate and use an exponential decay learning strategy. Set the batch size and epochs for model training. Evaluate the model using the validation set, setting the training evaluation metric to MioU. Select the model with the highest MioU as the optimal model.

[0022] Step S3: During prediction, remove the auxiliary output branch of the FHRNet-MLCA model, which is used to calculate Mco_loss. This auxiliary branch only improves the model's focus on feature maps at different scales during training. Therefore, it is removed during image prediction, and only the model's final output is used as the image prediction result. The SAR image to be tested is input, cropped into H×W×3 slices, and fed into the optimal model. The outputs are concatenated in the original format to obtain the final built-up area extraction result.

[0023] The advantages of the present invention compared with the prior art are:

[0024] The high-resolution HRNetV2 network is used for building area extraction in SAR images. During training, the HRNetV2 network consistently maintains high-resolution feature map representation and repeatedly integrates multi-scale resolution information, incorporating a certain degree of spatial attention. Furthermore, to better identify small target areas (high-frequency components) in the image, frequency-domain attention is introduced on the channel level, further strengthening the network's feature extraction capabilities and making building area extraction more accurate.

[0025] The feature fusion method has been improved, and a multi-scale context perception module has been introduced to obtain the contextual information of features of different scales while upsampling. Self-attention calculation is performed with the decoder backbone network, which effectively solves the problem of insufficient fusion of features of different scales, expands the receptive field and retains the high-level semantic information of the image, thereby improving the segmentation accuracy of the model.

[0026] The introduction of a multi-scale loss not only calculates the loss on the network's final output but also utilizes information from feature images at different scales, enabling the network to better adapt to objects of varying sizes and proportions. This is crucial for extracting built-up areas of varying sizes. Simultaneously, the multi-scale loss enables the network to simultaneously focus on local details and global structure, improving its understanding of internal relationships in image data, refining building segmentation edges, and improving building area extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the method for extracting building areas from SAR images based on frequency domain attention and multi-level context perception

[0028] Figure 2 This is the overall structure diagram of the model.

[0029] Figure 3 This is the network structure diagram of the encoder.

[0030] Figure 4 This is the network structure diagram of the frequency attention module.

[0031] Figure 5 Figure 2 is the network structure diagram of the ORR module.

[0032] Figure 6 This is the network structure diagram of MLCA.

[0033] Figure 7 This is the MSA network structure diagram.

[0034] Figure 8 The segmentation result visualization diagram.

[0035] Figure 9 A visualization of the extraction results. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings and implementation methods.

[0037] The present invention comprises the following steps:

[0038] Hainan Island was selected as the experimental study area, and Sentinel-1VV polarimetric GRD-level data were used as the image data source. The images were output into an image format suitable for data analysis through a preprocessing module. The preprocessing module includes five methods: orbit correction, thermal noise removal, radiometric calibration, terrain correction, and decibel conversion. The study area data was obtained by cropping and splicing multiple remote sensing images.

[0039] The DLR 2019 global built-up area segmentation result is selected as the ground truth image. The two images are registered to make the coordinate system consistent. The images are cropped using a sliding window of size 256×256 with overlapping parts to obtain the corresponding image-label dataset. At this time, the image data is single-channel PNG format data, and the label data is binary PNG format data. There are 6319 images.

[0040] Take an image-label dataset as input and calculate the ratio of the built-up area of ​​the binary labeled data to the entire image for each pair of slices in the dataset. Remove slices with no built-up areas or with a built-up area ratio of less than 5%. Split the remaining slices into training, test, and validation sets in a ratio of 7:2:1.

[0041] The dataset was augmented using data augmentation methods. Using Python, we designed five random augmentation methods: rotation, flipping, scaling, contrast, and Gaussian noise. We then used random numbers to control the training set's use of one or more augmentation methods, repeating this process five times. This yielded six times the number of images, or 6,319 × 0.7 × 6 = 26,540. The augmented image data was converted to a three-channel format, while the label data remained in a single-channel format, yielding the final dataset.

[0042] Construct the FHRNet-MLCA semantic segmentation network model, which consists of an encoder module, a multi-level context perception module, and a decoder module. The specific steps include:

[0043] In the encoder module, a training image of 256×256×3 is input. It first passes through two downsampling layers and four residual modules with a bottleneck structure, reducing the resolution to 64×64×256. Pooling then occurs, with the two parallel branches before and after pooling passing through four residual modules. A frequency attention module then extracts channel-wise attention. The two branches then exchange information through pooling and upsampling. The branch with the smallest dimension is then further downsampled. This process is repeated three times, with the number of information exchanges occurring 1, 4, and 3 times, respectively. This results in four layers of multi-scale feature maps with resolutions of 64×64×32, 32×32×64, 16×16×128, and 8×8×256, respectively. These four feature maps serve as the encoder output and are input to the decoder module and the multi-level context-aware module.

[0044] The Multi-Level Context-Aware (MLCA) module progressively upsamples the input four-layer multi-scale resolution feature maps. Specifically, the low-dimensional feature map is upsampled and concatenated with the high-dimensional feature map. The number of channels is then adjusted back to the original number to obtain a new high-dimensional feature map. These steps are repeated to obtain four new feature maps with different resolutions. The four feature maps are then resized to 512 channels using a 1×1 convolution to obtain pixel features of varying resolutions. These features are then expanded into four two-dimensional data of varying lengths, with the number of channels adjusted to 256. These are counted as key values ​​Q, with the sizes of the four Qs being 4096×256, 1024×256, 256×256, and 64×256, respectively.

[0045] The input of the decoder module is the output of the encoder module and the multi-level context perception module, and the output is the final result of the model. The decoder module contains an object region representation module (ORR) and a multi-level self-attention module (MSA). The ORR module upsamples the input four-layer multi-scale resolution feature map to make the dimensions consistent, and then concats to obtain a feature map of size 64×64×720. It calculates the object region representation of the feature map and outputs the key values ​​K and V. The sizes of K and V are 256×2 and 2×256 respectively; it calculates the pixel representation of the feature map and outputs the pixel feature. The pixel feature size is 256×256×512. The MSA module calculates self-attention on the key values ​​K and V output by the ORR module and the key value Q output by the context perception module, respectively, to obtain four sets of feature vectors, and then flattens them along the spatial dimension to restore them to the feature map of the original dimension size. These four sets of feature maps are concat with the pixel features output by the ORR module, and then the dimension size and number of channels are adjusted to match the true value. Figure 1 The final output of the network is 64×64×2.

[0046] Mco_loss consists of four losses. The first loss is the primary loss. The ground truth image is downsampled to 64×64×2 and the loss is calculated with the decoder module's output, referred to as loss1. This loss is calculated as a weighted sum of the Dice loss and the cross-entropy loss. The second, third, and fourth losses are auxiliary losses. In the multi-level context-aware module, four new feature images are generated through upsampling. The feature images in the second, third, and fourth layers are resized to match the number of categories through 1×1 convolutions, resulting in a coarse segmentation result. The ground truth image is downsampled to match the dimensions of the coarse segmentation result, resulting in three auxiliary losses: loss2, loss3, and loss4. The auxiliary losses are calculated using the cross-entropy loss function. The final total loss is the weighted sum of these four losses, calculated as loss = loss1 + 0.2loss2 + 0.15loss3 + 0.15loss4.

[0047] The training set of the dataset described in 1 was input into the FHRNet-MLCA network constructed in 3 for parameter training. The optimizer used the adaptive optimization algorithm Adam, and the internal parameter momentum was set to 0.9. The initial learning rate was set to 0.005, and an exponential decay learning strategy was used. The batch size for model training was set to 4 and the number of epochs was set to 30. The model was evaluated using the validation set of the dataset described in 1. The training evaluation metric was set to MioU, and the model with the highest MioU was selected as the optimal model.

[0048] Remove the three auxiliary losses in the model and use the output of the decoder module as the prediction result of the model. In order to verify the performance of the model, the optimal model is tested using the test set of the dataset described in 1. The test indicators are the mean intersection-over-union (MioU) and MPA. Table 1 shows the experimental results. Figure 7 The following is a visualization of the prediction results. The SAR image to be tested is input, and after being preprocessed and cropped into 256×256×3 slices, it is input into the optimal model. The output results are spliced ​​in the original way to obtain the final building area extraction result map. Figure 8 Visualize the extracted results.

[0049] Table 1: Comparison of model performance before and after improvement

[0050]

Claims

1. This patent proposes a method for extracting built-up areas from SAR images based on frequency-domain attention and multi-level context perception. This method constructs a semantic segmentation network model called FHRNet-MLCA, which uses an improved HRNet as the feature extraction network. On this basis, a frequency-domain attention module and a multi-scale context perception module are added, and the model's loss function is redesigned to achieve attention to high-frequency components in the image and extract effective feature information at different scales. This method can effectively identify small target areas in built-up areas in SAR images and improve the accuracy of image segmentation edges. The specific steps are as follows: S1: Construct the FHRNet-MLCA semantic segmentation network model, which consists of three modules: encoder, decoder and multi-level context perception module. The input of the encoder module is the SAR building area dataset. The module uses the HRNetV2 network structure as the feature extraction framework, and connects the frequency domain attention module in the middle of the network to enhance the channel dimension. Finally, it outputs a four-layer multi-scale resolution feature map. The multi-level context aware (MLCA) module takes the output of the encoder module as input and outputs a four-layer key value Q. This module upsamples the feature maps of four layers at different resolutions step by step to generate four new feature images. It then uses convolutional modules to obtain pixel feature representations for each layer, and then expands them in two dimensions to obtain the final four-layer key value Q. The decoder module takes as input the outputs of the encoder module and the multi-level context-aware module, and outputs the model's final result. The decoder aggregates the four-layer multi-scale resolution feature map obtained from the encoder module and computes the feature map's pixel features, key values ​​K, and key values ​​V using the object region representation (ORR) module. The multi-level context aware (MSA) module computes multi-scale attention on K, V, and the four-layer key values ​​Q obtained by the MLCA module. This attention is then combined with the pixel features, and the final output is obtained after adjusting the number of channels to match the number of categories. The model defines a network loss called multi-scale composite loss (Mco_loss), which consists of four losses. The first loss is the primary loss, calculated by downsampling the model's final output and the ground-truth image. The loss function is calculated as a weighted sum of the Dice loss function and the cross-entropy loss function. The second, third, and fourth losses are auxiliary losses. In the MLCA module, four new feature images are generated through upsampling. The feature images of the second, third, and fourth layers are adjusted to have the same number of channels and categories through 1×1 convolution to form a coarse segmentation result. The ground-truth image is downsampled to the same dimension as the coarse segmentation result, and three auxiliary losses are calculated. The auxiliary losses are calculated using the cross-entropy loss function. The final Mco_loss is the weighted sum of these four losses. S2: Use the data set obtained in S2 to input the FHRNet-MLCA network model built in S3 for parameter training, use the Mco_loss built in S3 to adjust and optimize the model parameters, and output the optimal model. S3: During prediction, the auxiliary output branch of the FHRNet-MLCA model, used to calculate Mco_loss, is removed. This auxiliary branch only improves the model's focus on feature maps at different scales during training. Therefore, it is removed during image prediction, leaving only the model's final output as the image prediction result. During prediction, the SAR image is input and predicted using the trained model, ultimately extracting the built-up areas within the image.

2. The method for extracting building areas from SAR images based on frequency domain attention and multi-level context perception according to claim 1, characterized in that: As described in S1, the FHRNet-MLCA semantic segmentation network model is constructed. This model is an encoder-decoder structure consisting of an encoder module, a multi-level context perception module, and a decoder module. The specific steps are: (1) In the encoder module, an image with a resolution of H×W×3 is input. It first passes through two downsampling layers and four residual modules with a bottleneck structure, and the resolution becomes H / 4×W / 4×C. Then, pooling is performed. The two parallel branches before and after pooling pass through four residual modules respectively. Then, the frequency attention module extracts the attention on the channel. After that, the two branches exchange information through pooling and upsampling operations. Then, the branch with the smallest dimension is further downsampled. The above process is repeated three times, where the number of information exchanges is 1, 4, and 3 respectively. Finally, four layers of multi-scale resolution feature maps are obtained, with resolutions of H / 4×W / 4×C, H / 8×W / 8×2C, H / 16×W / 16×4C, and H / 32×W / 32×8C respectively. These four feature maps are used as the output of the encoder and input into the decoder module and the multi-level context perception module. For the frequency domain attention module described in (1): cut the channel of the feature map into 2 N Components, DCT transform is performed on each component to obtain the frequency component and spliced ​​on the channel, and then output as the channel weight coefficient through FC and Sigmoid in turn, and finally multiplied with the original feature map on the channel to obtain the weighted feature image. (2) The multi-level context-aware module (MLCA) upsamples the input four-layer multi-scale resolution feature map step by step. The specific method is to concat the low-dimensional feature map with the high-dimensional feature map after upsampling, and then adjust the number of channels back to the original to obtain a new high-dimensional feature map. Repeat the above steps to finally obtain four new feature maps with different resolutions. The four feature maps are adjusted to have a channel number of 2 through 1×1 convolution. M , get pixel features of different resolutions, and then expand them into four two-dimensional data of different lengths, and adjust the number of channels to 2 M-1 , counted as key value Q, the sizes of the four Qs are (3) The input of the decoder module is the output of the encoder module and the MLCA module, and the output is the final result of the model. The decoder module contains an object region representation module (ORR) and a multi-level self-attention module (MSA). The ORR module upsamples the input four-layer multi-scale resolution feature map to make the dimensions consistent, and then concats to obtain a feature map of size H / 4×W / 4×15C, calculates the object region representation of the feature map, and outputs the key values ​​K and V; calculates the pixel representation of the feature map, and outputs the pixel features. The MSA module calculates self-attention on the key values ​​K and V output by the ORR module and the key value Q output by the context perception module, respectively, to obtain four sets of feature vectors, and then flattens them along the spatial dimension to restore them to the feature map of the original dimension size. These four sets of feature maps are concat with the pixel features output by the ORR module, and then the dimension size and number of channels are adjusted to be consistent with the true value map to obtain the final output of the network. The calculation formula for self-attention is as follows: Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively. k is the dimension of each column feature in the key matrix. (4) Define the multi-scale composite loss (Mco_loss) as the loss of the model. Mco_loss consists of four losses. The first loss is the main loss. The truth image is downsampled and the loss is calculated with the output of the decoder module, which is counted as loss1. The loss function is calculated as the weighted sum of the Dice loss function and the cross entropy loss function. The cross entropy loss function effectively avoids network oscillation in the early training and helps the network converge, while the Dice loss is more sensitive to extreme pixels and helps to identify small target areas. The second, third, and fourth losses are auxiliary losses. In the MLCA module, four layers of new feature images are generated by upsampling. The feature images of the second, third, and fourth layers are adjusted to the same number of channels and categories through 1×1 convolution to form a coarse segmentation result. The truth image is downsampled to the same dimension as the coarse segmentation result, and three auxiliary losses are calculated, which are counted as loss2, loss3, and loss4. The auxiliary losses are calculated using the cross entropy loss function. The final total loss is the weighted sum of these four losses, calculated as loss = λ1loss1+λ2loss2+λ3loss3+λ4loss4. The loss function calculation formula for the main losses described in (4) is as follows: Where α is a constant between 0 and 1, N is the number of samples, and y i is the true label, is the model prediction value, and ∈ is a small constant used to avoid the denominator being zero.

3. The method for extracting building areas from SAR images based on frequency domain attention and multi-level context perception according to claim 1, characterized in that: Training of the model described in S2. The specific steps are: Divide the SAR dataset into training, validation, and test sets proportionally. Input the training set into the FHRNet-MLCA network constructed in S1 for parameter training. Use the Mco_loss loss function defined in S1 and the Adam adaptive optimization algorithm as the optimizer. Set the initial learning rate and use an exponential decay learning strategy. Set the batch size and epochs for model training. Evaluate the model using the validation set, setting the training evaluation metric to MioU (MioU). Select the model with the highest MioU as the optimal model.

4. The method for extracting building areas from SAR images based on frequency domain attention and multi-level context perception according to claim 1, characterized in that: The prediction of the model described in S3 is as follows: During prediction, the auxiliary output branch of the FHRNet-MLCA model, used to calculate Mco_loss, was removed. This auxiliary branch only improves the model's focus on feature maps at different scales during training. Therefore, it was removed during image prediction, leaving only the model's final output as the image prediction result. The SAR image to be tested was cropped into H×W×3 slices and fed into the optimal model. The outputs were concatenated using the original method to obtain the final built-up area extraction result.

Images