A remote sensing image semantic segmentation method based on channel attention feature fusion
Through the ResNet-50 backbone network and the multi-scale channel attention fusion module, the problems of scale variation and foreground-background imbalance in remote sensing images are solved, and more accurate semantic segmentation of remote sensing images is achieved.
Patent Information
- Application Number
- CN202310454390.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-25
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2043-04-25
AI Technical Summary
Existing remote sensing image semantic segmentation methods suffer from scale variation and foreground-background imbalance problems in processing remote sensing images, which makes it difficult to locate and recognize objects, especially small objects in complex backgrounds.
The ResNet-50 backbone network is used to extract multi-scale features, and the global and local context information are dynamically fused through the multi-scale channel attention fusion module and the attention decoding module to eliminate the semantic gap between the encoder and decoder and achieve the unification of semantic patterns.
It improves the semantic segmentation accuracy of remote sensing images, alleviates the problems of scale change and foreground-background imbalance, and achieves more accurate remote sensing image segmentation.
Smart Images
Figure CN116645598B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to a remote sensing image semantic segmentation method based on channel attention feature fusion. Background Art
[0002] High spatial resolution (HSR) remote sensing images are acquired far from the Earth's surface and typically contain rich geospatial information. Remote sensing semantic segmentation, a fundamental task in remote sensing image understanding, is of great significance for environmental monitoring, post-disaster reconstruction, agriculture, forestry, and urban planning. Compared to natural images, remote sensing images face unique challenges in semantic segmentation due to their unique imaging mechanisms and scene characteristics. Foreground objects in remote sensing images exhibit large scale variations, making them difficult to locate and identify. Furthermore, foreground objects are often hidden in complex backgrounds, leading to foreground-background imbalance. Therefore, accurately capturing image semantic features, mitigating scale variations, and suppressing false positives are crucial for remote sensing image semantic segmentation.
[0003] Recently, many classic convolutional neural networks (CNNs) have achieved good performance in the field of image segmentation. Fully convolutional networks (FCNs) and their extended applications have also been widely used in the field of remote sensing segmentation. However, these methods directly adopt or modify the convolutional neural network architecture, without considering the weak internal features of small objects in remote sensing images, and the results have been less than ideal. U-Net and SegNet use an encoder-decoder structure to enhance deep features with strong spatial detail semantics using shallow features with high spatial resolution. However, they lack the ability to capture global information and have poor recognition capabilities for subtle targets. FPN and its related applications extract multi-scale features through a feature pyramid structure, alleviating the scale variation problem in remote sensing images. However, these methods affect large objects in the image and cannot solve the foreground-background imbalance problem. Summary of the Invention
[0004] In order to overcome the shortcomings of the existing technology, this application provides a remote sensing image semantic segmentation method based on channel attention feature fusion, which extracts features through ResNet-50 and uses multi-scale channel attention to perform multi-scale feature fusion, while paying attention to large and small targets in the image; through the fusion of multi-scale channel dimension information, the semantic gap between the encoder and decoder is eliminated, the semantic pattern is unified, and the false alarm problem is suppressed.
[0005] To achieve the above objectives, the technical solutions adopted in this application are as follows:
[0006] A remote sensing image semantic segmentation method based on channel attention feature fusion, including:
[0007] The remote sensing image to be predicted is input into the feature extraction module to extract the initial features {F1, F2, F3, F4} at four scales;
[0008] The extracted feature F4 is input into the global information module to expand the receptive field and make full use of the global context information to output the same feature and
[0009] The features The initial features {F3, F2, F1} are sequentially passed through the attention fusion module to gradually obtain the features after multi-scale channel attention fusion
[0010] The features Features after fusion with attention Pass the attention decoding module in turn to gradually obtain the decoded features
[0011] The last feature after decoding The final semantic segmentation result is obtained through convolution and upsampling, and then a Softmax classifier is used.
[0012] Preferably, the feature extraction module adopts the ResNet-50 backbone network. When the input remote sensing image size to be predicted is 896×896×3, the scales of the extracted four initial features {F1, F2, F3, F4} are: 224×224×256, 112×112×256, 56×56×1024, 28×28×2048 respectively.
[0013] As a preference, the extracted feature F4 is input into the global information module to expand the receptive field and make full use of the global context information to output the same feature and include:
[0014] S21. Pass feature F4 through three average pooling layers with target sizes of 1, 3, and 6 respectively to obtain three feature maps of different sizes The scales are 1×1×2048, 3×3×2048, and 6×6×2048;
[0015] S22. Feature map The channels are compressed by 1×1 convolution kernel to remove redundant channel information, and then upsampled to the same size as F4 to obtain feature maps. The scale is 28×28×256;
[0016] S23. The three feature maps Perform splicing on the channel dimension to obtain features with a scale of 28×28×768
[0017] S24.Characterize By using a 1×1 convolution kernel, the number of channels is reduced to 256, and a composite feature map with multiple scales is obtained. and The two are exactly the same, with a scale of 28×28×256.
[0018] As an advantage, the feature The initial features {F3, F2, F1} are sequentially passed through the attention fusion module to gradually obtain the features after multi-scale channel attention fusion include:
[0019] The initial features {F3, F2, F1} are input into their respective corresponding attention fusion modules, and the initial feature F3 and the feature Perform attention fusion to obtain the fused features Initial feature F2 and feature Perform attention fusion to obtain the fused features Initial feature F1 and feature Perform attention fusion to obtain the fused features
[0020] Preferably, the attention fusion module performs the following operations:
[0021] S31. Represents the previous level feature input by the attention fusion module, with F i Describe the initial features of the attention fusion module input;
[0022] S32, will Perform bilinear interpolation upsampling to obtain F i The number of channels is reduced by the 1×1 convolution kernel. Then add the elements to get the features
[0023] S33.Characterize Perform global and local context information extraction operations respectively to obtain global context information and local contextual information
[0024] S34. and Perform the addition of the broadcast mechanism and activate it with the Sigmoid function to obtain the multi-scale channel attention weights
[0025] S35. Weight and features The output of the attention fusion module is obtained by element-wise multiplication and weighted averaging.
[0026] As an advantage, the feature Features after fusion with attention Pass the attention decoding module in turn to gradually obtain the decoded features include:
[0027] The features after attention fusion Input into their respective corresponding attention decoding modules, features and features Perform attention decoding to obtain decoded features feature and features Perform attention decoding to obtain decoded features feature and features Perform attention decoding to obtain decoded features
[0028] Preferably, the attention decoding module performs the following operations:
[0029] S41. Represents the previous level feature of the attention decoding module input, Describes the features after attention fusion input to the attention decoding module;
[0030] S42. Upsampled to the same value by bilinear interpolation Same size, generate features
[0031] S43.Characteristics and All pass through the global average pooling operation and a linear layer to generate the feature vector and
[0032] S44. For the eigenvector and Take the average and activate with Sigmoid function to get the attention map representing the importance of the channel, which is consistent with the feature Element-wise multiplication is used to generate recalibrated features
[0033] S45. Features to be recalibrated and features Splicing, and restoring the number of channels through a 1×1 convolution kernel to get the output
[0034] This application provides a remote sensing image semantic segmentation method based on channel attention feature fusion. It uses an attention fusion module to dynamically and adaptively fuse the features extracted by ResNet-50 in a context-aware manner to prevent the deviation caused by only aggregating context information on a global scale. Using the attention decoding module, enough channel features are adaptively fused to achieve the unification of semantic patterns and eliminate the semantic differences between the encoder and decoder. It alleviates the scale change problem and foreground-background imbalance problem in remote sensing images, and obtains more accurate remote sensing image semantic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is the architecture diagram of the remote sensing image semantic segmentation method based on channel attention feature fusion in this application;
[0036] Figure 2 This is a schematic diagram of the global information module of an embodiment of the present application;
[0037] Figure 3 This is a schematic diagram of the attention fusion module of an embodiment of this application;
[0038] Figure 4 Schematic diagram of the attention decoding module of an embodiment of the present application. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0040] In an embodiment, a remote sensing image semantic segmentation method based on channel attention feature fusion is provided, which makes full use of the detail capture ability of CNN, the multi-scale context information capture ability of multi-scale channel attention and the multi-scale feature fusion ability to achieve accurate segmentation of remote sensing images.
[0041] Specifically, such as Figure 1 As shown, a remote sensing image semantic segmentation method based on channel attention feature fusion in this embodiment includes:
[0042] Step S1: Input the remote sensing image to be predicted into the feature extraction module to extract the initial features {F1, F2, F3, F4} of four scales.
[0043] In order to better utilize the stronger expressive power of ResNet through the residual mechanism, the ResNet-50 backbone network is used to extract features of remote sensing images.
[0044] In this embodiment, the remote sensing image to be predicted is 896×896×3, and ResNet-50 is used to extract features of the remote sensing image to extract initial features {F1, F2, F3, F4} at four scales, which are 224×224×256, 112×112×256, 56×56×1024, and 28×28×2048 respectively.
[0045] In this example, ResNet-50 is used to extract features from the input 896×896×3 remote sensing image. The purpose is to extract features at different scales. The high-level network has a larger receptive field and a strong ability to represent semantic information, which can help us accurately locate the target position. The low-level network has a smaller receptive field and a strong ability to represent geometric details, which helps to complete boundary details.
[0046] Step S2: Input the extracted feature F4 into the global information module to expand the receptive field and make full use of the global context information to output the same feature and
[0047] In this example, if Figure 2 As shown in the figure, for the highest-level feature F4, three average pooling operations of different sizes are used to obtain multiple feature maps of different sizes, aggregating contextual information from different regions, enabling the model to better capture global information. Then, 1×1 convolution and upsampling are used to restore the three features to the same size. These features are then concatenated in the channel dimension, and a 1×1 convolution kernel is used to fuse information at each scale and reduce the number of channels.
[0048] As a preference, the extracted feature F4 in step S2 is input into the global information module to expand the receptive field and make full use of the global context information to output the same feature and include:
[0049] S21. Pass feature F4 through three average pooling layers with target sizes of 1, 3, and 6 respectively to obtain three feature maps of different sizes The scales are 1×1×2048, 3×3×2048, and 6×6×2048;
[0050] S22. Feature map The channels are compressed by 1×1 convolution kernel to remove redundant channel information, and then upsampled to the same size as F4 to obtain feature maps. The scale is 28×28×256;
[0051] S23. The three feature maps Perform splicing on the channel dimension to obtain features with a scale of 28×28×768
[0052] S24.Characterize By using a 1×1 convolution kernel, the number of channels is reduced to 256, and a composite feature map with multiple scales is obtained. and The two are exactly the same, with a scale of 28×28×256.
[0053] This step obtains features with global context information through the global information module and The two are identical and are named separately for subsequent distinction.
[0054] Step S3: The initial features {F3, F2, F1} are sequentially passed through the attention fusion module to gradually obtain the features after multi-scale channel attention fusion
[0055] Specifically, this embodiment will feature The initial features {F3, F2, F1} are sequentially passed through the attention fusion module to gradually obtain the features after multi-scale channel attention fusion include:
[0056] The initial features {F3, F2, F1} are input into their respective corresponding attention fusion modules, and the initial feature F3 and the feature Perform attention fusion to obtain the fused features Initial feature F2 and feature Perform attention fusion to obtain the fused features Initial feature F1 and feature Perform attention fusion to obtain the fused features
[0057] In this example, if Figure 3 As shown, for the input feature F i and Depend on Upsampling and F i After reducing the channel dimension, the elements are added to obtain simple fusion features respectively Capturing global context information and local context information to achieve multi-scale channel attention. The two are combined through the addition of the broadcast mechanism and the activation function to obtain the multi-scale channel attention weight. Use this weight to perform weighted averaging on the initial input features to obtain features after fusion at different scales
[0058] Specifically, the attention fusion module performs the following operations:
[0059] S31. Represents the previous level feature input by the attention fusion module, with F i Describes the initial features of the attention fusion module input.
[0060] S32, will Perform bilinear interpolation upsampling to obtain F i The number of channels is reduced by the 1×1 convolution kernel. Then add the elements to get the features
[0061] This step is expressed using the formula as follows:
[0062]
[0063] Where Up(·) is bilinear interpolation upsampling, Conv(·) is 1×1 convolution, and + is element-wise addition. In this embodiment, the upsampling and convolution kernel operations each obtain a size of H i ×W i ×256 features, and then add the elements to obtain the features The size is still H i ×W i ×256.
[0064] S33.Characterize Perform global and local context information extraction operations respectively to obtain global context information and local contextual information
[0065] Global context information The method of obtaining it is to first perform a global average pooling operation and then perform two consecutive 1×1 convolutions; local context information The method of obtaining it is to perform two consecutive 1×1 convolutions. The calculation formula is as follows:
[0066]
[0067]
[0068] where GAP(·) is the global average pooling.
[0069] S34. and Perform the addition of the broadcast mechanism and activate it with the Sigmoid function to obtain the multi-scale channel attention weights
[0070] Obtained multi-scale channel attention weights Size is H i ×W i ×256, the calculation formula is as follows:
[0071]
[0072] Where ⊕ is the broadcast mechanism addition and Sigmoid is the activation function.
[0073] S35. Weight and features The output of the attention fusion module is obtained by element-wise multiplication and weighted averaging.
[0074] The output of the attention fusion module is obtained Size H i ×W i ×256, the calculation formula is as follows:
[0075]
[0076] in and They are all real numbers between 0 and 1 to achieve dynamic and adaptive feature fusion, and × is element-wise multiplication.
[0077] Step S4: Features after fusion with attention Pass the attention decoding module in turn to gradually obtain the decoded features
[0078] Specifically, the characteristics Features after fusion with attention Pass the attention decoding module in sequence to gradually obtain the decoded features include:
[0079] The features after attention fusion Input into their respective attention decoding modules, features and features Perform attention decoding to obtain decoded features feature and features Perform attention decoding to obtain decoded features feature and features Perform attention decoding to obtain decoded features
[0080] In this example, if Figure 4 As shown, for the input features and Depend on Upsampling is performed to obtain Features of the same size . and Global average pooling and linear transformation are performed, spatial compression is performed through global average pooling, global spatial information is embedded, and linear layers are used to represent the dependencies between channels. and After averaging the two, use the Sigmoid function to activate and recalibrate the original input Finally, the calibrated features are concatenated with another upsampled input and adjusted through 1×1 convolution to unify the semantic patterns using sufficient channel features. Gradually eliminate the semantic gap between the encoder and decoder to obtain features.
[0081] Specifically, the attention decoding module performs the following operations:
[0082] S41. Represents the previous level feature of the attention decoding module input, Describes the features after attention fusion input to the attention decoding module;
[0083] S42. Upsampled to the same value by bilinear interpolation Same size, generate features
[0084] S43.Characteristics and All pass through the global average pooling operation and a linear layer to generate the feature vector and
[0085] Generate feature vectors and The size is 1×1×256, and the calculation formula is as follows:
[0086]
[0087]
[0088] Among them, Linear(·) is a linear layer that performs linear transformation;
[0089] S44. For the eigenvector and Take the average and activate with Sigmoid function to get the attention map representing the importance of the channel, which is consistent with the feature Element-wise multiplication is used to generate recalibrated features
[0090] Recalibrated features Size H i ×W i ×256, the calculation formula is as follows:
[0091]
[0092] Where Avg(·) is the average value;
[0093] S45. Features to be recalibrated and features Splicing, and restoring the number of channels through a 1×1 convolution kernel to get the output
[0094] Step S5: decode the last feature The final semantic segmentation result is obtained through convolution and upsampling, and then a Softmax classifier is used.
[0095] This step, features Through a 1×1 convolution and 4 times bilinear interpolation upsampling, the image is restored to the same size as the original input remote sensing image to be predicted, and the final semantic segmentation result is obtained.
[0096] In a specific embodiment, the present application Figure 1 The semantic segmentation model shown in the figure calculates the loss function by comparing the output with the true value and performs backpropagation during training. Specifically, the cross entropy loss is used to calculate the loss between the final remote sensing image segmentation result and the true label.
[0097] Cross entropy CE is a widely used loss in classification. The calculation formula is as follows:
[0098]
[0099] Where x is the segmentation result, y is the true label, C is the number of categories, and N is the batch dimension.
[0100] This example uses a global information module to adjust the size of the receptive field to aggregate contextual information from different regions, improving the model's ability to acquire global information. Multi-scale channel attention is used to capture both global and local contextual information, dynamically and adaptively fusing features at different scales to mitigate issues caused by scale shifts and small objects. Furthermore, during decoding, sufficient channel features are fused to eliminate semantic discrepancies between the encoder and decoder, unifying semantic patterns and mitigating false positives caused by foreground-background imbalance in remote sensing images.
[0101] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A remote sensing image semantic segmentation method based on channel attention feature fusion, characterized in that: The remote sensing image semantic segmentation method based on channel attention feature fusion includes: Input the remote sensing image to be predicted into the feature extraction module to extract the initial features of four scales ; The features extracted Input the global information module to expand the receptive field and make full use of the global context information to output the same features and ; The features With the initial features Through the attention fusion module in sequence, the features after multi-scale channel attention fusion are gradually obtained ; The features Features after fusion with attention Pass the attention decoding module in turn to gradually obtain the decoded features ; The last feature after decoding Through convolution and upsampling, the final semantic segmentation result is obtained using the Softmax classifier; The attention decoding module performs the following operations: S41. Represents the previous level feature of the attention decoding module input, Describes the features after attention fusion input to the attention decoding module; S42. Upsampled to the same value by bilinear interpolation Same size, generate features ; S43.Characteristics and All pass through the global average pooling operation and a linear layer to generate the feature vector and ; S44. For the eigenvector and Take the average and activate it with Sigmoid function to get the attention map representing the importance of the channel, which is consistent with the feature Element-wise multiplication is used to generate recalibrated features ; S45. Features to be recalibrated and features Splicing and passing The convolution kernel recovers the number of channels and gets the output .
2. The remote sensing image semantic segmentation method based on channel attention feature fusion according to claim 1 is characterized in that: The feature extraction module uses the ResNet-50 backbone network. When the input remote sensing image to be predicted is of size When , four initial features are extracted The scales are: , , , .
3. The remote sensing image semantic segmentation method based on channel attention feature fusion according to claim 2 is characterized in that: The features to be extracted Input the global information module to expand the receptive field and make full use of the global context information to output the same features and ,include: S21.Character Three feature maps of different sizes are obtained by three average pooling layers with target sizes of 1, 3 and 6 respectively. , the scales are , , ; S22. Feature map Through The convolution kernel is used to compress the channel to remove redundant channel information, and then upsample to The same size, get the feature map , the scales are ; S23. The three feature maps Perform splicing on the channel dimension and obtain the scale Features ; S24.Characterize pass The convolution kernel reduces the number of channels to 256 and obtains a composite feature map that combines multiple scales. and , the two are exactly the same, the scale is .
4. The remote sensing image semantic segmentation method based on channel attention feature fusion according to claim 1 is characterized in that The characteristics With the initial features Through the attention fusion module in sequence, the features after multi-scale channel attention fusion are gradually obtained ,include: The initial features Input into their respective attention fusion modules respectively, the initial features and features Perform attention fusion to obtain the fused features , initial features and features Perform attention fusion to obtain the fused features , initial features and features Perform attention fusion to obtain the fused features .
5. The remote sensing image semantic segmentation method based on channel attention feature fusion according to claim 4 is characterized in that: The attention fusion module performs the following operations: S31. represents the previous level feature input by the attention fusion module, Describe the initial features of the attention fusion module input; S32, will Perform bilinear interpolation upsampling to obtain , pass The convolution kernel is used to reduce the number of channels to obtain , then add the elements to get the features ; S33.Characterize Perform global and local context information extraction operations respectively to obtain global context information and local contextual information ; S34. and Perform the addition of the broadcast mechanism and activate it with the Sigmoid function to obtain the multi-scale channel attention weights ; S35. Weight and features 、 The output of the attention fusion module is obtained by element-wise multiplication and weighted averaging. .
6. The remote sensing image semantic segmentation method based on channel attention feature fusion according to claim 1, characterized in that: The characteristics Features after fusion with attention Pass the attention decoding module in turn to gradually obtain the decoded features ,include: The features after attention fusion Input into their respective corresponding attention decoding modules, features and features Perform attention decoding to obtain decoded features ,feature and features Perform attention decoding to obtain decoded features ,feature and features Perform attention decoding to obtain decoded features .