A semantic segmentation method for high-resolution remote sensing images based on supervised self-attention network

By introducing a supervised self-attention network in semantic segmentation of high-score remote sensing images, the network's receptive field and semantic feature representation are enhanced, and the problems of low accuracy and confusing feature relationships in the existing technology are solved, achieving a more efficient semantic segmentation effect .

CN114549405BActive Publication Date: 2024-12-03CHINA UNIV OF GEOSCIENCES (WUHAN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210022342.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-10
Publication Date
2024-12-03
Estimated Expiration
2042-01-10

AI Technical Summary

Technical Problem

The existing high-score remote sensing image semantic segmentation method has problems of low accuracy and confusing feature relationships when acquiring context spatial information, resulting in poor segmentation effect.

Method used

Using a method based on supervised self-attention network, including basic feature extraction module, channel self-attention module, category self-attention module and spatial self-attention module, to enhance the network's receptive field through effective supervision information and build advanced semantics Feature and feature classification dependencies.

Benefits of technology

The accuracy of semantic segmentation of remote sensing images is improved, the contextual relationship between features is clarified, and the semantic segmentation effect is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549405B_ABST
    Figure CN114549405B_ABST
Patent Text Reader

Abstract

The invention discloses a high-resolution remote sensing image semantic segmentation method based on a supervised self-attention network, comprising the following steps: constructing a remote sensing image semantic segmentation network, the network comprising: a basic feature extraction module, a channel self-attention module, a category supervised self-attention module, a spatial supervised self-attention module and an advanced feature fusion module; obtaining remote sensing images taken by a drone or a satellite as a training set, using the training set to train the remote sensing image semantic segmentation network to obtain a trained network model; using the trained network model to segment the high-resolution remote sensing image to be segmented, and obtaining the target object in the high-resolution remote sensing image to be segmented. The invention has the beneficial effect of improving the overall accuracy of remote sensing image semantic segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation, and in particular relates to a high-resolution remote sensing image semantic segmentation method based on a supervised self-attention network. Background Art

[0002] With the continuous development of satellite remote sensing technology, it has become easy to obtain large-scale high-resolution remote sensing images. Detailed observations of surface objects can be obtained through the airborne sensors of drones and satellites. At the same time, high-resolution remote sensing images provide richer textures and spatial features, and contain rich semantic information. Therefore, semantic segmentation of high-resolution remote sensing images can obtain the shapes and classifications of different objects in remote sensing images, obtain visual semantic segmentation images, replace manual information analysis and mining of remote sensing images, and apply the obtained information to different fields such as urban planning, environmental monitoring, military reconnaissance, and intelligent transportation.

[0003] Thanks to the powerful feature expression ability of the fully convolutional deep network (FCN), many semantic segmentation methods can achieve good segmentation of remote sensing images. However, simply stacking multiple convolutional layers cannot effectively obtain a larger receptive field, which is reflected in the fact that the network can only obtain limited contextual spatial information during feature extraction, which will affect the overall accuracy of semantic segmentation of high-resolution remote sensing images and lead to incomplete understanding of scenes and objects in the image. Therefore, various semantic segmentation methods are trying to improve the receptive field of the network and obtain as much contextual spatial information as possible in the scenes of high-resolution remote sensing images. At present, the better method is the semantic segmentation method based on the self-attention mechanism, such as channel self-attention, spatial self-attention and other methods. The contextual spatial relationship of the full-image features of high-resolution remote sensing images is constructed through the self-attention mechanism to improve the receptive field extracted by the network. However, the intermediate layer features extracted by the network are numerous and complex. The spatial feature relationship and channel feature relationship obtained through the self-attention mechanism have high computational complexity, and the information acquisition is relatively redundant. The description of the contextual scene and object information of high-resolution remote sensing images is not clear. Therefore, the self-attention mechanism will select the wrong context-dependent spatial relationship in the process of feature re-expression, which will affect the effect of semantic segmentation.

[0004] In general, the above-mentioned self-attention mechanism method has no effective supervision in the process of constructing feature context information and category information, and obtains redundant information, which leads to confusion of different contextual relationships and causes a decrease in the overall accuracy of semantic segmentation of high-resolution remote sensing images. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a high-resolution remote sensing image semantic segmentation method based on a supervised self-attention network, which aims to solve the technical problems of low segmentation accuracy and confusing relationship between segmented objects and scene context in existing semantic segmentation algorithms.

[0006] The technical solution of the present invention provides a high-resolution remote sensing image semantic segmentation method based on a supervised self-attention network, and the method comprises the following steps:

[0007] S1. Construct a remote sensing image semantic segmentation network, which includes: a basic feature extraction module, a channel self-attention module, a category supervision self-attention module, a spatial supervision self-attention module and an advanced feature fusion module;

[0008] S2, obtaining remote sensing images taken by a drone or satellite as a training set, and using the training set to train the remote sensing image semantic segmentation network to obtain a trained network model;

[0009] S3. Use the trained network model to segment the high-resolution remote sensing image to obtain the target object in the high-resolution remote sensing image to be segmented.

[0010] Furthermore, the basic feature extraction module is used to extract basic features of the input remote sensing image and output the basic features of the image. The module uses Dialated ResNet as the backbone network.

[0011] Furthermore, the channel self-attention module is used to construct the relationship between channels of the extracted basic features, obtain the weights between the relationships between the basic feature channels, and multiply them with the basic features to obtain the channel self-attention enhanced features.

[0012] Furthermore, the category-supervised self-attention module is used to perform category semantic feature learning on basic features, construct the relationship between categories and features, and weight them to the basic features to obtain category-supervised self-attention features.

[0013] Furthermore, the spatial supervised self-attention module is used to learn spatial context features of basic features, obtain spatial pixel context prior relationships, intra-class features and heterogeneous features, and supervise through an ideal spatial similarity matrix to obtain spatial supervised self-attention features.

[0014] The advanced feature fusion module is used to cascade fuse the channel self-attention enhancement features, the category supervision self-attention features and the space supervision self-attention features, and upsample the fused feature map to obtain the final semantic segmentation category result.

[0015] Compared with the prior art, the beneficial effects of the present invention include: utilizing the self-attention mechanism and providing effective supervision information at the same time, which can increase the receptive field acquired by the network as much as possible, enhance the semantic representation of the intermediate layer module, and then obtain more advanced semantic features, thereby effectively constructing the dependency relationship of feature classification and enhancing the accuracy of semantic segmentation of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a method for semantic segmentation of high-resolution remote sensing images based on a supervised self-attention network provided by the present invention;

[0017] Figure 2 It is a structural diagram of the remote sensing image semantic segmentation network of the present invention;

[0018] Figure 3 Schematic diagram of the channel self-attention module of the present invention;

[0019] Figure 4 Schematic diagram of the category-supervised self-attention module of the present invention;

[0020] Figure 5 It is a schematic diagram of the spatial supervised self-attention module. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0022] This paper provides a high-resolution remote sensing image semantic segmentation method based on supervised self-attention network. Please refer to Figure 1 , Figure 1 It is a flow chart of the method of the present invention; the method comprises the following steps:

[0023] S1. Construct a remote sensing image semantic segmentation network, which includes: a basic feature extraction module, a channel self-attention module, a category supervision self-attention module, a spatial supervision self-attention module and an advanced feature fusion module;

[0024] Please refer to Figure 2 , Figure 2 It is a structural diagram of the remote sensing image semantic segmentation network of the present invention;

[0025] The basic feature extraction module is used to extract basic features of the input remote sensing image and output the basic features of the image. The module uses Dialated ResNet as the backbone network, which is a variant of ResNet. The maximum pooling downsampling layer in the last two stages is removed and replaced with a hollow convolution to keep the spatial resolution of the feature map unchanged during the calculation process of the last two stages.

[0026] Please refer to Figure 3 , Figure 3It is a schematic diagram of the channel self-attention module of the present invention; the channel self-attention module is used to construct the relationship between channels of the extracted basic features, obtain the weights between the relationships between the basic feature channels, and multiply them with the basic features to obtain channel self-attention enhanced features; the channel self-attention module includes feature dimension transformation and matrix multiplication, obtains a channel self-attention matrix of size C×C, and weights the channel information to the original basic features.

[0027] Please refer to Figure 4 , Figure 4 It is a schematic diagram of the category supervision self-attention module of the present invention; the category supervision self-attention module is used to learn the category semantic features of the basic features, construct the relationship between the category and the feature, and weight it to the basic features to obtain the category supervision self-attention features; the category supervision self-attention module includes feature dimension transformation and matrix multiplication to obtain a category self-attention matrix of size C'×32, and weight the category information to the original basic features.

[0028] Please refer to Figure 5 , Figure 5 Schematic diagram of a spatial supervised self-attention module; the spatial supervised self-attention module is used to learn spatial context features for basic features, obtain spatial pixel context prior relationships, intra-class features and heterogeneous features, and perform supervision through an ideal spatial similarity matrix to obtain spatial supervised self-attention features; the spatial supervised self-attention module includes feature dimension transformation and matrix multiplication to obtain a 4096×4096 size spatial similarity matrix to construct context prior information between pixels.

[0029] The advanced feature fusion module is used to cascade fuse the channel self-attention enhancement features, the category supervision self-attention features and the space supervision self-attention features, and upsample the fused feature map to obtain the final semantic segmentation category result.

[0030] For better explanation, the detailed working process of each module is as follows:

[0031] The high-resolution remote sensing image is input into the basic feature extraction module. The basic feature module uses Dialated ResNet as the backbone network. The backbone network removes the maximum pooling layer in the last two stages of ResNet and replaces it with a hole convolution layer to keep the size of the feature map unchanged. At the same time, the basic feature module also includes a 1×1 convolution layer, a batch normalization layer, and a nonlinear activation layer to reduce the number of feature channels output by the backbone network, relatively reduce the amount of calculation of subsequent modules, and ensure the trainability of the semantic segmentation network. Finally, the size of the feature map output by the basic feature module is set to C×H×W, and the specific size of the basic feature is 256×64×64.

[0032] refer to Figure 3 The channel self-attention module transforms the dimension of the basic features. The features after the dimension transformation are of sizes C×(H×W) and (H×W)×C, respectively. Matrix multiplication is performed to obtain a channel similarity matrix of size C×C. This matrix represents the similarity between the channels of the basic features. Then, the weighted matrix is ​​used to multiply the original basic features to obtain the channel self-attention features.

[0033] refer to Figure 4 Similarly, the category-supervised self-attention module also transforms the dimension of the basic features. First, the channel is reduced through a 1×1 convolution layer, and then the features with the number of channels equal to the number of categories to be classified are obtained. The features are then upsampled and restored to the size of the original input remote sensing image. Then, they are classified through a softmax layer to obtain the classification probability of the spatial pixels. Here, the moment size of the classification probability is C'×N', where C' is the number of categories 6 and N' is 32. This application multiplies the classification probability matrix with the original features after dimension transformation to obtain the category self-attention features with classification category supervision, where the real label supervision is required when calculating the cross entropy loss after classification.

[0034] refer to Figure 5 , parallel to the above-mentioned channel self-attention module and category self-attention module is the spatial supervision self-attention module. Similarly, this application transforms the dimension of the basic features, and performs matrix multiplication on the features after the dimension transformation, with sizes of (H×W)×C and C×(H×W) respectively, to obtain a spatial similarity matrix of size N×N, where N=H×W is 4096, and obtains a spatial similarity matrix of size N×N, where N is 4096. Therefore, the spatial similarity matrix here represents the similarity of the feature pixels of the intermediate layer, and after multiplying the matrix with the basic features, the spatial self-attention feature can be obtained.

[0035] Since the above spatial similarity matrix lacks effective supervision, the real label matrix size is 512×512, and it is downsampled 8 times to obtain a label matrix size of 64×64 with a resolution that meets the characteristics of this layer. According to the characteristics that the same type is 1 and the different type is 0, an ideal spatial similarity matrix is ​​constructed. Because the spatial similarity matrix represents the relationship between each pixel in the space, its size is 4096×4096. The cross entropy loss is calculated with the actual spatial similarity matrix to ensure the learning effect of the module and obtain supervised spatial self-attention features that can better reflect the spatial category relationship.

[0036] Finally, the channel self-attention features, category self-attention features and supervised space self-attention features obtained above are concatenated and sent to the high-level feature fusion module as a whole. The module uses a 1×1 convolution layer, a batch normalization layer and a non-linear activation layer to finally obtain a high-level semantic feature of size 6×64×64. The feature is upsampled 8 times by bilinear interpolation and then the softmax function is calculated to obtain the classification probability of the corresponding category of the pixel. The maximum probability is taken as the category to which the pixel belongs as the final judgment category of this model.

[0037] S2, obtaining remote sensing images taken by a drone or satellite as a training set, and using the training set to train the remote sensing image semantic segmentation network to obtain a trained network model;

[0038] As an embodiment, specifically, the remote sensing images in the present invention generally refer to RGB images taken by drones and remote sensing satellites, and through certain corrections and image registration, the captured scenes are generally urban scenes, including buildings, roads, trees, cars and other landforms.

[0039] The input remote sensing image is a high-resolution remote sensing image with a size of 512×512×3 and is an RGB three-channel image;

[0040] As an embodiment, the network training process is as follows:

[0041] This method uses the ISPRS Potsdam high-resolution remote sensing image semantic segmentation dataset with a spatial resolution of 5cm. It is a semantic segmentation dataset with full spatial pixel labels, and the shooting scene is a city. This application selects the training data in the dataset as the training set for this method. The training set contains 32 RGB high-resolution remote sensing images with a size of 6000×6000. Due to the large size of the original image, this application cuts it into 512×512 images with a step size of 128, and obtains approximately 12,000 training images, which contain 6 categories pre-set in the dataset, namely roads, buildings, low shrubs, trees, cars and other miscellaneous categories.

[0042] In this training, the application randomly divides the high-resolution remote sensing images in the training set into training batches of equal size. In this method, each training batch consists of 8 images, and data augmentation is performed on the data images of each training batch to improve the richness of the data. Specifically, it includes calculating the mean of each channel of the image in the training set; subtracting the image mean from each image in the training batch; randomly flipping horizontally, and randomly scaling, with a scaling factor of a random one among {0.5, 0.75, 1.0, 1.5, 2.0}.

[0043] In this method, one training batch is trained each time, and the end of all training batches is calculated as the end of one iteration. Repeat the above operation until the number of iterations reaches the upper limit, and obtain the weight parameters of the final network training, so as to realize the training of the high-resolution remote sensing image semantic segmentation neural network of deep learning. At the same time, in order to speed up the training process, this application adopts the pre-training model on the ImageNet data set to initialize some weight parameters of the semantic segmentation model. In this method, the upper limit of the number of iterations is selected to be 120,000 times.

[0044] The training process of a batch is as follows: the neural network forward propagation algorithm and back propagation algorithm are used to train the above semantic segmentation network parameters. Forward propagation calculates the loss function corresponding to each training batch, and back propagation obtains the gradient corresponding to this training batch. Using softmax classification and cross entropy loss function, the total loss function of the semantic segmentation network is calculated as L = λ 1 L 1 +λ 2 L 2 +λ 3 L 3 , where L 1 is the main loss function, L 2 is the category supervision loss function, L 3 is the spatial supervised self-attention loss function; parameter λ 1 ,λ 2 ,λ 3 Used to balance the main loss function, category supervision loss function and spatial supervision self-attention loss function. The value is preset to 1:0.5:0.4. The mini-batch stochastic gradient descent algorithm is used to minimize the loss function and then update the weights. The learning rate is preset to 0.01 and the learning rate is reduced in the form of exponential decay. After updating the network training parameters, the updated network parameters are used as the initial values ​​of the next training batch.

[0045] After preprocessing, the high-resolution remote sensing image to be segmented is input into the trained high-resolution remote sensing image semantic segmentation model to obtain the ground object category to which each pixel in the high-resolution remote sensing image to be segmented belongs.

[0046] The high-resolution remote sensing image semantic segmentation algorithm proposed in the present invention can be applied to urban land use censuses, forest and water system area censuses, etc., and can also be extended to autonomous driving and video surveillance.

[0047] S3. Use the trained network model to segment the high-resolution remote sensing image to be segmented, and obtain the target object in the high-resolution remote sensing image to be segmented. As an embodiment, the target object in this application is: specific categories such as buildings, roads, and vehicles.

[0048] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0049] The beneficial effects of the present invention are:

[0050] 1. The supervised category self-attention module in the present invention can effectively utilize the semantic category information of the entire image, enhance the overall category's ability to express features, relatively effectively distinguish the categories to which some features belong, and improve the overall accuracy of remote sensing image semantic segmentation;

[0051] 2. The spatial supervised self-attention module in the present invention can effectively supervise the calculated similarity matrix based on the ideal similarity matrix constructed by the real labels, and can clearly constrain the relationship between similar and different features, and then learn the clear contextual prior of the features, which is beneficial to the reasoning of pixel category in semantic segmentation and improving the overall accuracy of semantic segmentation of remote sensing images.

[0052] 3. The present invention utilizes the self-attention mechanism and provides effective supervision information at the same time, which can increase the receptive field acquired by the network as much as possible, enhance the semantic representation of the intermediate layer module, and then obtain more advanced semantic features, thereby effectively constructing the dependency relationship of feature classification and enhancing the accuracy of semantic segmentation of remote sensing images.

[0053] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A high-resolution remote sensing image semantic segmentation method based on supervised self-attention network, characterized by: The following steps are involved: S1. Construct a remote sensing image semantic segmentation network, which includes: a basic feature extraction module, a channel self-attention module, a category supervision self-attention module, a spatial supervision self-attention module and an advanced feature fusion module; The basic feature extraction module is used to extract basic features of the input remote sensing image and output the basic features of the image. The module uses DialatedResNet as the backbone network, removes the maximum pooling downsampling layer in the last two stages, and replaces it with a dilated convolution to keep the spatial resolution of the feature map unchanged during the calculation process of the last two stages; The channel self-attention module is used to construct the relationship between channels of the extracted basic features, obtain the weights between the basic feature channels, and multiply them with the basic features to obtain the channel self-attention enhanced features; the channel self-attention module includes feature dimension transformation and matrix multiplication to obtain a channel self-attention matrix of size C×C, and weight the channel information to the original basic features; The category supervision self-attention module is used to learn category semantic features of basic features, build the relationship between categories and features, and weight them to the basic features to obtain category supervision self-attention features; the category supervision self-attention module includes feature dimension transformation and matrix multiplication to obtain a category self-attention matrix of size C'×32, and weight the category information to the original basic features; The spatial supervision self-attention module is used to learn the spatial context features of the basic features, obtain the spatial pixel context prior relationship, intra-class features and heterogeneous features, and supervise through the ideal spatial similarity matrix to obtain the spatial supervision self-attention features; the spatial supervision attention module includes feature dimension transformation and matrix multiplication to obtain a 4096×4096 size spatial similarity matrix to construct the context prior information between pixels; The advanced feature fusion module is used to cascade fuse the channel self-attention enhancement feature, the category supervision self-attention feature and the space supervision self-attention feature, and upsample the fused feature map to obtain the final semantic segmentation category result; S2, obtaining remote sensing images taken by a drone or satellite as a training set, and using the training set to train the remote sensing image semantic segmentation network to obtain a trained network model; S3. Use the trained network model to segment the high-resolution remote sensing image to obtain the target object in the high-resolution remote sensing image to be segmented.