Semantic detail fusion and context enhancement remote sensing image segmentation method based on DeepLabv3 +

Abstract

The invention discloses a semantic detail fusion and context enhancement remote sensing image segmentation method based on DeepLabv3 +. On the basis of Deeplabv3 +, a multi-scale context feature fusion module adopting a multi-scale pooling, convolution and feature fusion structure is constructed, context information in a complex scene is effectively captured, and the capabilities of detail processing and global structure understanding are improved. A global context enhanced self-attention module is introduced to improve the capture capability of the model on remote information, and the module combines a self-attention mechanism and global average pooling and introduces residual connection to balance local details and global features. In order to effectively fuse semantic information and spatial details, a semantic detail information fusion module is provided to effectively capture and fuse multi-level features. In order to verify the effectiveness of the proposed model, an experiment is carried out on an unmanned aerial vehicle aerial image data set, and the result shows that the improved model obtains 67.2% mIoU on the data set, and the performance is remarkably improved compared with a traditional Deeplabv3 + model.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image segmentation, and in particular relates to a semantic detail fusion and context enhancement remote sensing image segmentation method based on DeepLabv3+. Background Art

[0002] With the rapid advancement of remote sensing image acquisition technology, remote sensing images are increasingly used in various fields such as agriculture, building monitoring, urban planning, and marine mapping, and play a vital role. As an intensive prediction task, remote sensing semantic segmentation assigns specific category labels to each pixel and accurately identifies the boundaries between different categories by deeply analyzing the contextual semantic information of the image. This task is not only basic but also critical, but due to the wide variety of objects and complex and changeable backgrounds in remote sensing images, the challenge of semantic segmentation has been significantly increased.

[0003] In recent years, deep learning methods have gradually become the most advanced semantic segmentation technology. Convolutional neural networks used for classification are usually connected to fully connected layers after several convolutional layers, and upsampling operations are used to generate prediction maps. The deep convolutional network performs end-to-end pixel-by-pixel semantic segmentation on the input image, achieving better segmentation results than traditional methods. However, semantic segmentation methods based on convolutional neural networks still face some challenges. On the one hand, the limited receptive field makes it difficult to capture long-distance information in the image. On the other hand, downsampling or pooling operations in convolutional neural networks will lead to reduced resolution, making the edge segmentation effect unsatisfactory.

[0004] In remote sensing images, each image contains many targets, and similar targets are densely arranged, with a wide range of size variations, large differences in color and texture, and most targets are small, which brings great challenges to the semantic segmentation of remote sensing images. In order to solve the above problems, some multi-scale models have been developed for the segmentation of remote sensing images. DeepLabV3+ is an improved deep fully convolutional neural network that combines an encoder-decoder structure and an improved dilated spatial pyramid pooling module to improve the accuracy of segmentation boundaries. It has strong semantic understanding capabilities and better captures the details of objects and regions through dilated convolutions and multi-scale contextual information. On the other hand, in order to capture the long-range dependencies in the image, VisionTransformer introduces Transformer into the field of computer vision to further enhance the context modeling ability of the network. However, both the multi-scale method and the method of introducing Transformer will inevitably increase the number of network parameters, thereby increasing the computational complexity. Therefore, while pursuing higher segmentation performance, how to balance the complexity of the model with segmentation accuracy has become an important topic of current research. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a semantic detail fusion and context-enhanced remote sensing image segmentation method based on DeepLabv3+. A global context-enhanced self-attention module is designed based on DeepLabv3+, which combines the self-attention mechanism with global average pooling to extract richer information and make the network focus on important features. In order to better solve the problem of unsatisfactory edge segmentation effect, a semantic detail information fusion module is designed. A multi-scale context feature fusion module is proposed to solve the problems of complex scenes and multi-scale feature processing of remote sensing images.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A remote sensing image segmentation method based on semantic detail fusion and context enhancement based on DeepLabv3+, comprising the following steps:

[0008] (1) Improve the DeepLabv3+ network and propose a multi-scale context feature fusion module in the encoder part;

[0009] (2) The decoder uses a global context enhanced self-attention module that combines the self-attention mechanism with global average pooling;

[0010] (3) In order to effectively balance the semantic information and spatial details in remote sensing images, a semantic detail information fusion module is proposed, which performs convolution, upsampling, batch normalization and other operations in the semantic branch and the spatial detail branch respectively to better fuse the semantic information and spatial detail information;

[0011] (4) Complete the training of the entire network and save the optimal network model for network fusion to obtain more accurate segmentation results.

[0012] Furthermore, the specific method of proposing a multi-scale context feature fusion module in the encoder part in step (1) is as follows:

[0013] (1.1) The input feature map is processed by six branches (branch 0 to branch 5) to extract multi-scale feature information. Branch 0 and branch 5 directly convolve the input features to extract basic features; the remaining branches downsample the input features through pooling kernels of different sizes to obtain features of different spatial scales. Specifically, branch 1 downsamples the features to the specified size through adaptive average pooling and 1×1 convolution operations; branches 2, 3, and 4 use pooling kernels of sizes 17×17, 9×9, and 5×5 for downsampling, respectively, and then use 3×3 dilated convolutions to improve feature expression capabilities.

[0014] (1.2) Each scale feature after downsampling is optimized through processing modules such as normalization, activation and convolution, and restored to the size of the input feature through upsampling. Subsequently, these features are added to the feature map extracted by branch 5 to introduce more contextual information and detailed features. Finally, the features of different scales are integrated through splicing and subjected to feature compression operations to achieve feature fusion and dimensionality reduction, reduce redundant information, and improve the effectiveness and interpretability of features. In addition, the model uses residual connections to add the spliced ​​multi-scale features to the features generated by branch 0, which not only retains the original features but also enhances the semantic information of the integrated features. Through multi-scale processing, feature fusion and spatial attention mechanism, the model significantly improves the feature expression ability, which helps to achieve better performance in a variety of visual tasks.

[0015] Furthermore, the specific method of the global context enhanced self-attention module proposed in step (2) is as follows:

[0016] (2.1) The input feature map is first subjected to a 1×1 convolution operation to generate three feature representations: query, key, and value. The query and key features are subjected to dimensionality reduction operations to retain key semantic information while reducing computational complexity and maintaining spatial position correlation. Subsequently, the query feature and the key feature are inner-producted to generate an energy map, which is used to represent the correlation between each position in the feature map.

[0017] (2.2) The energy map is normalized by the softmax function to obtain the attention map, which reflects the importance of each position in the feature map. For remote sensing images, the attention mechanism helps to enhance the correlation recognition between different land object categories, thereby improving the segmentation accuracy. The value feature map is multiplied by the attention map to generate a weighted feature map. Through this weighted method, the network can effectively focus on key land object features while suppressing background noise.

[0018] (2.3) This mechanism also introduces a global context path. The input feature map is compressed into a fixed-size feature map through global average pooling to extract global context information. Subsequently, the global feature map is processed using 1×1 convolution and sigmoid activation function, and multiplied point by point with the input feature to complete the weighting of the global feature. This process is particularly important for capturing large-scale features of objects in remote sensing images, ensuring that the network can effectively obtain global context information while focusing on local information during segmentation.

[0019] (2.4) The generated attention-weighted feature map is added to the output feature map of the global context path, and the original input features are introduced through residual connections to generate the final output feature map. Residual connections can effectively alleviate the problem of gradient vanishing in deep networks and retain the detailed information in the original features, so that the segmentation network can take into account both local details and global information when processing remote sensing images, thereby improving the segmentation effect.

[0020] Furthermore, the specific method of the semantic detail information fusion module proposed in step (3) is as follows:

[0021] (3.1) In the feature fusion module of the semantic segmentation network, the semantic features of the semantic branch are first processed. Channel dimension reduction is achieved through a 1×1 convolution layer and a batch normalization layer, and the processed features are upsampled to generate features. Figure 1 In order to better integrate the features of the spatial detail branch, the semantic branch is further processed by a 1×1 convolution layer and a batch normalization layer, and then upsampled by a 3×3 convolution layer and a batch normalization layer to generate features. Figure 2 In the spatial detail branch, the detail features are processed by a 3×3 convolution layer and a batch normalization layer and then merged with the feature Figure 2 Multiply element by element and generate features by summing Figure 3 This process aims to organically integrate semantic information and spatial detail information so that the feature map contains both global semantic information and spatial detail features.

[0022] (3.2) The features Figure 3 It is divided into two branches: local attention branch and global attention branch. The local attention branch extracts local information through a sequence module consisting of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function; the global attention branch uses adaptive average pooling (reducing the feature map to a specified size) and then processes it through a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function to achieve weighted global features, so that the network can better capture global context information.

[0023] (3.3) Add local and global weighted features to get the feature Figure 4 , and generate attention weights through the sigmoid activation function. Then, use the attention weights and features Figure 1 Multiply, and multiply (1-attention weight) with the feature map of the spatial detail branch, and finally add the two parts of features to complete feature fusion. This operation effectively retains the global semantic information, takes into account the spatial details, enhances the network's expression ability, and thus improves the accuracy and robustness of semantic segmentation.

[0024] Furthermore, the specific method of step (4) is as follows:

[0025] (4.1) After steps (1), (2), and (3), segmentation is completed through the softmax layer. Assume that the segmentation category of the remote sensing image is C. For each pixel i{i=1,2,3,...,N}, the true category label is The sample passes through the codec to obtain the C-dimensional output feature vector: Use the softmax function to transform the feature vector The linear prediction values ​​of all categories in are converted into probability values, then the probability value of belonging to the Cth category is

[0026]

[0027] (4.2) After obtaining the probability value, the cross entropy loss function is used to calculate the 1oss value between the real data and the predicted value to quantify the gap between the two. N is the total number of pixels, and the model is trained using the stochastic gradient descent optimization algorithm;

[0028] (4.3) The mean intersection over union (mIoU) evaluation metric is used to evaluate the effectiveness of the proposed method. Where K represents the total number of categories and k is used as a normalization factor to ensure that mIoU is the average of all category IoUs. TP, FP, and FN represent true positives, false positives, and false negatives, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structural diagram of a semantic detail fusion and context enhanced remote sensing image segmentation method based on DeepLabv3+ described in the present invention;

[0030] Figure 2 This is a structural diagram of the multi-scale context feature fusion module of the present invention;

[0031] Figure 3 This is a structural diagram of the global context enhanced self-attention module described in the present invention;

[0032] Figure 4 This is a structural diagram of the semantic detail fusion module described in the present invention;

[0033] Figure 5 This is a comparison chart of the method proposed in the present invention and other advanced methods on an aerial drone image dataset. DETAILED DESCRIPTION

[0034] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention.

[0035] The present invention provides a remote sensing image segmentation method based on DeepLabv3+ with semantic detail fusion and context enhancement. Figure 1 As shown, the specific implementation is as follows:

[0036] (1) Data preprocessing stage:

[0037] (1.1) Load the original dataset. Each image is captured by a high-resolution camera with a resolution of 6000×4000 pixels, and there are 400 images in total.

[0038] (1.2) By applying data augmentation techniques such as random noise addition, horizontal flipping, vertical flipping, lighting adjustment, and blurring, the dataset was expanded to 1,200 images, and the size of each image was resized to 520×520 pixels.

[0039] (1.3) In order to train and evaluate the machine learning model, the dataset is divided into a training set and a validation set with a ratio of 80% and 20%.

[0040] (2) Network model construction stage:

[0041] (2.1) The network described in the present invention is an end-to-end neural network, which adopts an encoder-decoder structure, such as Figure 1 As shown in the figure. The encoder part uses ResNet101 as the backbone network and combines the multi-scale context feature fusion module to extract rich semantic information. The global context enhanced self-attention module captures global semantics and local details by introducing global context information and self-attention mechanism. The decoder part adopts the semantic detail fusion module to enhance the feature expression ability by combining global semantics and spatial details.

[0042] (2.2) Figure 2 As shown in the figure, a multi-scale context feature fusion module is constructed. The input feature map is fed to six different processing branches, namely branch 0, branch 1, branch 2, branch 3, branch 4 and branch 5. Specifically, branch 0 and branch 5 use 1×1 convolution operations to obtain basic feature representations; branch 1 first uses global average pooling and then downsamples the image through 1×1 convolution; branch 2, branch 3, and branch 4 use 17×17, 9×9, and 5×5 pooling kernels to downsample the feature map, and then use 3×3 dilated convolutions with dilation rates of 18, 12, and 6, respectively, to extract feature information of different scales.

[0043] (2.3) The feature maps extracted by branches 1, 2, 3, and 4 are upsampled to the original image size and added to the feature map extracted by branch 5 to retain contextual information and spatial detail information. Feature maps of different scales are concatenated, and the features are integrated and the dimension of the feature maps is reduced through compression operations, thereby reducing the redundant information of the features and improving the effectiveness and interpretability of the features. In addition, residual connections are introduced in the multi-scale context feature fusion module to add the concatenated multi-scale features to the features extracted by branch 0 to retain the original features and introduce more semantic information into the integrated features.

[0044] (2.3) Figure 3As shown in the figure, a global context enhanced self-attention module is constructed. The input feature map is converted into three feature representations of query, key and value through a 1×1 convolution operation, and then the similarity between the query feature and the key feature is calculated through the inner product, and an energy map is generated to represent the correlation of each position in the feature map. In this process, in order to reduce the computational complexity, the resolution of the query feature map and the key feature map is reduced.

[0045] (2.4) Use the softmax function for normalization to obtain the attention feature map. The value feature map is multiplied by the attention map through matrix multiplication to generate the attention weighted feature map, so that the network can automatically focus on important features of the object while suppressing unimportant background information. In addition, this module introduces a global context path, which first compresses the feature map to a fixed size through global average pooling, and then processes it through a 1×1 convolution operation and a sigmoid activation function. The obtained feature map is multiplied with the original input feature map to weight the global features of the input image. The attention feature map, the weighted feature map, and the original feature map are added together to generate the final output feature map.

[0046] (2.5) Figure 4 As shown in the figure, a semantic detail fusion module is constructed. The semantic feature map of the semantic branch is processed, and the channel dimension of the semantic feature is reduced through a 1×1 convolution operation and a batch normalization operation, and the processed feature is upsampled to generate the feature Figure 1 . Then use 3×3 convolution to process the semantic features to generate features Figure 2 The spatial detail branch feature map is processed by a 3×3 convolution layer and a batch normalization layer. Figure 2 Multiply element by element to generate features Figure 3 .

[0047] (2.6) The feature Figure 3 Further processing is performed through two branches: one is the local attention branch, which uses a sequence module of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function to capture local information; the other is the global attention branch, which includes an adaptive average pooling layer and a sequence module consisting of a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function to weight the global features to help the network better capture global contextual information.

[0048] (2.7) Add the feature map generated by the local attention branch (including local context information) to the feature map generated by the global attention branch (including global context information) to obtain the feature Figure 4 Next, the characteristics Figure 4 Processed by sigmoid activation function and respectively with features Figure 2Multiply it with the spatial detail feature map and finally add the two together to generate the final output of the network.

[0049] (3) Training phase:

[0050] (3.1) The images in the aerial drone image dataset are all collected by a high-resolution camera with a resolution of 6000×4000 pixels. The shooting angle is from the air, and the shooting height is between 5 and 30 meters, covering more than 20 houses. In order to focus on urban scenes, only a small number of images are retained, and a total of 400 images are obtained.

[0051] (3.2) In order to increase the diversity and generalization ability of the dataset, a variety of data augmentation techniques were applied, including random noise addition, horizontal flipping, vertical flipping, lighting adjustment, and blurring, expanding the original dataset to 1,200 images. The size of all images was uniformly adjusted to 520×520 pixels. The dataset was divided into a training set and a validation set with a ratio of 80% and 20%, which were used for training and evaluation of machine learning models.

[0052] (3.3) In this invention, stochastic gradient descent optimizer (SGD) is used for model training and evaluation. The initial learning rate is set to 0.014, and the standard gradient descent algorithm is used for optimization.

[0053] (3.4) In the semantic segmentation task, the mean intersection over union (mIoU) is an important indicator for measuring model performance, especially in the remote sensing image segmentation task. mIoU measures the accuracy of the model by calculating the ratio of the intersection over union of the true label and the predicted label of each category. The formula is: Where K represents the total number of categories and k is used as a normalization factor to ensure that mIoU is the average of all category IoUs. TP, FP, and FN represent true positives, false positives, and false negatives, respectively.

[0054] (3.5) After experiments, the method proposed in this paper achieved an mIoU of 67.2% in the remote sensing image semantic segmentation task. Figure 5 The comparison results between the method of the present invention and the existing advanced methods are shown. Figure 5 (a) is the original input image, Figure 5 (b) is the true label, Figure 5 (c) is the prediction result of the method of the present invention. In addition, Figure 5 The rectangular box in the figure highlights some areas with good segmentation effect. Figure 5 It can be clearly seen that the method of the present invention has a more detailed segmentation effect on the edge of the image, and is particularly outstanding in the recognition of small objects and fine features.

[0055] (3.6) The present invention significantly improves the accuracy of remote sensing image segmentation and can more accurately identify the boundaries of different types of objects, especially in the identification of small objects and fine objects. The model's adaptability to scenes with uneven distribution of objects is improved, and the layout still maintains high segmentation accuracy in complex environments, enhancing the robustness of the method in a variety of remote sensing scene images. This method is not only suitable for a variety of remote sensing tasks such as satellite images and drone images, but can also be widely used in urban land use, resource surveying, environmental monitoring, precision agriculture and other fields. At the same time, the improvement of the later module design is reasonable in terms of computational cost, while ensuring accuracy while reducing the computational burden, and has good potential for actual control deployment.

[0056] The above is a detailed description of the semantic detail fusion and context-enhanced remote sensing image segmentation method based on DeepLabv3+ provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear the corresponding legal liability.

Claims

1. A semantic detail fusion and context enhanced remote sensing image segmentation method based on DeepLabv3+, characterized in that: The steps include: Step S1: Improve the DeepLabv3+ network and propose a multi-scale context feature fusion module in the encoder part; Step S2: The decoder selects a global context enhanced self-attention module, which combines the self-attention mechanism with global average pooling; Step S3: In order to effectively balance the semantic information and spatial details in remote sensing images, a semantic detail information fusion module is proposed, which performs convolution, upsampling, and batch normalization operations in the semantic branch and the spatial detail branch respectively to better fuse the semantic information and spatial detail information; Step S4: Complete the training of the entire network and save the optimal network model for network fusion to obtain a more accurate segmentation effect.

2. The method for remote sensing image segmentation based on semantic detail fusion and context enhancement based on DeepLabv3+ according to claim 1, characterized in that: The step S1 includes: the input feature map passes through six different processing branches, namely branch 0, branch 1, branch 2, branch 3, branch 4 and branch 5. Among them, branch 0 and branch 5 use 1×1 convolution operation to obtain basic feature representation; branch 1 first uses global average pooling and then downsamples the image through 1×1 convolution; branch 2, branch 3, and branch 4 first downsample with pooling kernel sizes of 17, 9, and 5 respectively, and then use 3×3 dilated convolution with dilation rates of 18, 12, and 6 respectively to extract feature information of different scales. The feature maps extracted by branch 1, branch 2, branch 3, and branch 4 are upsampled to the original image size, and then added to the feature map extracted by branch 5; feature maps of different scales are spliced ​​together, and feature integration and dimensionality reduction are performed through compression operation; residual connection is introduced in the scale context feature fusion module, and the spliced ​​multi-scale features are added to the features extracted by branch 0.

3. The method for remote sensing image segmentation based on semantic detail fusion and context enhancement based on DeepLabv3+ according to claim 1, characterized in that: The step S2 includes: converting the input feature map into three feature representations of query, key and value through a 1×1 convolution operation, and then calculating the similarity between the query feature and the key feature through the inner product, and generating an energy map to represent the correlation of each position in the feature map; using the softmax function to perform normalization processing to obtain an attention feature map, and using matrix multiplication to multiply the value feature map with the attention map to generate an attention weighted feature map; at the same time, introducing a global context path in the mechanism, the path compresses the feature map into a 1×1 feature map by using global average pooling, and then processes the feature map through a 1×1 convolution operation and a sigmoid activation function, and multiplies the obtained feature map with the original input feature map to weight the global features of the input image; adding the attention feature map, the weighted feature map and the original feature map to generate a final output feature map.

4. The method for remote sensing image segmentation based on semantic detail fusion and context enhancement based on DeepLabv3+ according to claim 1, characterized in that: The step S3 includes: processing the semantic feature map of the semantic branch, subjecting the semantic features to a 1×1 convolution operation and a batch normalization operation, performing channel dimension reduction on the semantic features, performing an upsampling operation on the processed features, and recording the generated feature map as feature map 1; and continuing to use 3×3 convolution to process the semantic features to generate feature map 2; processing the detail feature map of the spatial detail branch through a 3×3 convolution layer and a batch normalization layer, and performing element-by-element multiplication of the generated feature map with the feature map 2 to generate feature map 3; passing the feature map 3 through two branches, one of which is a local attention branch, which consists of two 3×3 convolution layers, a batch normalization layer and a ReLU activation function. The first branch is a sequence module composed of a plurality of numbers to capture local information; the second branch is a global attention branch, which includes an adaptive average pooling layer and a sequence module composed of a 1×1 convolution layer, a batch normalization layer and a ReLU activation function, which performs weighted processing on global features so that the network can better capture global context information; the feature map containing local context feature information generated by the local attention branch is added to the feature map containing global context information generated by the global attention branch to obtain feature map 4; the feature map 4 is processed by a sigmoid function and then multiplied by the feature map 2 and the detail feature map of the spatial detail branch, and then added to obtain the final output.

5. The method for remote sensing image segmentation based on semantic detail fusion and context enhancement based on DeepLabv3+ according to claim 1, characterized in that: The step S4 includes: preparing a data set, the data set includes 1200 images, and the size of each image is adjusted to 520×520 pixels; dividing the data set into a training set and a validation set with a ratio of 80% and 20%, and training and evaluating the machine learning model; using a stochastic gradient descent (SGD) optimizer for model evaluation: the initial learning rate is set to 0.014, and a gradient descent algorithm is used; using the mean intersection over union (mIoU) as a standard metric.

Images