An image segmentation method based on residual feature optimization and attention mechanism
This image segmentation method, which utilizes residual feature optimization and attention mechanisms, addresses the problem of insufficient extraction of detailed features in image segmentation, achieving more efficient feature extraction and robustness, and improving the accuracy and adaptability of image segmentation.
Patent Information
- Application Number
- CN202210792289.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-05
- Publication Date
- 2026-03-03
- Estimated Expiration
- 2042-07-05
AI Technical Summary
Existing technologies in image segmentation suffer from problems such as insufficient extraction of detailed features, limited receptive field, information loss due to downsampling, and blurred prediction, especially with performance degradation on small objects.
We employ an image segmentation method based on residual feature optimization and attention mechanism. By constructing a network framework, we use an encoder, attention mechanism and decoder, combine ASPP and residual feature maps, add global and spatial attention mechanisms, and optimize detail learning by utilizing Binary cross entropy and Dice Loss.
It improves the accuracy and robustness of image segmentation, can extract features more accurately, integrate information of different scales and details, adapt to complex image targets, and improve the performance of the model.
Smart Images
Figure CN115311454B_ABST
Abstract
Description
[0001] This invention relates to the field of computer vision technology, and in particular to an image segmentation method based on residual feature optimization and attention mechanism. Background Technology
[0002] Semantic segmentation is a challenging task that aims to assign dense labels to all pixels in an image—a process that moves from concrete to abstract. It has potential applications in fields such as autonomous driving and remote sensing. For most of these applications, maintaining efficient inference speed and impressive performance is a challenge, especially on resource-constrained mobile devices. Furthermore, semantic segmentation is now arguably a pixel-level task, as each image exhibits significant variations due to numerous factors such as small objects, background, and noise.
[0003] Previous work has modified classification networks by replacing them with fully convolutional neural networks (CNNs). This involves converting fully connected layers into convolutional layers and upsampling via deconvolution; fine-tuning using transfer learning; and employing skip connections to combine semantic and representational information, resulting in accurate and fine-grained segmentation. End-to-end, pixel-to-pixel CNNs surpass previous state-of-the-art techniques, but as network depth increases, the receptive field grows slowly, and this limited receptive field cannot fully simulate the global relationships between pixels in an image. Furthermore, downsampling leads to blurred predictions because fine details disappear at lower resolutions compared to the original image. Therefore, segmentation predictions are often blocky, with unsatisfactory boundary details, resulting in a significant performance degradation, especially for small objects. Summary of the Invention
[0004] To address the shortcomings of the existing technologies, this invention provides an image segmentation method model based on residual feature optimization and attention mechanism. Its purpose is to solve the problem of insufficient extraction of detailed features and to extract more robust and discriminative features with internal correlation, including the following steps;
[0005] S1. Construct the network framework, which consists of an encoder, an attention mechanism, and a decoder. Input the given image into the backbone network to extract features. First, we reduce the size of the input image by 16 times through the encoder.
[0006] S2. During the feature extraction process, information loses the interaction of global spatial channels. Therefore, we adopted a global attention mechanism that preserves information to amplify the cross-dimensional interaction of global information.
[0007] S3. Features extracted from the backbone network are processed through ASPP. Its core idea is to aggregate receptive fields of different scales. ASPP was proposed to solve the problem of different scales for different segmentation targets. It consists of a 1×1 convolution kernel and three 3×3 dilated convolutions with dilation rates of 3, 6 and 12 respectively.
[0008] S4. Use a 1×1 convolution to reduce the number of channels, followed by a BN, ReLU activation function and Dropout; use bilinear interpolation to upsample the alignment by 4 times.
[0009] S5. Subtracting the image that has been downsampled and then upsampled from the shallow feature map extracted from the backbone network, the resulting Up(Down(Gi)) differs from Gi because the information lost during downsampling cannot be fully recovered through upsampling. In other words, downsampling is irreversible. The result is a residual feature map, obtained by subtracting the estimated value from the original pixel value, used to store the difference between the downsampled image and the original image. Then, the residual feature map is added to the shallow feature map to obtain the final feature map.
[0010] S6. Add the residual feature map to the shallow feature map to obtain the feature map part. Insert a detail head to generate binary classification detail labels. Then use the binary classification detail labels as a guide for the detail feature map to guide the lower layer to learn spatial detail features. Finally, pass 1×1 convolution to reduce the number of channels in the feature map part, and then connect a BN and ReLU activation function.
[0011] S7, concatenate the feature maps of S4 and S6; perform 3×3 convolution, BN, ReLU, Dropout; upsample by 4 times to obtain the final result;
[0012] S8 uses Binary cross entropy and Dice Loss, Focal Loss, and CE Loss to jointly optimize detailed learning.
[0013] This invention provides an image segmentation method model based on residual feature optimization and attention mechanism. Compared with existing technologies, it has the following advantages:
[0014] First, considering the problem of insufficient detailed feature extraction, more accurate and flexible image extraction was performed, improving the robustness of the model. To address the issue that the features extracted from the backbone network did not consider the relationships between and within features, spatial and channel attention mechanisms were added to extract the relationships between features. The source images output from the backbone network were decomposed into different spatial channels, and processing was performed separately on each channel. This allows for the targeting of features and details on different channels of different decomposition layers, highlighting features and details on specific channels, potentially fusing features and details from different images. A loss function combining Binary Cross Entropy and Dice loss was used to guide shallow information to encode spatial information, and feedback iterative optimization of the model's loss was performed to minimize the final loss, thereby improving the accuracy and robustness of the features. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is the overall network structure diagram of an image segmentation method model based on residual feature optimization and attention mechanism in this invention patent.
[0017] Figure 2 This is a structural diagram of the spatial and channel attention mechanism in this invention patent.
[0018] Figure 3 This is a structural diagram of the detailed module added in this invention patent. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. A model for providing an image segmentation method based on residual feature optimization and attention mechanism includes steps S1 to S8:
[0021] S1. Construct the network framework, which consists of an encoder, an attention mechanism, and a decoder. Input the given image into the backbone network to extract features. First, we reduce the size of the input image by 16 times through the encoder.
[0022] S2. During the feature extraction process, information loses the interaction of global spatial channels. Therefore, we adopted a global attention mechanism that preserves information to amplify the cross-dimensional interaction of global information.
[0023] S3. Features extracted from the backbone network are processed through ASPP. Its core idea is to aggregate receptive fields of different scales. ASPP was proposed to solve the problem of different scales for different segmentation targets. It consists of one 1*1 convolutional kernel and three 3*3 dilated convolutions with dilation rates of 3, 6 and 12 respectively.
[0024] S4. Use a 1*1 convolution to reduce the number of channels, followed by a BN, ReLU activation function and Dropout; use bilinear interpolation to upsample the alignment by 4 times.
[0025] S5. Subtracting the image that has been downsampled and then upsampled from the shallow feature map extracted from the backbone network, the resulting Up(Down(Gi)) differs from Gi because the information lost during downsampling cannot be fully recovered through upsampling. In other words, downsampling is irreversible. The result is the difference between the original pixel value and the estimated value, used to store the difference between the downsampled image and the original image. Then, the residual feature map is added to the shallow feature map to obtain the final feature map.
[0026] S6. Add the residual feature map to the shallow feature map to obtain the feature map part. Insert a detail head to generate binary classification detail labels. Then use the binary classification detail labels as a guide for the detail feature map to guide the lower layer to learn spatial detail features. Finally, in the feature map part, pass 1*1 convolution to reduce the number of channels, and then pass a BN and ReLU activation function.
[0027] S7, concatenate the feature maps of S4 and S6; perform 3*3 convolution, BN, ReLU, Dropout; upsample by 4 times to obtain the final result;
[0028] S8 uses Binary cross entropy and Dice Loss, Focal Loss, and CE Loss to jointly optimize detailed learning.
[0029] The following is a detailed description of each step.
[0030] In step S1, the network architecture is constructed, such as... Figure 1As shown, a network framework is constructed, consisting of an encoder, an attention mechanism, and a decoder. The specific steps are as follows:
[0031] The backbone network used in the feature extraction part is MobileNetV2. MobileNetV2 has appeared in many classic network architectures and has been widely recognized and proven. Since we need to construct a lightweight network, we use MobileNetV2 to extract features from images.
[0032] In step S2, during feature extraction, these features lose their interaction across global spatial channels. Therefore, we employ a global attention mechanism that preserves information to amplify cross-dimensional interactions of global information. Thus, we name this method the global attention mechanism, as follows: Figure 2 As shown.
[0033] S201, the channel attention submodule uses a 3D arrangement to preserve three-dimensional information, and then uses two layers of MLP (multilayer perceptron) to amplify the spatial dependence of cross-dimensional channels. MLP is a codec structure with a compression ratio of r.
[0034] S202. In the spatial attention submodule, to focus on spatial information, we use two convolutional layers for spatial information fusion. We also use the same reduction rate r as the channel attention submodule in BAM. Meanwhile, max pooling reduces the amount of information and has a negative impact. We remove the pooling layers to further preserve feature maps. Therefore, the spatial attention module can sometimes significantly increase the number of parameters.
[0035] S203, Input feature F1∈R C×H×W Mc and Ms are the channel and spatial attention diagrams, respectively; The formulas for F2 and the output feature F3, expressed by element-wise multiplication, are as follows:
[0036]
[0037]
[0038] In step S3, the obtained features are processed by ASPP. Its core idea is to concentrate the receptive field at multiple scales. ASPP was proposed to solve the problem of different scales for different segmentation targets. The specific steps are as follows:
[0039] The core idea of S301 and ASPP is to concentrate the receptive field of multiple scales. ASPP consists of a 1*1 convolution and three 3*3 dilated convolutions with dilation rates of 3, 6 and 12 respectively.
[0040] S301. Add the feature map obtained by 1*1 convolution and the feature map obtained by three 3*3 dilated convolutions with dilation rates of 3, 6 and 12 respectively. The final feature map is the output map of ASPP.
[0041] In step S4, a 1*1 convolution is used to reduce the number of channels, followed by a batch normalization (BN), ReLU activation function, and Dropout; bilinear interpolation is used to upsample the alignment by a factor of 4; thus, high-level semantic features are obtained.
[0042] In step S5, we first extract features from the shallow features at the 4x scaling point using the following operation: Figure 3 As shown, the number of channels is then reduced sequentially through 1*1 convolution, Batch Normalization (BN), and ReLU; the specific steps are as follows:
[0043] S501. Subtracting the image that has been downsampled and then upsampled from the shallow feature map results in Up(Down(Gi)) which differs from Gi. This is because the information lost during downsampling cannot be fully recovered through upsampling; in other words, downsampling is irreversible. The result is the difference between the original pixel value and the estimated value, used to store the difference between the downsampled image and the original image. This is called the residual feature map, and the formula is as follows:
[0044] Ri = Gi - Up(Down(Gi)) (3)
[0045] S502. Add the residual feature map to the shallow feature map to obtain the feature map;
[0046] In step S6, compared to the shallow features of the backbone, spatial paths can encode more spatial details, such as boundaries and corners. Based on this observation, we use a detail-guided module to guide the lower layers to learn spatial information in a single-stream manner, and we model detail prediction. We insert a detail head, including a 3×3 Conv, BN, and ReLU, followed by a 1×1 convolution to obtain the output dimension N, which is set to the number of classes. The specific steps are as follows:
[0047] S601. We first generate binary detail labels from the segmented semantic ground truth labels using a Laplacian convolution kernel, as shown in the figure. We insert detail headers into the shallow feature layer to generate binary detail labels, and then use these binary detail labels as guides for the detail feature map, guiding the lower layer to learn spatial detail features. Compared to the low-level feature results, the binary detail label map with detail guidance can encode more spatial details.
[0048] S602, we generate binary detail background labels from semantically segmented background labels using the detail aggregation module. This operation can be implemented using a 2D Laplacian convolution kernel and a 1×1 convolution. We use... Figure 1 The Laplacian convolution kernel shown generates detail feature maps with different strides to obtain multi-scale detail information. We then upsample the detail feature maps to their original size and fuse them with a trainable 1×1 convolution for dynamic redrawing.
[0049] S603. Finally, we use a threshold of 0.1 to convert the predicted details into final binary detail labels with boundary and corner information. Since the number of detail pixels is much smaller than that of non-detail pixels, detail prediction is a classic balance problem. Because weighted cross-entropy always leads to coarse results, we use Binary cross-entropy and Dice loss to jointly optimize detail learning. Dice loss measures the overlap between the predicted map and the ground truth label. Furthermore, it is insensitive to the number of foreground / background pixels, meaning it can alleviate the class imbalance problem.
[0050] Therefore, for a predicted detail map with height H and width W, the detail loss L detail The formula is as follows:
[0051] L detail (p d g d ) = L dice (p d g d )+L bce (p d g d (5)
[0052] Where p d ∈R H×W Indicates prediction details and g d ∈R H×W L represents the corresponding real label. bce L represents the binary cross-entropy loss. dice The Dice loss is represented as follows:
[0053]
[0054] Where i represents the i-th pixel, and ∈ is the Laplacian smoothing term. We set ∈ = 1 to estimate the probability of a phenomenon that has not occurred before. For example... Figure 1As shown, we use a detail head to generate detail feature maps, guiding shallow information to encode spatial information. The detail head includes a 3×3 Conv, BN, and ReLU, followed by a 1×1 convolution to obtain the output details. The detail head effectively enhances the feature representation. Finally, the learned detail features are fused with the contextual features of the deep blocks in the decoder for segmentation prediction. However, this branch is discarded during the inference stage. Therefore, this side information can easily improve the accuracy of the segmentation task without any inference cost.
[0055] The shallow features and residual features are then added together to obtain the detail feature map, which is then passed through 1*1 convolutions to reduce the number of channels, followed by BN and ReLU.
[0056] Ri = Cov(Concat(Gi,Ri)) (4)
[0057] In step S7, the Feature Maps from S4 and S6 are concatenated; after 3*3 convolution, BN, ReLU, and Dropout, the final result is obtained by upsampling by 4 times.
[0058] In step S8, we use Binary cross entropy, Dice loss, and Focal loss to jointly optimize the learning of details. This is because the training process of the network model is a process of continuously optimizing the loss. The currently obtained loss is fed back into the network model for continuous iterative optimization to reduce the loss and thus obtain more robust features.
[0059] This invention, based on DeepLab, improves the accuracy and robustness of the model by refining the feature extraction method. Firstly, considering the problem of insufficient detail feature extraction, where information lost during downsampling cannot be fully recovered through upsampling, this patent uses the difference between the original pixel value and the estimated value to store the difference between the downsampled image and the original image. This allows for more accurate and flexible image extraction, improving the model's robustness. Secondly, addressing the issue that the features extracted by the backbone network do not consider the relationships between or within features, spatial and channel attention mechanisms are added to extract the relationships between features. Thirdly, the source images output from the backbone network are decomposed into different spatial frequency bands, and different features are applied to each band. This allows for the targeting of features and details in different frequency bands at different decomposition layers, highlighting features and details in specific frequency bands, potentially fusing features and details from different images. Finally, a loss mechanism combining Binary Cross Entropy and Dice loss guides shallow information to encode spatial information, performing feedback iterative optimization of the model's loss to minimize the final loss, thereby improving feature accuracy and robustness.
[0060] This method makes full use of the multi-scale feature maps extracted by the encoder, combining detailed features and global features. It can adapt to the complex and varied characteristics of segmented image targets, and constructs a new and effective method for semantic segmentation, providing a more efficient framework for semantic segmentation in practical applications.
[0061] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.
[0062] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. An image segmentation method based on residual feature optimization and attention mechanism, characterized in that, Including the following methods: Acquire high-resolution images and annotate the images to obtain training, validation, and test samples; An image segmentation network based on residual feature optimization and attention mechanism is constructed. The segmentation network includes a backbone network, an image semantic encoder based on dual attention mechanism, and a decoder based on feature residual module. The training samples are input into the backbone network MobileNetV2; the feature maps output by the backbone network are extracted and processed sequentially through the dual attention mechanism module and the ASPP module; Subtracting the image that has been scaled down and then scaled up from the shallow feature map extracted from the backbone network yields a residual feature map obtained by subtracting the estimated value from the original pixel value. This residual feature map is used to store the difference between the downsampled image and the original image. Then, the residual feature map is added to the shallow feature map to obtain the detail feature map. The original image is processed by a 3×3 convolution, a batch normalization (BN) layer, and a ReLU activation layer. The output is a 1×1 convolutional layer with the number of channels equal to the number of classes N, which yields the detail header. The obtained feature map is then processed by a Laplacian operator to generate binary detail labels after inserting a detail head. These binary detail labels are then used as a guide for the underlying learning of spatial detail features. Finally, the feature map is processed sequentially with 1×1 convolutions to reduce the number of channels. The feature maps and detail feature maps obtained by the concatenated encoder are subjected to 3×3 convolution and then upsampled by 4 times to obtain the final result. During training, Binary cross entropy and Dice Loss, Focal Loss and CE Loss are used to jointly optimize detail learning. The network is trained to obtain a trained image semantic segmentation algorithm model based on feature residual modules and dual attention mechanism; the image to be tested is obtained and input into the trained segmentation model to obtain the prediction result of the image.
2. The image segmentation method based on residual feature optimization and attention mechanism according to claim 1, characterized in that, The acquired image samples are a dataset of drone aerial images, including: Aerial image samples collected using drones; The collected aerial image samples are labeled using image annotation tools. The result of semantic segmentation is to turn the image into color blocks with certain semantic information. Semantic segmentation technology can identify the semantic category of each color block and label each pixel with its corresponding label. A sample dataset is constructed using labeled aerial images. The sample dataset is divided into a training dataset, a validation dataset, and a test dataset. The training dataset is then preprocessed.
3. The image segmentation method based on residual feature optimization and attention mechanism according to claim 1, characterized in that, The backbone network used in the feature extraction part is MobileNetV2.
4. The image segmentation method based on residual feature optimization and attention mechanism according to claim 1, characterized in that, After extracting features from the backbone network, a global attention mechanism module was inserted: The channel attention submodule uses a 3D arrangement to preserve three-dimensional information, and then uses two MLP (Multilayer Perceptron) layers to amplify the spatial dependencies of cross-dimensional channels. MLP is a codec structure with a compression ratio of r. In the spatial attention submodule, two convolutional layers are used to fuse spatial information.
5. The image segmentation method based on residual feature optimization and attention mechanism according to claim 1, characterized in that, In the encoder stage, that is, after the shallow feature maps extracted by the backbone network, a detail feature extraction module is added: First, extract the shallow feature map and subtract it from the image that was downsampled and then upsampled to obtain a residual feature map that stores the difference between the downsampled image and the original image. This residual feature map is then added to the shallow feature map to obtain the final feature map.
6. The image segmentation method based on residual feature optimization and attention mechanism according to claim 1, characterized in that, In this training process, Binary cross entropy, Diceloss, and Focalloss are used to jointly optimize the learning of details. The training process of the network model is a process of continuously optimizing the loss, feeding the currently obtained loss back into the network model for continuous iterative optimization.
7. The image segmentation method based on residual feature optimization and attention mechanism according to claim 1, characterized in that, The method includes: Acquire high-resolution images and annotate the images to obtain training, validation, and test samples; An image semantic segmentation network based on feature residual modules and a dual attention mechanism is constructed. The segmentation network includes a backbone network, an image semantic encoder based on a dual attention mechanism, and a decoder based on feature residual modules. The training samples are input into the backbone network MobileNetV2; the feature maps output by the backbone network are extracted and processed sequentially through the dual attention mechanism module and the ASPP module; Subtracting the image that has been scaled down and then scaled up from the shallow feature map extracted from the backbone network yields the difference between the original pixel value and the estimated value, known as the Residual feature. This Residual feature stores the difference between the downsampled image and the original image. Then, the Residual feature is added to the shallow feature map to obtain the feature map. The original image is processed by a 3×3 convolution, a batch normalization (BN) layer, and a ReLU activation layer. The output is a 1×1 convolutional layer with the number of channels equal to the number of classes N, which yields the detail header. The obtained feature map is inserted with a detail head and then processed by the Laplacian operator to generate binary detail labels. These binary detail labels are then used as a guide for the detail feature map to generate the underlying learning space detail features. During training, Binary cross entropy, Dice Loss, Focal Loss, and CE Loss are used to jointly optimize the learning of details. The network is trained to obtain a trained image semantic segmentation algorithm model based on feature residual modules and dual attention mechanism; the image to be tested is obtained and input into the trained segmentation model to obtain the prediction result of the image to be tested.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
9. A computer-readable storage device having a computer program stored thereon, characterized in that... When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.