An FDA-DeepLab semantic segmentation algorithm based on double attention mechanism fusion

Abstract

The application designs a FDA-DeepLab semantic segmentation algorithm based on double attention mechanism fusion, mainly including the following steps: a feature extraction network with ResNet-50 structure is built according to the DeepLabv3+ model framework, and an ASPP module of spatial pyramid pooling is built after the feature extraction network; a double attention mechanism feature fusion module is designed; based on the double attention mechanism feature fusion module, a feature fusion module is designed, high-level feature maps and low-level feature maps are input into the feature fusion module, output images are obtained, and semantic segmentation results are obtained through depth separable convolution and up sampling, and thus the FDA-DeepLab model is built; the pre-trained model is used to initialize the FDA-DeepLab backbone model, the model is trained, the loss function is improved to optimize the training, and the trained FDA-DeepLab model and DeepLabv3+ model are used for image segmentation on the test set, and the performance is compared.

Description

Technical Field

[0001] This invention relates to an FDA-DeepLab semantic segmentation algorithm based on dual attention mechanism fusion, belonging to the field of image processing. Background Technology

[0002] Traditional semantic segmentation faces several challenges: The continuous downsampling operations in traditional classification CNNs lead to a continuous decrease in feature map resolution. Multi-scale detection typically involves rescaling and aggregating feature maps, which is computationally expensive. To address these issues, the DeepLab model was developed. The DeepLabv3+ model is an evolution of the DeepLab model. The DeepLabv3+ model uses DeepLabv3 as its Encoder to extract multi-scale features. A Decoder is added, forming a new method that integrates ASPP, Encoder, and Decoder, effectively improving the segmentation of object boundaries. However, several problems exist in practice:

[0003] 1. It is easy to misjudge similar objects.

[0004] 2. Small goals are easily overlooked.

[0005] 3. Large boundary segmentation error.

[0006] 4. The predicted output contains gaps. Summary of the Invention

[0007] Purpose of the Invention: To address the problems existing in the prior art, this invention provides an FDA-DeepLab semantic segmentation algorithm based on a dual attention mechanism fusion. The main objectives are to solve the problems of broken and hollow objects segmented by the original DeepLabv3+ model; to solve the problem of large segmentation error of image boundaries by the original DeepLabv3+ model; to solve the problem of easy misclassification of similar objects by the original DeepLabv3+ model; and to solve the problems of imbalanced sample classes and imbalanced sample classification difficulty that may exist in the actual training process.

[0008] Technical solution:

[0009] An FDA-DeepLab semantic segmentation algorithm based on dual attention mechanism fusion is characterized by the following steps:

[0010] Step 1: Build the feature extraction network and the Spatial Pyramid Pooling (ASPP) module according to the DeepLabv3+ model framework;

[0011] Step 2: Design a dual-attention mechanism feature fusion module;

[0012] Step 3: Design a feature fusion module based on a double attention mechanism feature fusion module;

[0013] Step 4: Perform depth separable convolution and up-sampling on the output image obtained by the feature fusion module, and complete the model building;

[0014] Step 5: Train the model, optimize the training by improving the loss function, and compare the performance of different models.

[0015] The step 1 comprises:

[0016] Step 1.1: Use a ResNet-50 convolutional neural network model to build a feature extraction network, and obtain low-level feature maps with downsampling rates of 4, 8, and 16.

[0017] Step 1.2: Build an ASPP module after the feature extraction network to obtain high-level feature maps.

[0018] The step 2 comprises:

[0019] Step 2.1: For the same double attention mechanism fusion module, the low-resolution feature map input is U LI , the feature map resolution is H'xW', the high-resolution feature map input is U HI , and the feature map resolution is HxW.

[0020] Step 2.2: Perform up-sampling operation on U LI to obtain U L′I′ , so that the resolution of U L′I′ is consistent with U HI , that is, the resolution becomes HxW. The formula is as follows:

[0021] U L'I' =f up (U LI ), U L'I' ∈HxWxC

[0022] In the formula, f up represents the up-sampling operation, which generally uses the bilinear interpolation method;

[0023] Step 2.3: Perform channel attention operation on U L′I′ to obtain U LI′ , perform spatial attention operation on U HI to obtain weight F S , multiply weight F S and U LI′ to obtain U LO′ . The formula is as follows:

[0024] U LI′ =f(WR *z)*U L′I′

[0025] F S =[f(s 1,1 ),f(s 1,2 ),…,f(s i,j ),…,f(s H,W )]

[0026]

[0027] where f() denotes a Sigmoid function, s is a mapping feature, W R is a parameter corresponding to a convolution operation, and z is a compressed feature.

[0028] Step 2.4: add U LO′ and U HI , and add a 1x1 convolution kernel for dimension reduction. The formula is as follows:

[0029] U O =c(U LO' +U HI )

[0030] where c denotes a 1x1 convolution operation.

[0031] The step 3 comprises:

[0032] Step 3.1: the low-level feature map with a down-sampling rate of 16 obtained in step 1.1 and the high-level feature map obtained in step 1.2 are input into the double attention mechanism feature fusion module designed in step 2 to obtain output feature Figure 1 .

[0033] Step 3.2: the low-level feature map with a down-sampling rate of 8 obtained in step 1.1 and the output feature Figure 1 obtained in step 3.1 are input into the double attention mechanism feature fusion module designed in step 2 to obtain output feature Figure 2 .

[0034] Step 3.3: the low-level feature map with a down-sampling rate of 4 obtained in step 1.1 and the output feature Figure 2 obtained in step 3.2 are input into the double attention mechanism feature fusion module designed in step 2 to obtain output feature Figure 3 .

[0035] The step 4 comprises:

[0036] Step 4.1: the output feature Figure 3 obtained in step 3.3 is subjected to a depth separable convolution with a 3x3 convolution kernel and 4 times up-sampling.

[0037] The step 5 comprises:

[0038] Step 5.1: Training the model. The ResNet-50 pre-trained model pre-trained on the ImageNet dataset is used to initialize the FDA-DeepLab backbone model. The batch size is set to 10, the number of iterations is set to 40000, the total downsampling factor of the base feature extraction network is set to 16, the initial learning rate is set to 0.007, the training data size is set to 513x513, and the "poly" learning rate strategy is used.

[0039] Step 5.2: Optimizing the training by improving the loss function. The focal loss function is used instead of the conventional cross-entropy loss function, and the formula is as follows:

[0040] L FL (p t )=-α t (1-p t ) γ log p t

[0041] In the formula, α is the weight parameter between classes (0-1 binary classification), (1-p t ) y is the simple / difficult sample adjustment factor, γ is the focusing parameter, and p t is the probability of the predicted result corresponding to the label. In this experiment, γ = 2 and α = 0.25.

[0042] Step 5.3: Testing the model. MIoU is used as the performance evaluation index, and MIoU has the characteristics of simplicity and strong representation, and is the most commonly used evaluation standard in the field of semantic segmentation.

[0043] Advantages:

[0044] 1. The problem of object segmentation with cracks and holes in the original DeepLabv3+ model is solved;

[0045] 2. The problem of large segmentation error of the original DeepLabv3+ model for image boundaries is solved;

[0046] 3. The problem of easy misjudgment of similar objects by the original DeepLabv3+ model is solved;

[0047] 4. The problem of data set sample class imbalance and sample classification difficulty imbalance in the actual training process is solved. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the network model diagram of the original DeepLabv3+ model;

[0049] Figure 2is a double attention mechanism fusion module structure diagram designed by the application;

[0050] Figure 3 is a general architecture diagram of the application;

[0051] Figure 4 is an attention mechanism comparison experiment result diagram;

[0052] Figure 5 is a double attention mechanism multi-feature fusion comparison experiment result diagram;

[0053] Figure 6 is a loss function comparison experiment result diagram;

[0054] Figure 7 is a DeepLabv3+ improvement before and after comparison experiment result diagram;

[0055] Figure 8 is a different algorithm comparison experiment result diagram. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application.

[0057] As shown in the figure, an FDA-DeepLab semantic segmentation algorithm based on double attention mechanism fusion includes the following steps:

[0058] Step 1: build a feature extraction network and a spatial pyramid pooling ASPP module according to a DeepLabv3+ model framework;

[0059] Step 2: design a double attention mechanism feature fusion module;

[0060] Step 3: design a feature fusion module based on the double attention mechanism feature fusion module;

[0061] Step 4: perform depth separable convolution and up-sampling on the output image obtained by the feature fusion module, and the model is built;

[0062] Step 5: train the model, optimize the training by improving the loss function, and compare the performance of different models.

[0063] The step 1 includes:

[0064] Step 1.1: A ResNet-50 convolutional neural network model is used to build a feature extraction network, and low-level feature maps with downsampling rates of 4, 8, and 16 are obtained.

[0065] Step 1.2: An ASPP module is built after the feature extraction network to obtain high-level feature maps.

[0066] The step 2 includes:

[0067] Step 2.1: For the same double attention mechanism fusion module, the low-resolution feature map input is U LI , the feature map resolution is H'xW', the high-resolution feature map input is U HI , and the feature map resolution is HxW.

[0068] Step 2.2: U LI is upsampled to obtain U L′I′ , so that the resolution of U L′I′ is consistent with U HI , i.e., the resolution becomes HxW. The formula is as follows:

[0069] U L'I' =f up (U LI ), U L'I' ∈HxWxC

[0070] In the formula, f up represents the upsample operation, which generally uses the bilinear interpolation method.

[0071] Step 2.3: Channel attention is performed on U L′I′ to obtain U LI′ , spatial attention is performed on U HI to obtain weight F S . The weight F S is multiplied by U LI′ to obtain U LO′ . The formula is as follows:

[0072] U LI′ =f(W R *z)*U L′I′

[0073] F S =[f(s 1,1 ),f(s 1,2 ),…,f(s i,j ),…,f(s H,W )]

[0074]

[0075] In the formula, f() represents a Sigmoid function, s is a mapping feature, W is a parameter corresponding to a convolution operation, and z is a compressed feature. R

[0076] Step 2.4: Add U LO′ and U HI , and add a 1x1 convolution kernel for dimension reduction. The formula is as follows:

[0077] U O = c(U LO' + U HI )

[0078] In the formula, c represents a 1x1 convolution operation.

[0079] The step 3 comprises:

[0080] Step 3.1: The low-level feature map with a down-sampling rate of 16 obtained in step 1.1 and the high-level feature map obtained in step 1.2 are input into the double attention mechanism feature fusion module designed in step 2 to obtain output feature Figure 1 .

[0081] Step 3.2: The low-level feature map with a down-sampling rate of 8 obtained in step 1.1 and the output feature Figure 1 obtained in step 3.1 are input into the double attention mechanism feature fusion module designed in step 2 to obtain output feature Figure 2 .

[0082] Step 3.3: The low-level feature map with a down-sampling rate of 4 obtained in step 1.1 and the output feature Figure 2 obtained in step 3.2 are input into the double attention mechanism feature fusion module designed in step 2 to obtain output feature Figure 3 .

[0083] The step 4 comprises:

[0084] Step 4.1: The output feature Figure 3 obtained in step 3.3 is subjected to a depth separable convolution with a convolution kernel of 3x3 and 4 times up-sampling.

[0085] The step 5 comprises:

[0086] ​Step 5.1: training the model. The built FDA-DeepLab model is trained using the public dataset PASCAL VOC 2012 dataset. The ResNet-50 pre-training model pre-trained on the ImageNet dataset is used to initialize the FDA-DeepLab backbone model. The batch size is set to 10, the iteration step is set to 40000, the total down-sampling multiple of the basic feature extraction network is set to 16, the initial learning rate is set to 0.007, the training data size is set to 513*513, and the 'poly' learning rate strategy is adopted.

[0087] Step 5.2: improve the loss function to optimize the training. The focal loss function is used instead of the conventional cross-entropy loss function, and the formula is as follows:

[0088] L FL (p t )=-α t (1-p t ) γ log p t

[0089] In the formula, alpha is the weight parameter between classes (0-1 binary classification), (1-p t ) y is a simple / difficult sample adjustment factor, gamma is a focusing parameter, and p t is the probability of the predicted result corresponding to the label. In this experiment, gamma is set to 2 and alpha is set to 0.25.

[0090] Step 5.3: test the model. MIoU is used as the performance evaluation index, and MIoU index has the characteristics of simplicity and strong representation, and is the most commonly used evaluation standard in the field of semantic segmentation.

[0091] The present application solves the shortcomings of the original DeepLabv3+ model in image semantic segmentation by adding a double attention mechanism feature fusion module, a feature fusion module and improving the loss function to the DeepLabv3+ model, and improves the image segmentation effect.

[0092] wherein, Figure 1 is the network model diagram of the original DeepLabv3+ model. The original DeepLabv3+ model is divided into an Encoder and a Decoder module. Specifically, the Encoder module includes a backbone network responsible for basic feature extraction and an ASPP module, and effectively extracting image features is the key to high-precision semantic segmentation. The Decoder module is responsible for gradually up-sampling the feature maps obtained by the Encoder module, and adopts the FPN feature fusion idea to fuse high and low features, solves the problem of detail loss in feature extraction, and finally obtains the semantic segmentation result.

[0093] Figure 2 is the structure diagram of the dual attention mechanism fusion module designed by the application. The application effectively fuses low-level spatial details and high-level semantic clues by combining the advantages of two attention mechanisms, and obtains an attention mechanism model with better effect. The commonly used fusion method at present is to perform two attention mechanism operations respectively on the same feature map and fuse the results, and the difference is more in different feature fusion methods. The low-level feature map with high resolution is suitable for spatial attention operation to extract the spatial position information of the input image and locate the important parts; the high-level feature map with low resolution is suitable for channel attention operation to pay more attention to the relevant feature channels and ignore other interference information. Therefore, the application adopts different attention mechanisms on feature maps with different resolutions, and then fuses them, thereby improving the fusion effect.

[0094] Figure 3 is the overall architecture of the application.

[0095] Figure 4 、 5 , 6, 7 and 8 are respectively attention mechanism comparison experiment results, dual attention mechanism multi-feature fusion comparison experiment results, loss function comparison experiment results, DeepLabv3+ comparison experiment results before and after improvement and different algorithm comparison experiment results carried out on PASCAL VOC 2012 verification set. From the experiment results, it can be seen that the dual attention mechanism module designed by the application is better than the channel attention mechanism or the spatial attention mechanism alone, and is better than the original model; the feature fusion method based on the dual attention mechanism designed by the application is better than other fusion methods; the focal loss function designed by the application also has certain improvement for the public data set with relatively balanced data distribution. In general, the application designs a dual attention mechanism fusion based FDA-DeepLab semantic segmentation algorithm by designing a dual attention mechanism feature fusion module, a feature fusion module and an improved loss function on the DeepLabv3+ model, improves the shortcomings of the original DeepLabv3+ model and improves the segmentation effect.

[0096] The technical means disclosed in the application scheme is not limited to the technical means disclosed in the above embodiments, but also includes technical solutions composed of any combination of the above technical features. It should be noted that, for ordinary skilled persons in the art, without departing from the principles of the application, a number of improvements and refinements can be made, and these improvements and refinements are also considered within the protection scope of the application.

Claims

1. An FDA-DeepLab semantic segmentation algorithm based on dual attention mechanism fusion, characterized in that, Includes the following steps: Step 1: Build the feature extraction network and the Spatial Pyramid Pooling (ASPP) module according to the DeepLabv3+ model framework; Step 2: Design a dual-attention mechanism feature fusion module; Step 3: Design a feature fusion module based on the dual attention mechanism; Step 4: Perform depthwise separable convolution and upsampling on the output image obtained from the feature fusion module; the model is now complete. Step 5: Train the model, optimize the training by improving the loss function, and compare the performance of different models; Step 1 includes the following steps: Step 1.1: Using the ResNet-50 convolutional neural network model, a feature extraction network is built to obtain low-level feature maps with downsampling rates of 4, 8, and 16; Step 1.2: After the feature extraction network, build the Spatial Pyramid Pooling (ASPP) module to obtain the high-level feature map; Step 2 includes the following steps: Step 2.1: Let U be the input of the low-resolution high-level feature map for the same dual-attention mechanism fusion module. LI The feature map resolution is H′×W′, and the input of the high-resolution low-level feature map is U. HI The feature map resolution is H×W; Step 2.2: For U LI Perform an upsampling operation to obtain U L′I′ , making U L′I′ Resolution and U HI Consistent resolution, meaning the resolution becomes H×W; the formula is as follows: IN L'I' =f up (IN LI ),IN L'I' ∈H×W×C In the formula, f up This indicates an upsampling operation, which typically uses bilinear interpolation. Step 2.3: For U L′I′ Perform channel attention operation to obtain U LI′ , for U HI Perform spatial attention operation to obtain weight F S ; Set weight F S with U LI′ Multiply to get U LO′ The formula is as follows: U LI′ =f(W R *z)*U L′I′ F S =[f(s 1,1 ),f(s 1,2 ),…,f(s i,j ),…,f(s H,W )] In the formula, f() represents the Sigmoid function, s is the mapping characteristic, and W R Here are the parameters for the corresponding convolution operation, and z represents the compressed feature; Step 2.4: Put U LO′ and U HI The components are added together, and then a 1×1 convolution kernel is added at the end for dimensionality reduction; the formula is as follows: IN O =c(U LO' +U HI ) In the formula, c represents a 1×1 convolution operation.

2. The FDA-DeepLab semantic segmentation algorithm based on dual attention mechanism fusion according to claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Combine the low-level feature map with a downsampling rate of 16 obtained in Step 1.1 and the high-level feature map obtained in Step 1.2 with the dual attention mechanism feature fusion module designed in Step 2 to obtain the output feature map 1; Step 3.2: Combine the low-level feature map with a downsampling rate of 8 obtained in Step 1.1 and the output feature map 1 obtained in Step 3.1 with the dual attention mechanism feature fusion module designed in Step 2 to obtain the output feature map 2; Step 3.3: Combine the low-level feature map with a downsampling rate of 4 obtained in Step 1.1 and the output feature map 2 obtained in Step 3.2 with the dual attention mechanism feature fusion module designed in Step 2 to obtain the output feature map 3.

3. The FDA-DeepLab semantic segmentation algorithm based on dual attention mechanism fusion according to claim 2, characterized in that, Step 4 includes the following steps: Step 4.1: Perform a depthwise separable convolution with a kernel of 3×3 and upsampling by 4 times on the output feature map 3 obtained in step 3.

3.

4. The FDA-DeepLab semantic segmentation algorithm based on dual attention mechanism fusion according to claim 1, characterized in that, Step 5 includes the following steps: Step 5.1: Train the model; Initialize the FDA-DeepLab backbone model using a ResNet-50 pre-trained model pre-trained on the ImageNet dataset; Set the batch size to 10, the number of iterations to 40,000, the total downsampling factor of the basic feature extraction network to 16, the initial learning rate to 0.007, the training data size to 513×513, and adopt the "poly" learning rate strategy. Step 5.2: Improve the loss function to optimize training; replace the conventional cross-entropy loss function with the focus loss function, as shown in the following formula: THE FL (p t )=-α t (1-p t ) γ logp t In the formula, α t For the weight parameters between categories, (1-p) t ) y The simple / difficult sample adjustment factor, γ is the focusing parameter, and p t To predict the probability of the label corresponding to the prediction result; set γ = 2, α t =0.25; Step 5.3: Test the model; use MIoU as the performance evaluation metric.

Images