Instance segmentation method for multi-scale extrusion excitation self-attention and mask fractional graph loss

Abstract

The invention provides an instance segmentation method based on multi-scale extrusion excitation self-attention and mask fractional graph loss so as to improve segmentation precision and robustness. Firstly, a Vovnet backbone network and a multi-layer pyramid network (FPN) are used for carrying out feature extraction on an image, and a preliminary feature map is generated; then, deep feature extraction is carried out through a multi-scale extrusion excitation self-attention module, and the module combines an extrusion excitation function and a self-attention mechanism, so that the extraction capability of target feature information is effectively enhanced, and interference information is suppressed; secondly, classification, centrality and frame regression calculation are carried out in the feature map through FCOS BOX HEAD; and finally, designing a mask branch to accurately predict the target instance, including introducing an RA module and mask fractional graph loss for aggregating features and improving mask prediction precision. According to the method, the defects of the existing instance segmentation technology in the aspects of target feature extraction and interference information elimination are overcome, and accurate segmentation and mask prediction of the target instance are realized.

Description

Technical Field

[0001] The invention belongs to the technical field of image segmentation of machine vision, and relates to an instance segmentation method of multi-scale squeezing-inspired self-attention and mask score map loss. Background Art

[0002] Machine vision is an important research direction in the era of artificial intelligence. Image segmentation is an important research field of machine vision. At present, image segmentation technology mainly includes three categories: semantic segmentation, instance segmentation and panoptic segmentation. Instance segmentation Instance segmentation is to determine the target in the image and then perform semantic segmentation. It is a way to effectively classify instances of the same category and achieve effective recognition and classification of instances by marking semantic information. Image instance segmentation methods play an irreplaceable role in many research fields. In the medical field, the shape-aware adversarial instance segmentation framework for gland segmentation; in the field of remote sensing images, instance segmentation models can accurately process remote sensing images; in industry and life, various defects can still be detected using instance segmentation networks; there are also instance segmentation applications such as face recognition, background intelligent replacement and video intelligent barrage in daily life. Therefore, practical applications require the study of efficient instance segmentation methods for multiple scenarios.

[0003] Existing instance segmentation research has achieved a lot of results [1][2] ,Existing instance segmentation methods are mainly divided into traditional instance segmentation methods and instance segmentation methods based on deep learning [3][4] Although traditional instance segmentation methods have the advantages of strong controllability, fewer parameters and less resource consumption, they cannot meet actual needs in terms of accuracy and efficiency. The instance segmentation method based on deep learning is the mainstream method. It relies on deep neural networks to extract image features and guarantees segmentation performance by designing different types of model structures, which has achieved success. Existing instance segmentation methods based on deep learning [5][6] It mainly includes single-stage and two-stage methods. The single-stage method is mainly used for target detection and semantic segmentation, while the two-stage method focuses on using anchor boxes to achieve segmentation. The main challenges faced by existing instance segmentation methods based on deep learning include: (1) the need to deal with occlusion problems between objects; (2) the need to solve the problem of imprecise edge segmentation; (3) the need to take into account the real-time and high-precision of the segmentation method; (4) the need to filter interference information such as lighting and background. Existing instance segmentation methods based on deep learning cannot extract target features well. At the same time, when segmenting target instances from predicted target features, they cannot accurately predict target instance masks. The instance segmentation method of multi-scale squeeze-inspired self-attention and mask score map loss involved in the present invention will better solve these problems.

[0004] [1]Kaiming He,Georgia Gkioxari,Piotr Dollar,Ross Girshick.Mask r-cnn.In Proceedings of the IEEE International Conference on Computer Vision(ICCV),pp.2961-2969,2017.

[0005] [2]Jifeng Dai,Kaiming He,Yi Li,Shaoqing Ren,Jian Sun.Instance-sensitive fully convolutional for Semantic Segmentation.European conferenceon computer vision.Springer,Cham,pp.534-549,2016.

[0006] [3]Yi Li,Haozhi Qi,Jifeng Dai,Xiangyang Ji,Yichen Wei.Fullyconvolutional instance-aware semantic segmentation.In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition(CVPR),pp.2359-2367,2017.

[0007] [4]Daniel Bolya,Chong Zhou,Fanyi Xiao,Yong Jae Lee.Yolact:Real-timeinstance segmentation.In Proceedings of the IEEE / CVF International Conferenceon Computer Vision(ICCV),pp.9157-9166,2019.

[0008] [5]Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee. YOLACT++: BetterReal-Time Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44, pp.1108-1121, 2022.

[0009] [6] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, Xinggang Wang. Mask scoring r-cnn. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6409-6418, 2019. Summary of the invention

[0010] The purpose of the present invention is to provide an instance segmentation method with multi-scale squeezing-inspired self-attention and mask score map loss, which solves the problems that the existing instance segmentation method based on deep learning cannot extract target feature information well, cannot effectively discard interference information, and cannot accurately predict the target instance mask when segmenting the target instance from the predicted target features. The invention greatly improves the accuracy and robustness of the instance segmentation method and completes the accurate segmentation of the target instance.

[0011] The technical solution adopted by the present invention is

[0012] The instance segmentation method of multi-scale squeeze excitation self-attention and mask score map loss first performs image preprocessing and preliminary feature extraction; the method preprocesses the image, and uses the Vovnet backbone network and the multi-layer pyramid network (FPN) to extract features from the image to obtain preliminary feature information. Then a multi-scale squeeze excitation self-attention module is designed for deep feature extraction. On the basis of the multi-layer pyramid network, the method designs a multi-scale squeeze excitation function self-attention module, which constructs a multi-scale self-attention module, and designs a squeeze excitation function to improve the feature extraction ability of the self-attention module. At the same time, the multi-scale squeeze excitation function self-attention module is combined with the multi-layer pyramid network to further extract the perfect target feature information of the object in instance segmentation, remove the interference information, and obtain a more accurate and perfect feature map; secondly, FCOS BOXHEAD is used to calculate the required results. The method uses FCOS BOX HEAD to calculate the classification, center degree and box regression results in the feature map obtained by the MSEA module. Finally, the method designs a new mask branch for prediction, and designs mask score map loss and mask loss. The RA module is designed in the mask branch, and the mask score map loss and mask score map loss are proposed. After the mask branch pools the input data, the RA module is used to aggregate features and perform mask prediction on its input.

[0013] The beneficial effects of the present invention are

[0014] 1. Design a multi-scale squeeze-inspired self-attention and mask score map loss instance segmentation method (MSSAN), which can enhance the useful information channels in instance segmentation while making the model focus on the target area of ​​instance segmentation.

[0015] 2. Design a multi-scale self-attention module and construct a squeezing excitation function to improve the multi-scale self-attention module. At the same time, combine the multi-layer pyramid network to form a feature extraction network, which can obtain complete target feature information for image segmentation, remove interference information, and improve the accuracy and robustness of the instance segmentation method.

[0016] 3. Design a new mask branch, construct mask score map loss and mask loss in the mask branch, which can solve the problem of inaccurate edge and internal segmentation of instance masks. Mask score map loss can promote the predicted labels of the mask branch to be closer to the true value labels, and mask loss can enhance instance mask features and suppress non-instance masks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall framework diagram of the method of the present invention; Figure 2 It is the structure of the multi-scale self-attention (MSEA) module; Figure 3 is the structure of the self-attention module, Figure 4 It is the structure of the mask score map loss function. DETAILED DESCRIPTION

[0018] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The method is divided into four steps: (1) Preprocessing and preliminary feature extraction. The method preprocesses the image and uses the Vovnet backbone network and the multi-layer pyramid network (FPN) to perform preliminary feature extraction on the image. (2) Design a multi-scale squeeze-excited self-attention module for deep feature extraction. The method designs a multi-scale squeeze-excited self-attention module to perform in-depth feature extraction on the image in order to extract features of different scales in FPN; the method first introduces the squeeze-excitation (SE) module into the residual connection of the self-attention to construct a squeeze-excited self-attention (SEA) module, and combines multiple SEA modules to construct a multi-scale squeeze-excited self-attention (MSEA) module. Through the MSEA module, a richer and more complete instance segmentation feature map is obtained. (3) Use FCOSBOX HEAD to calculate the required results. The method uses FCOS BOX HEAD to calculate the classification, center degree and box regression results in the feature map obtained by the MSEA module. (4) Design a mask branch for prediction, and design mask score map loss and mask loss. Methods A new mask branch is designed, and an RA module is designed in the mask branch. After pooling the input data, the mask branch uses the RA module to aggregate features and performs mask prediction on its input. At the same time, mask score map loss and mask loss are proposed. To address the problem of inaccurate instance mask edge and internal segmentation, the score map loss is first calculated using the true value label and the mask prediction label to generate a mask score map. Then, the mask score map is multiplied by the output mask to calculate the mask loss value.

[0020] The first step is image preprocessing and preliminary feature extraction. The input image is sequentially input into the Vovnet deep backbone network and the multi-layer pyramid network (FPN). Considering the real-time and time complexity of the method, the method constructs a 5-layer pyramid network to perform preliminary feature extraction on the input image, and uses the feature extraction information x as the subsequent input.

[0021] Then the second step is to design a multi-scale squeeze-excited self-attention module for deep feature extraction. This step includes at least the following processes:

[0022] (1) Constructing the Self-Attention Module (SA)

[0023] Enter x into the attached Figure 3 The three convolutional neural networks f shown k 、f q and f v , we get k = f k (x), q = f q (x) and v = f v(x), the output of the atten network is obtained as

[0024] x atten =softmax(qk T s), (1)

[0025] Simultaneously define

[0026] x sa =x atten v. (2)

[0027] (2) Constructing a Multi-Scale Squeezing Excited Self-Attention (MSEA) Module

[0028] The squeeze excitation SE module consists of a squeeze part and an excitation part, and its output is

[0029] x se =x*f conv2 (R relu (f conv1 (f gvd (x)))) (3)

[0030] where f gvd is the adaptive average pooling function. R relu It is an activation function that is simple to calculate and converges quickly. conv1 ,f conv2 There are two convolutional neural networks with the same number of input and output channels and a 1x1 convolution kernel.

[0031] The SE module is then inserted into the residual connection of the self-attention mechanism by se And the residual connection output x with SE module sa The output of the squeeze-excited self-attention (SEA) module is obtained by adding

[0032] x sea =x se +x sa (4)

[0033] And by combining multiple SEA modules to construct the MSEA module, the output of the MSEA module is

[0034]

[0035] Where i = 1, ..., K, K is a positive integer, which is a parameter that can adjust the complexity of the model and needs to be flexibly set according to needs; when i = 1, x sea(0) =x.

[0036] Then in the third step, FCOS BOX HEAD is used to calculate the required results. The method uses FCOS BOX HEAD to calculate the classification, center and box regression results in the feature map obtained by the MSEA module.

[0037] Then in the fourth step, we build the RA module for prediction and design the mask score map loss and mask loss. The specific process includes at least:

[0038] (1) On the mask branch, the RA module is designed, and two types of convolutional neural networks, 1x1 and 3x3, are used to replace the consecutive identical convolutional neural networks. First, the features of the ROI output are passed through two 1x1 and one 3x3 convolutional neural network networks, and then the outputs of the three convolutional neural networks are added together to obtain x1:

[0039]

[0040] Then x1 is input into two convolutional neural networks with convolution kernel sizes of 1x1 and 3x3 respectively, and the outputs of the two convolutional networks are added together to get x2:

[0041]

[0042] Finally, x2 is input into a 3x3 convolutional network to obtain x3:

[0043]

[0044] Where 1x1 and 3x3 represent the size of the convolution kernel.

[0045] (2) The design process of the mask score map loss function is as follows:

[0046] The structure of the mask score map loss function is shown in the attached Figure 4 .

[0047] First, let the output of the RA module be T, perform a convolution operation on T and predict the feature map pre1, compare pre1 with the true value label Igt, and then obtain the loss function as follows:

[0048]

[0049] Where i, j represent the position, I gt represents the true value label, I pre Represents the predicted label of the network output.

[0050] Then, the L2 loss function is introduced in the mask branch to calculate the score map loss value. The mask score map loss is

[0051]

[0052] Where N and M are the length and width of the feature map respectively.

[0053] (3) The mask loss is designed as follows.

[0054] First, the score map is multiplied by the mask feature map to obtain the mask feature map. The score map loss and mask accuracy loss are calculated for the score map and the mask feature map respectively. The two losses are weighted and combined to obtain the mask loss.

[0055] First, define the initial mask score map m(x ij )for

[0056]

[0057] Further transform the mask score map m(x ij ) is multiplied by the mask feature x to obtain the mask feature map:

[0058] x m =m(x)*x (12)

[0059] The mask score map is then multiplied by the output T of the RA module and input into the convolutional neural network to obtain the feature map pre2, which is then compared with the true value label I gt Calculate the mask accuracy loss l mask ;

[0060] Secondly, the mask score loss l m and mask accuracy loss l mask The weighted combination obtains the mask loss l seg for

[0061] l seg =λ1l m (x)+λ2l mask (m(x)) (13)

[0062] Among them l m is the mask score map loss, l mask is the mask accuracy loss, variables λ1,λ2 are weight parameters to balance the two losses, and finally the above mask branch is used to perform mask prediction on the input.

Claims

1. An instance segmentation method based on multi-scale squeezing-inspired self-attention and mask score map loss is first characterized by performing image preprocessing and preliminary feature extraction; the method preprocesses the image, and uses a Vovnet deep backbone network and a multi-layer pyramid network (FPN) to extract features from the image to obtain preliminary image feature information.

2. Multi-scale squeezing-inspired instance segmentation method with self-attention and mask score map loss, which is then characterized by Design a multi-scale squeeze-excited self-attention module for deep feature extraction; On the basis of the multi-layer pyramid network, the method designs a multi-scale squeezed excitation function self-attention module, which constructs a multi-scale self-attention module and designs a squeezed excitation function to improve the feature extraction ability of the self-attention module. At the same time, the multi-scale squeezed excitation function self-attention module is combined with the multi-layer pyramid network to further extract the perfect target feature information of the object in the instance segmentation, remove the interference information, and obtain a more accurate and perfect feature map. This step at least includes the following processes: (1) Constructing the Self-Attention (SA) Module Input x into the three convolutional neural networks f shown in Figure 3 k 、f q and f v , we get k = f k (x), q = f q (x) and v = f v (x), the output of the atten network is obtained as x atten =softmax(qk T s), (1) At the same time x sa =x atten v; (2) (2) Constructing a Multi-Scale Squeezing Excited Self-Attention (MSEA) Module The squeeze excitation (SE) module consists of a squeeze part and an excitation part, and its output is x se =x*f conv2 (R relu (f conv1 (f gvd (x)))), (3) where f gvd is the adaptive average pooling function, R relu is the activation function, f conv1 and f conv2 They are two convolutional neural networks with the same number of input and output channels and a 1x1 convolution kernel; The SE module is then inserted into the residual connection of the self-attention mechanism by se and x sa The output of the squeeze-excited self-attention (SEA) module is obtained by adding x sea =x se +x sa (4) And by combining multiple SEA modules to construct the MSEA module, the output of the MSEA module is x sea(i) =f i (x sea(i-1) ) (5) Where i = 1, ..., K, K is a positive integer, which is a parameter that can adjust the complexity of the model and needs to be flexibly set according to needs; when i = 1, x sea(0) =x.

3. Multi-scale squeeze-inspired self-attention and mask score map loss instance segmentation method, secondly, it is characterized by FCOS BOX HEAD is used to calculate the required results. Methods FCOS BOX HEAD is used to calculate classification, center and box regression results in the feature map obtained by the MSEA module.

4. Multi-scale squeeze-inspired self-attention and mask score map loss instance segmentation method, the last feature is Construct the RA module for prediction, design the mask score map loss and the mask score map loss, and the specific process includes at least: (1) On the mask branch, the RA module is designed, and two types of convolutional neural networks, 1x1 and 3x3, are used to replace consecutive identical convolutional neural networks. First, the features of the ROI output are passed through two 1x1 and one 3x3 convolutional neural network networks, and then the outputs of the three convolutional neural networks are added together to obtain x1: Then x1 is input into two convolutional neural networks with convolution kernel sizes of 1x1 and 3x3 respectively, and the outputs of the two convolutional networks are added together to get x2: Finally, x2 is input into a 3x3 convolutional network to obtain x3: Among them, 1x1 and 3x3 represent the size of the convolution kernel; (2) The design process of the mask score map loss function is as follows: The structure of the score map loss function is shown in Figure 4; First, let the output of the RA module be T, perform a convolution operation on T and predict the feature map pre1, and compare pre1 with the true value label I gt Compare, and then get the loss function as follows Where i, j represent the position, I gt represents the true value label, I pre Represents the predicted label of the network output; Then, the L2 loss function is introduced in the mask branch to calculate the score map loss value. The mask score map loss is Where N and M are the length and width of the feature map respectively; (3) The mask loss is designed as follows: First, the score map is multiplied by the mask feature map to obtain the mask feature map. The score map loss and mask accuracy loss are calculated for the score map and the mask feature map respectively. The two losses are weighted and combined to obtain the mask loss. First, define the initial mask score map m(x ij )for Further, the mask score map m(x ij ) is multiplied by the mask feature x to obtain the mask feature map: x m =m(x)*x (12) The mask score map is then multiplied by the output T of the RA module and input into the convolutional neural network to obtain the feature map pre2, which is then compared with the true value label I gt Calculate the mask accuracy loss l mask ; Secondly, the mask score loss l m and mask accuracy loss l mask The weighted combination obtains the mask loss l seg for l seg =λ1l m (x)+λ2l mask (m(x)) (13) Among them l seg is the final mask loss, l m is the score map loss function, l mask is the mask accuracy loss, and variables λ1 and λ2 are weight parameters that balance the two losses. Finally, the above mask branch is used to perform mask prediction on the input.

Images