An end-to-end weakly supervised instance segmentation method and system

Abstract

The present invention belongs to the technical field of pattern recognition, and specifically discloses an end-to-end weakly supervised instance segmentation method, comprising the following steps: obtaining an input image, and preprocessing the input image to obtain a preprocessed image; performing feature extraction on the preprocessed image to obtain a multi-level feature map with the same channel; performing feature fusion on the multi-level feature map with the same channel to obtain hidden features of each instance in the image; obtaining the category and bounding box of each instance in the image based on the hidden features of each instance; obtaining the segmentation result of each instance in the image based on the category and bounding box of each instance in the image, thereby realizing end-to-end weakly supervised instance segmentation. The present invention proposes an end-to-end weakly supervised instance segmentation method equipped with a lightweight shared mask decoder. Transformer The proposed architecture is used for bounding box supervised instance segmentation, which solves the problem that existing bounding box-level instance segmentation methods are complex to train and require one-to-many label assignment and non-maximum suppression techniques.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to an end-to-end weakly supervised instance segmentation method and system. Background Art

[0002] Instance segmentation is a fundamental task in computer vision and is widely used in fields such as autonomous driving and healthcare. Accurate mask prediction poses a significant challenge to this task. MaskFormer , Mask2Former , Mask R-CNN Methods such as have shown competitiveness, but there are still many difficulties in the high cost of per-pixel mask annotation. Therefore, weakly supervised instance segmentation methods have become an exploratory direction to address these challenges.

[0003] Based on the level of supervision, weakly supervised instance segmentation can be divided into three categories: class-level, point / graffiti-level, and bounding box-level. Class-level supervision may lead to inaccurate mask predictions due to its difficulty in distinguishing different instances. In contrast, point / graffiti-level labels have the ability to distinguish objects and have been used in previous studies. However, point-level annotations may not provide enough information to accurately identify instances. Bounding box-level supervision is worthy of attention because it can more accurately locate instances and has been shown to be effective in weakly supervised instance segmentation.

[0004] Bounding box level instance segmentation is the focus of researchers' development and experimentation. Previous methods can be roughly divided into two categories: one is to generate pseudo masks and use them to train fully supervised instance segmentation networks, but such methods usually require multi-stage training or supplementary data; the other method requires the formulation of constraints consistent with bounding box annotation and segmentation tasks, adopts a convolutional architecture, and requires one-to-many label assignment and non-maximum suppression (NMS) technology. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing bounding box level instance segmentation method is complex to train and requires one-to-many label assignment and non-maximum suppression technology, and proposes an end-to-end weakly supervised instance segmentation method and system.

[0006] The technical solution of the present invention is as follows: In a first aspect, an end-to-end weakly supervised instance segmentation method is provided, the method comprising the following steps:

[0007] S1. Get the input image and extract the features of the input image to obtain a multi-level feature map with the same channel ;

[0008] S2. Multi-level feature maps with the same channel Perform feature fusion to obtain the hidden features of each instance in the image ;

[0009] S3. Hidden features for each instance Through the transformation and calculation of the category branch, regression branch and mask branch, the category, bounding box and mask query of each instance in the image are obtained. ;

[0010] S4. Query based on the category, bounding box and mask of each instance in the image , the segmentation results of each instance in the image are obtained, and end-to-end weakly supervised instance segmentation is achieved.

[0011] The beneficial effects of the present invention are:

[0012] 1. The present invention proposes a method based on Transformer A novel end-to-end weakly supervised instance segmentation method based on the architecture without using one-to-many label assignment and NMS , therefore, the method of the present invention is more direct and efficient.

[0013] 2. The method provided by the present invention has better performance than the weakly supervised instance segmentation method without the need for additional data or a multi-stage training process.

[0014] Preferably, the step S1 specifically includes the following steps:

[0015] S11. Get the input image and use the backbone network to Perform feature extraction to obtain a multi-level feature map ;

[0016] S12. Multi-level feature maps through convolutional layers Processing is performed to obtain a multi-level feature map with the same channel ,in, .

[0017] The beneficial effects of the above preferred solution are:

[0018] Through the above preferred solution, the multi-dimensional feature map is processed, and all feature maps share the same dimension, so that the feature map has more information, which is helpful Transformer Encoder processing.

[0019] Preferably, the step S2 specifically comprises the following steps:

[0020] S21. Using attention mechanism to deal with multi-level feature maps with the same channel Flatten and connect the reconstruction to obtain a reconstructed multi-level feature map;

[0021] S22. Leveraging Learnable Object Queries Based on Multi-Scale Deformable Attention Methods and Input Attention Mechanisms , through the cross-attention mechanism to establish the reconstructed multi-level feature map and object query The correlation between them can be used to obtain the hidden features of each instance in the image. ;

[0022] The hidden features The calculation formula is:

[0023]

[0024] in, represents the output of the encoder, Represents the output of the decoder.

[0025] The beneficial effects of the above preferred solution are:

[0026] Through the above preferred solution, feature maps of different levels are fused to enhance the representation capability of the feature maps.

[0027] Preferably, the step S4 specifically comprises the following steps:

[0028] S41. Multi-level feature map In Processing to obtain feature map ;

[0029] S42. Query the mask As query input, feature map As key and value input, output the associated masked query and feature map ;

[0030] S43. Query the associated mask and feature map Perform dot product operation to get the mask ;

[0031] S44. According to the category, bounding box and mask of each instance in the image , the segmentation results of each instance in the image are obtained, and end-to-end weakly supervised instance segmentation is achieved.

[0032] In a second aspect, an end-to-end weakly supervised instance segmentation system is provided, the system comprising:

[0033] The backbone network module is used to receive the preprocessed image, extract features from the preprocessed image, and output a multi-level feature map with the same channel. ;

[0034] Transformer An encoder-decoder module for receiving the multi-level feature map having the same channel , output the hidden features of each instance in the image ;

[0035] The prediction head module is used to receive and convert the hidden features of each instance in the image. , and outputs the category, bounding box, and mask query for each instance in the image ;

[0036] A mask encoder-decoder module that receives a query of the category, bounding box, and mask of each instance in the image. , and output the segmentation results of each instance in the image.

[0037] The beneficial effects of the present invention are:

[0038] The mask decoder proposed in the present invention utilizes the adaptability of attention to establish connections between instances, which can be implemented through efficient matrix multiplication operations, and can obtain more accurate mask predictions.

[0039] Preferably, the Transformer The encoder-decoder module includes a deformable encoder and a deformable decoder;

[0040] The deformable encoder includes six layers of network units, each of which is composed of a stack of multi-scale deformable attention layers and feedforward network layers;

[0041] The deformable decoder includes six layers of network units, each of which is composed of a stack of self-attention layers and cross-attention layers.

[0042] The beneficial effects of the above preferred solution are:

[0043] By stacking multi-scale deformable attention layers and feedforward network layers and introducing a deformable encoder, the fusion of feature maps at different levels is achieved, and the representation ability of feature maps is enhanced; at the same time, by introducing self-attention layers and cross-attention layers into the deformable decoder, the Transformer Performance of the encoder-decoder module.

[0044] Through the above preferred solution,

[0045] Preferably, the prediction head module comprises a category branch, a regression branch and a mask branch connected in parallel;

[0046] A linear layer is provided on the category branch, and the formula for calculating the category of each instance is:

[0047]

[0048] in, Indicates the category of each instance, Represents the output of the linear layer;

[0049] The regression branch is provided with a multi-layer perceptron, and the formula for calculating the bounding box is:

[0050]

[0051] in, represents the bounding box, Represents the output of the multilayer perceptron on the regression branch;

[0052] The mask branch is provided with a multi-layer perceptron, which calculates the mask query The formula is:

[0053]

[0054] in, Represents the output of the multilayer perceptron on the mask branch.

[0055] The beneficial effects of the above preferred solution are:

[0056] By introducing the mask branch, the mask prediction of each instance is achieved.

[0057] Preferably, the mask encoder-decoder module comprises a mask encoder and a shared mask decoder;

[0058] The shared mask decoder consists of two stacked cross-attention layers and self-attention layers.

[0059] The beneficial effects of the above preferred solution are:

[0060] The mask queries are associated with feature maps through cross-attention units, which helps predict the final mask; object queries are connected to each other through self-attention units, so that weakly supervised instance segmentation can achieve higher performance.

[0061] Preferably, the loss function calculation formula of the system during training is:

[0062]

[0063] in, represents the detection loss function; represents the focal loss function; represents the segmentation loss function; Represents a multi-level feature map; Indicates the classification result of the predicted positive sample; represents a positive sample; represents the regression loss; Indicates a callout box; Represents the prediction box; represents the mapping loss; represents the prediction mask; represents pairwise loss; Indicates the predicted color similarity; Indicates edge color similarity; Indicates the number of edges with at least one point in the true value box; Indicates the maximum value operation in the y-axis direction; Indicates the maximum value operation in the x-axis direction; Represents the indicator function, when it satisfies Its value is 1 when it is, otherwise it is 0; Represents the Dice coefficient loss function; Represents the logarithmic function with the natural logarithm e as the base.

[0064] The beneficial effects of the above preferred solution are:

[0065] Through the focal loss function, the supervised classification task is achieved; through the segmentation loss function, objects are localized by leveraging bounding box annotations and more accurate mask predictions can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 The embodiment 1 of the present invention provides Transformers Flowchart of the end-to-end weakly supervised instance segmentation method.

[0067] Figure 2 The figure shows a structural diagram of an end-to-end weakly supervised instance segmentation system provided by Embodiment 2 of the present invention.

[0068] Figure 3 Shown is the comparison result between the present invention and the latest method on the weakly supervised instance segmentation task provided by Example 3 of the present invention. DETAILED DESCRIPTION

[0069] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0070] Embodiment 1:

[0071] like Figure 1 As shown, an end-to-end weakly supervised instance segmentation method comprises the following steps:

[0072] S1. Obtain an input image, preprocess the input image to obtain a preprocessed image, and extract features from the preprocessed image to obtain a multi-level feature map with the same channel ;

[0073] S2. Multi-level feature maps with the same channel Perform feature fusion to obtain the hidden features of each instance in the image ,in, Indicates the number of prediction boxes , Indicates the feature map dimension;

[0074] S3. Hidden features for each instance Through the transformation and calculation of the category branch, regression branch and mask branch, the category, bounding box and mask query of each instance in the image are obtained. ;

[0075] S4. Query based on the category, bounding box and mask of each instance in the image , the segmentation results of each instance in the image are obtained, and end-to-end weakly supervised instance segmentation is achieved.

[0076] In this embodiment, step S1 specifically includes the following steps:

[0077] S11. Get the input image and process the preprocessed image through the backbone network Perform feature extraction to obtain a multi-level feature map ,in, represents a real number, represents the length of the input image, represents the width of the input image, Indicates The length of the layer feature map, Indicates The width of the layer feature map, , , and The specific values ​​of vary according to certain proportions;

[0078] S12. Multi-level feature maps through convolutional layers Processing is performed to obtain a multi-level feature map with the same channel .

[0079] In this embodiment, step S2 specifically includes the following steps:

[0080] S21. Using attention mechanism to deal with multi-level feature maps with the same channel Flatten and connect the reconstruction to obtain a reconstructed multi-level feature map;

[0081] S22. Leveraging Learnable Object Queries Based on Multi-Scale Deformable Attention Methods and Input Attention Mechanisms , through the cross-attention mechanism to establish the reconstructed multi-level feature map and object query The correlation between them can be used to obtain the hidden features of each instance in the image. ;

[0082] The hidden features The calculation formula is:

[0083]

[0084] in, represents the output of the encoder, Represents the output of the decoder.

[0085] In this embodiment, step S4 specifically includes the following steps:

[0086] S41. Multi-level feature map In Processing to obtain feature map ;

[0087] S42. Query the mask As query input, feature map As key and value input, output the associated masked query and feature map ;

[0088] S43. Query the associated mask and feature map Perform dot product operation to get the mask ;

[0089] S44. According to the category, bounding box and mask of each instance in the image , the segmentation results of each instance in the image are obtained, and end-to-end weakly supervised instance segmentation is achieved.

[0090] Embodiment 2:

[0091] The embodiment of the present invention provides an end-to-end weakly supervised instance segmentation system. Figure 2 As shown, for executing the end-to-end weakly supervised instance segmentation method described in configuration embodiment 1, the system includes:

[0092] The backbone network module is used to receive an input image, extract features from the input image, and output a multi-level feature map with the same channel. ;

[0093] Transformer An encoder-decoder module for receiving the multi-level feature map having the same channel , output the hidden features of each instance in the image ;

[0094] The prediction head module is used to receive and convert the hidden features of each instance in the image. , and outputs the category, bounding box, and mask query for each instance in the image ;

[0095] A mask encoder-decoder module that receives a query of the category, bounding box, and mask of each instance in the image. , and output the segmentation results of each instance in the image.

[0096] In this embodiment, the backbone network is ResNet Series or Swin series, and introduces an additional convolutional layer to ensure that all feature maps share the same dimension , Position encoding is also introduced in the backbone network. For a certain layer of feature map, position encoding uses the periodicity of the sine function to generate a new feature map of the same size as the feature map to represent the position encoding of the feature map.

[0097] In this embodiment, the Transformer The encoder-decoder module includes a deformable encoder and a deformable decoder;

[0098] The deformable encoder includes six layers of network units, each of which is composed of a stack of multi-scale deformable attention layers and feedforward network layers; the attention mechanism is applied to the feature map , first flatten it into , which are then concatenated and the encoder computes the reconstructed multi-level feature maps, which are subsequently transferred to the decoder for further processing.

[0099] Due to the computational complexity introduced by the attention mechanism, this paper adopts a multi-scale deformable attention method. In order to explain the permutation invariance of attention, position encoding is introduced in the input of each attention layer to represent the position of each pixel. In addition, scale-level embedding is combined to distinguish the feature level to which each query pixel belongs.

[0100] The deformable decoder includes six layers of network units, each of which is composed of a stack of self-attention layers and cross-attention layers; the self-attention layers promote the interaction between object queries, while the cross-attention layers enable object queries to interact with multi-level feature maps, thereby improving the target hidden features. accuracy.

[0101] Object Query are proposed features or learnable parameters belonging to different instances. These object queries are initially processed by the self-attention module into query, key, and value elements, thereby establishing the correlation between them. The resulting output of the self-attention module is then used as the query input of the cross-attention, where the multi-level feature maps are used as key and value inputs. In addition, the decoder also incorporates positional encodings. The length of the object query is represented as , that is, the number of prediction boxes , set to 300.

[0102] In this embodiment, the prediction head module includes a category branch, a regression branch, and a mask branch connected in parallel;

[0103] The category branch is provided with a linear layer, and the category branch transforms the hidden features through a single linear layer. Converted into classification results, the formula for calculating the category of each instance is:

[0104]

[0105] in, Indicates the category of each instance, Represents the output of the linear layer;

[0106] The regression branch is provided with a multi-layer perceptron. ReLU The activated multilayer perceptron generates information about each bounding box. The normalized center coordinates, height and width predictions are , and its calculation formula is:

[0107]

[0108] in, Represents the output of the multilayer perceptron on the regression branch;

[0109] The mask branch is provided with a multi-layer perceptron, and the mask branch uses the multi-layer perceptron to calculate the target hidden features , and select the positive sample as the mask query , which is then transferred to the mask encoder-decoder module, where the mask query The calculation formula is:

[0110]

[0111] in, Represents the output of the multilayer perceptron on the mask branch.

[0112] In this embodiment, the mask encoder-decoder module includes a mask encoder and a shared mask decoder; the feature map This is processed in the Mask Encoder module to generate the output , which is then passed as key and value input to the mask decoder module;

[0113] The shared mask decoder consists of two stacked cross-attention layers and self-attention layers. In the cross-attention module, the mask query is used as query input, and The criss-cross attention module is designed to associate the mask query with the feature map, which helps predict the final mask. Considering that the object may appear anywhere in the image, the traditional attention module is used instead of the deformable attention module. In the self-attention module, the mask query serves as the input of the query, key, and value. By connecting the object queries to each other, weakly supervised instance segmentation can achieve higher performance. Finally, to generate the final mask, the output of the decoder is fused using a dot product operation. and The resulting mask is expressed as .

[0114] In this embodiment, the Hungarian algorithm is used as the optimal strategy for label assignment. For supervised classification tasks, focal loss is used. IoU The loss and L1 loss are applied for bounding box regression, the projection loss is used to localize the object by leveraging the bounding box annotations, and for more accurate mask prediction, the pairwise loss is adopted ( pairwise loss ). This loss exploits the fact that when a pixel and its neighbors have similar colors, they tend to belong to the same class; the loss function calculation formula for the end-to-end weakly supervised instance segmentation during training is:

[0115]

[0116] in, represents the detection loss function; represents the focal loss function; represents the segmentation loss function; Represents a multi-level feature map; Indicates the classification result of the predicted positive sample; represents a positive sample; represents the regression loss; Indicates a callout box; Represents the prediction box; represents the mapping loss; represents the prediction mask; represents pairwise loss; Indicates the predicted color similarity; Indicates edge color similarity; Indicates the number of edges with at least one point in the true value box; Indicates the maximum value operation in the y-axis direction; Indicates the maximum value operation in the x-axis direction; Represents the indicator function, when it satisfies Its value is 1 when it is, otherwise it is 0; Represents the Dice coefficient loss function; Represents the logarithmic function with the natural logarithm e as the base.

[0117] Embodiment 3:

[0118] On the basis of Example 1, the method provided by the present invention is experimentally compared with the latest method, and experiments are conducted on a widely used instance segmentation dataset, namely MS COCO . MS COCO It is a widely used dataset for object detection and instance segmentation, consisting of a training set (115K images), a validation set (5K images), and a test set (20K images). The dataset includes 80 object classes, such as people, bicycles, cars, and motorcycles. The experimental results are as follows: Figure 3 As shown, it can be obtained that the present invention has a better technical effect.

[0119] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An end-to-end weakly supervised instance segmentation method, characterized in that: The method comprises the following steps: S1. Get the input image and extract the features of the input image to obtain a multi-level feature map F with the same channel i , where i∈3,4,5, step S1 specifically includes the following steps: S11. Get the input image, extract features from the input image I through the backbone network, and obtain a multi-level feature map C i ; S12. Through the convolution layer, the multi-level feature map C i Processing is performed to obtain a multi-level feature map F with the same channel i ; S2. For multi-level feature maps F with the same channel i Perform feature fusion to obtain the hidden features Z of each instance in the image q , step S2 specifically includes the following steps: S21. Use the attention mechanism to analyze the multi-level feature maps F with the same channel i Flatten and connect the reconstruction to obtain a reconstructed multi-level feature map; S22. Based on the input of the multi-scale deformable attention method and the attention mechanism, the correlation between the reconstructed multi-level feature map and the object query q is established through the cross attention mechanism to obtain the hidden features Z of each instance in the image. q ; The hidden feature Z q The calculation formula is: Z q =Decoder(q,Encoder(F i )) Where Encoder(·) represents the output of the encoder, and Decoder(·) represents the output of the decoder; S3. Hidden features Z for each instance q Through the transformation and calculation of the category branch, regression branch and mask branch, the category, bounding box and mask query m of each instance in the image are obtained. q ; S4. Query m based on the category, bounding box and mask of each instance in the image q , the segmentation results of each instance in the image are obtained, and end-to-end weakly supervised instance segmentation is achieved.

2. The end-to-end weakly supervised instance segmentation method according to claim 1, characterized in that The step S4 specifically comprises the following steps: S41. Multi-level feature map F i Process F3 in the figure to obtain feature map F3′; S42. Query the mask m q As query input, feature map F3′ is input as key and value, and the associated mask query m′ is output q and feature map F3″; S43. Query the associated mask m′ q Perform dot product operation with feature map F3″ to obtain mask m; S44. According to the category, bounding box and mask m of each instance in the image, the segmentation result of each instance in the image is obtained to achieve end-to-end weakly supervised instance segmentation.

3. An end-to-end weakly supervised instance segmentation system, used to perform the end-to-end weakly supervised instance segmentation method according to any one of claims 1-2, characterized in that: The system comprises: The backbone network module is used to receive an input image, extract features from the input image, and output a multi-level feature map F with the same channel. i ; Transformer encoder-decoder module, used to receive the multi-level feature map F with the same channel i , output the hidden features Z of each instance in the image q ; The prediction head module is used to receive and convert the hidden features Z of each instance in the image. q , and output the category, bounding box and mask query m of each instance in the image q ; A mask encoder-decoder module is configured to receive a category, a bounding box, and a mask query m for each instance in the image. q , and output the segmentation results of each instance in the image; The Transformer encoder-decoder module includes a deformable encoder and a deformable decoder; The deformable encoder includes six layers of network units, each of which is composed of a stack of multi-scale deformable attention layers and feedforward network layers; The deformable decoder includes six layers of network units, each of which is composed of a stack of self-attention layers and cross-attention layers.

4. The end-to-end weakly supervised instance segmentation system according to claim 3, characterized in that The prediction head module includes a category branch, a regression branch and a mask branch connected in parallel; A linear layer is provided on the category branch, and the formula for calculating the category of each instance is: p=Linear(Z q ) Where p represents the category of each instance, and Linear(·) represents the output result of the linear layer; The regression branch is provided with a multi-layer perceptron, and the formula for calculating the bounding box is: b=MLP1(Z q ) Where b represents the bounding box, MLP1(·) represents the output of the multilayer perceptron on the regression branch; The mask branch is provided with a multi-layer perceptron, which calculates the mask query m q The formula is: m q =MLP2(Z q ) Where MLP2(·) represents the output result of the multilayer perceptron on the mask branch.

5. The end-to-end weakly supervised instance segmentation system according to claim 3, characterized in that The mask encoder-decoder module includes a mask encoder and a shared mask decoder; The shared mask decoder consists of two stacked cross-attention layers and self-attention layers.

6. The end-to-end weakly supervised instance segmentation system according to claim 3, characterized in that The loss function calculation formula of the system during training is: Among them, L overall represents the detection loss function; L det represents the focal loss function; L seg represents the segmentation loss function; C i Represents a multi-level feature map; Indicates the classification result of the predicted positive sample; represents a positive sample; L box (·) represents regression loss; b l Indicates a callout box; represents the prediction box; L proj (·) represents mapping loss; represents the prediction mask; L pair (·) represents pairwise loss; s l Indicates the predicted color similarity; Indicates edge color similarity; N indicates the number of edges with at least one point in the true value box; max y (·) indicates the maximum value operation in the y-axis direction; max x (·) represents the maximum value operation in the x-axis direction; represents the indicator function, when s is satisfied l Its value is 1 when ≥τ, otherwise it is 0; L(·,·) represents the Dice coefficient loss function; log represents the logarithmic function with the natural logarithm e as the base.

Images