A single-stage instance segmentation method based on deep learning

Abstract

The application discloses a single-stage instance segmentation method based on deep learning, wherein edge information of each instance is extracted through a traditional edge detection operator, so that the network shows supervision on the learning of the instance contour in the process of training and learning, the integration of the edge information not only improves the segmentation precision of the instance mask, but also makes the algorithm after the integration of the edge information more delicate in the performance at the edge of the segmentation mask; and the application increases the fusion process of the prototype mask from one time to two times through a method of grouping and then fusing the prototype mask, so that the two-time fusion not only improves the segmentation effect of the instance mask, but also increases the generalization ability of the whole algorithm.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and computer vision technology, and specifically relates to a single-stage instance segmentation method based on deep learning. Background Technology

[0002] Instance segmentation is a computer vision task that most closely resembles human visual perception. Instance segmentation aims to solve the problems of target localization and segmentation simultaneously.

[0003] With the development of deep learning technology, numerous algorithms have emerged in the fields of object detection and semantic segmentation, greatly promoting the development of instance segmentation. Many excellent instance segmentation algorithms have been proposed. Most existing instance segmentation algorithms extend existing excellent object detectors, introducing an instance mask prediction branch on top of the object detection algorithm. Mainstream instance segmentation algorithms can be broadly divided into two categories: algorithms based on single-stage detector extensions and algorithms based on two-stage detector extensions. Instance segmentation algorithms based on two-stage detectors, due to the use of more complex visual features and more rigorous object detectors, often achieve more accurate segmentation results, higher localization accuracy, and stronger scalability. They perform better in handling complex scenes and large-scale datasets.

[0004] However, existing methods such as Yolact and Yolact++ have the following drawbacks:

[0005] The problem of insufficient attention to instance edge information leads to an uneven segmentation mask contour.

[0006] In the process of combining prototype masks, there are errors in the prediction of combination coefficients, resulting in inaccurate segmentation results. Summary of the Invention

[0007] To address the problems mentioned in the background art, this invention provides a single-stage instance segmentation method based on deep learning, which solves the problems of insufficient attention to instance edge information and errors in the prediction of combination coefficients in the prior art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A single-stage instance segmentation method based on deep learning includes the following steps:

[0010] S1: Based on the YOLACT method, instance segmentation is divided into two parallel tasks: one task is to generate the prototype mask of the current image, and the other task is to predict the mask coefficients of each instance.

[0011] S2: Perform a linear combination of the prototype mask and mask coefficients, and activate the combination result using an activation function to obtain the prediction mask;

[0012] S3: Use the predicted mask as the input to the edge information prediction branch, apply the edge operator to the predicted mask to obtain the prediction result of the instance edge mask, and apply the edge operator to the real mask of the instance to obtain the instance edge mask label.

[0013] S4: Add a prediction instance edge map branch. The head input of the edge map prediction branch is the prediction result and the matching instance edge mask label. Convolution is performed by the convolution operator to obtain the prediction edge map and the label of the corresponding edge map. Construct the loss function and optimize the edge loss to obtain the final model.

[0014] S5: Input the instance to be segmented into the final model to obtain the segmentation result.

[0015] Preferably, the method for generating the prototype mask is as follows: k prototype masks are generated for the entire image by predicting the prototype mask prediction branch in YOLACT. The prototype mask sub-network protonet is implemented as an FCN. The last layer of the FCN network has k channels, each channel representing a prototype mask. The input of protonet is the feature layer of the backbone network or the output of the feature layer after the feature fusion network.

[0016] Preferably, the method for generating the mask coefficients is as follows:

[0017] The prediction head of the anchor-based target detector in YOLACT has two branches:

[0018] One branch predicts the classification confidence of c classes for each anchor box, where c represents the number of classes of the target in a specific dataset;

[0019] The other branch predicts four regression box coefficients for each anchor box;

[0020] By adding a third branch to predict the coefficients of k prototype masks, with each prototype mask coefficient corresponding to a prototype mask, Yolac expands the detector's prediction header output from 4+c prediction coefficients to 4+c+k prediction coefficients.

[0021] The prototype mask has a base of 8, denoted as symbol b, where b is an adjustable hyperparameter.

[0022] The k=32 prototype masks generated by the prototype mask prediction branch in the YOLACT structure are split along the channel dimension. Each prototype mask group has a corresponding prototype mask prediction coefficient that will be split into g groups from the channel dimension.

[0023] Based on the Yolact prototype mask coefficient prediction branch, a fourth branch is added in parallel to predict the group coefficients of the prototype mask, expanding the output of the prediction head of the Yolact detector from 4+c+k predicted coefficients to 4+c+k+g predicted coefficients.

[0024] Preferably, the linear combination method is as follows:

[0025] The k prototype masks are ordered into groups of b in each channel dimension, and the k prototype mask coefficients are also ordered into groups of b in each channel dimension, to obtain inputs Pi and Ci respectively.

[0026] By combining Pi and Ci, the intermediate prototype mask generated after the combination of each group is obtained;

[0027] The group coefficient Gi of each group is obtained from the channel dimension of the coefficients of the g prototype mask groups, which is then used as the input for the second fusion process.

[0028] The fusion result of each group is obtained by combining the intermediate prototype mask and the prototype mask group coefficients;

[0029] The entire fusion process can be defined by summing the fusion results of each group and generating a prediction mask after activation by the nonlinear function Sigmoid:

[0030] ;

[0031] Where σ is the Sigmoid activation function, Pi is an h×w×b matrix for the prototype mask of each group, Ci is the n instances retained after NMS and thresholding, the prototype mask coefficients in each group are an n×b matrix, and Gi is an n×1 matrix for the group coefficients of the prototype mask of the n instances.

[0032] Preferably, the method for obtaining the instance edge mask label in S3 is as follows:

[0033] Let M be the prediction mask for instances in the image. p The mask of n instances in an image is represented as an n×w×h tensor, and the n×w×h tensor is used as the input of the edge information prediction branch.

[0034] Apply the Sobel or Laplacian edge detection operator to the prediction mask M of the instance. p Then, the prediction result M of the instance edge mask is obtained. pedge ;

[0035] Accordingly, the edge detection operator is applied to the real mask M of the instance. gt This will give you the label M of the instance edge mask. gtedge ;

[0036] Let K be the operator for edge detection, and let M be the mask that interacts with K. mask The entire process of extracting the instance edge mask can then be defined as:

[0037] M edge =K∗M mask .

[0038] Preferably, in S4, a branch for predicting the edge map of the instance is added, and the input of the edge map prediction head is the predicted mask map M. p and the matching real mask M gt The predicted edge map M is obtained by performing convolution operations using convolution operators respectively. pedge And the corresponding edge map label M gtedge The binary cross-entropy function is chosen to optimize the edge loss, and can be defined as:

[0039] Loss=L cls +L box +L mask +L edge ;

[0040] Where L cls For classification loss, L box For bounding box regression loss, L mask For the binary cross-entropy loss of the instance mask, L edge This is the binary cross-entropy loss of the edge graph.

[0041] Preferably, the labels of the instance edge mask are processed using an image smoothing filter, and then the loss is calculated between the smoothed instance edge mask labels and the predicted mask.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This application extracts edge information for each instance using traditional edge detection operators, enabling the network to learn instance contours under explicit supervision during training. The incorporation of edge information not only improves the segmentation accuracy of the instance mask but also results in a more refined performance at the edges of the segmented mask.

[0044] This application increases the prototype mask fusion process from one to two by grouping prototype masks and then fusing them. The two-step fusion method not only improves the segmentation effect of the instance mask but also increases the generalization ability of the entire algorithm. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the network structure of the present invention. Detailed Implementation

[0046] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0047] Example 1

[0048] like Figure 1 As shown, a single-stage instance segmentation method based on deep learning includes the following steps:

[0049] S1: Based on the YOLACT method, instance segmentation is divided into two parallel tasks: one task is to generate the prototype mask of the current image, and the other task is to predict the mask coefficients of each instance.

[0050] S2: Perform a linear combination of the prototype mask and mask coefficients, and activate the combination result using an activation function to obtain the prediction mask;

[0051] S3: Use the predicted mask as the input to the edge information prediction branch, apply the edge operator to the predicted mask to obtain the prediction result of the instance edge mask, and apply the edge operator to the real mask of the instance to obtain the instance edge mask label.

[0052] S4: Add a prediction instance edge map branch. The head input of the edge map prediction branch is the prediction result and the matching instance edge mask label. Convolution is performed by the convolution operator to obtain the prediction edge map and the label of the corresponding edge map. Construct the loss function and optimize the edge loss to obtain the final model.

[0053] S5: Input the instance to be segmented into the final model to obtain the segmentation result.

[0054] In this embodiment, the application extracts edge information for each instance using traditional edge detection operators, thereby enabling the network to learn instance contours under explicit supervision during training. The incorporation of edge information not only improves the segmentation accuracy of the instance mask but also results in a more refined performance at the mask edges. Furthermore, the application increases the prototype mask fusion process from one to two by grouping and then fusing prototype masks. This two-step fusion approach not only improves the instance mask segmentation effect but also enhances the generalization ability of the entire algorithm.

[0055] Example 2

[0056] The difference between this embodiment and Embodiment 1 is that the method for generating the prototype mask is as follows: the purpose of the prototype mask prediction branch in YOLACT is to predict k prototype masks for the entire image. The prototype mask sub-network protonet is implemented as FCN. The last layer of the FCN network has k channels, each channel representing a prototype mask. The input of protonet is the feature layer of the backbone network or the output of the feature layer after the feature fusion network.

[0057] This modeling approach is similar to that of standard semantic segmentation networks, but the difference is that in instance segmentation tasks, the computation of the explicit loss in the prototype mask is often discarded. Instead, the prototype mask subnetwork is trained by supervising the loss between the assembled instance mask and its labeled mask.

[0058] Example 3

[0059] The difference between this embodiment and Embodiment 2 is that the method for generating the mask coefficients is as follows:

[0060] The prediction head of the anchor-based target detector in YOLACT has two branches:

[0061] One branch predicts the classification confidence of c classes for each anchor box, where c represents the number of classes of the target in a specific dataset;

[0062] The other branch predicts four regression box coefficients for each anchor box;

[0063] YOLACT adds a third branch in parallel to predict the coefficients of k prototype masks, each coefficient corresponding to a prototype mask. k can be understood as a weight similar to channel attention, which represents the contribution of each prototype mask to the entire instance mask generation process.

[0064] Yolact expands the output of the detector head from 4+c coefficients to 4+c+k coefficients;

[0065] Here we rethink the role of the k prototype coefficients in the process of combining prototype masks to generate the final instance mask. For the final mask of an instance, addition can enhance the quality of a particular instance mask, while subtraction can remove the influence of other instance masks on our desired target instance mask. This means that for the coefficients of the generated k prototype masks, we need subtraction operations instead of only addition operations during the combination process. We choose Tanh instead of ReLU as the activation function for the k prototype mask coefficients.

[0066] In the Yolac algorithm, when the number of prototype masks k=8, a good instance segmentation mask can be generated. Therefore, we choose the prototype mask base number of each prototype mask group to be 8, denoted as b, where b is an adjustable hyperparameter.

[0067] The k=32 prototype masks generated by the prototype mask prediction branch in the YOLACT structure are split along the channel dimension. Each prototype mask group has a corresponding prototype mask prediction coefficient that will be split into g groups from the channel dimension.

[0068] Based on the Yolact prototype mask coefficient prediction branch, a fourth branch is added in parallel to predict the group coefficients of the prototype mask, expanding the output of the prediction head of the Yolact detector from 4+c+k coefficients to 4+c+k+g coefficients.

[0069] Example 4

[0070] The difference between this embodiment and embodiment 3 is that: Figure 1 As shown, the linear combination method is:

[0071] To obtain the final instance mask, Yolcat combines the outputs of the prototype mask prediction branch and the prototype mask coefficient prediction branch.

[0072] Yolact generates the final instance mask by linearly combining the prototype mask and its coefficients, and then activating the combined output using the nonlinear sigmoid function. This modeling approach can be implemented as a simple matrix multiplication and sigmoid activation, as follows:

[0073] M=σ(PCT)

[0074] Where σ is the Sigmoid activation function, P indicates that the prototype mask is an h×w×k matrix, and C represents the prototype mask coefficients of the n instances that are still retained after nonmaximum suppression and thresholding, which is an n×k matrix.

[0075] More complex combinatorial steps are possible, but to maintain the algorithm's speed and simplicity, Yolact chooses the simplest linear combinatorial approach. We use three losses to train the Yolact model:

[0076] Classification loss Lcls;

[0077] Bounding box regression loss (Lbox);

[0078] 3) Instance mask loss Lmask; the weights of each loss in the experiment were 1, 1.5, and 6.125, respectively. We computed pixel-wise binary cross entropy between the assembled instance mask and the labeled ground truth instance mask, which can be defined as:

[0079] L mask =BCE(M, M gt )

[0080] Where M is the instance mask generated by the Yolac model assembly, and Mgt is the instance mask of the real data annotation.

[0081] The k prototype masks are ordered into groups of b in each channel dimension, and the coefficients of the k prototype masks are also ordered into groups of b in each channel dimension, to obtain the inputs Pi and Ci respectively.

[0082] By combining Pi and Ci, the intermediate prototype mask generated after the combination of each group is obtained;

[0083] The group coefficient Gi for each group is obtained in the channel dimension of the g prototype mask group coefficients, which is then used as the input for the second fusion process.

[0084] The fusion result of each group is obtained by combining the intermediate prototype mask and the prototype mask group coefficients;

[0085] The entire fusion process can be defined by summing the fusion results of each group and generating a prediction mask after activation by the nonlinear function Sigmoid:

[0086] ;

[0087] Where σ is the Sigmoid activation function, Pi is an h×w×b matrix for the prototype mask of each group, Ci is the n instances retained after NMS and thresholding, the coefficients of the prototype mask in each group are an n×b matrix, and Gi is an n×1 matrix for the group coefficients of the prototype mask of the n instances.

[0088] Example 5

[0089] The difference between this embodiment and embodiment 4 is that: Figure 1 As shown, the method for obtaining the instance edge mask label in S3 is as follows:

[0090] Let M be the prediction mask for instances in the image. p The mask of n instances in an image can be represented as an n×w×h tensor, and the n×w×h tensor serves as the input to the edge information prediction branch.

[0091] Apply the Sobel or Laplacian edge detection operator to the prediction mask M of the instance. p Then, the prediction result M of the instance edge mask is obtained. pedge ;

[0092] Accordingly, the edge detection operator is applied to the real mask M of the instance. gt This will give you the label M of the instance edge mask. gtedge ;

[0093] The specific method for optimizing the gap between the prediction result and the instance edge mask label in S4 is as follows:

[0094] Let K be the operator for edge detection, and let M be the mask that interacts with K. mask The entire process of extracting the instance edge mask can then be defined as:

[0095] M edge = K ∗ M mask .

[0096] Example 6

[0097] The difference between this embodiment and embodiment 5 is that in S4, to calculate the final edge loss, we add a branch for predicting instance edge maps to the GroupYolact network structure. The input to the edge map prediction head is the predicted mask map M. p and the matching real mask M gt The predicted edge map M is obtained by performing convolution operations using convolution operators respectively. pedge And the corresponding edge map label M gtedge The binary cross-entropy function is chosen to optimize the edge loss, and can be defined as:

[0098] Loss=L cls +L box +L mask +L edge ;

[0099] Where L cls For classification loss, L box For bounding box regression loss, L mask For the binary cross-entropy loss of the instance mask, L edge This is the binary cross-entropy loss of the edge graph.

[0100] Example 7

[0101] The difference between this embodiment and embodiment 6 is that: the label of the instance edge mask is processed by an image smoothing filter, and then the loss is calculated by comparing the label of the smoothed instance edge mask with the predicted mask.

[0102] In this embodiment, to suppress the jagged edges of the track segmentation mask output by the network during inference, we add an image smoothing filter to the predicted mask position to suppress this jagged edge phenomenon. During network training and parameter optimization, we use the image smoothing filter to process the interpolated track mask labels, and then calculate the loss between the smoothed track mask and the predicted mask. It is worth noting that since the smoothing filter operates during network training and parameter optimization, it does not affect the network's inference speed. However, the edge smoothing process causes some information loss in the real mask labels, which slightly affects the segmentation accuracy of the algorithm.

[0103] Experimental data

[0104] To verify the effectiveness of the improved method, all hyperparameter settings were kept consistent with those in the Yolact model during GroupYolact training. Images were resized to 800×800 before being input into the network because the images in the Cityscapes dataset are relatively large. Table 1 shows the comparison results of the improved algorithm and the original algorithm on the Citscapes dataset. We can clearly see that the improved algorithm performs better on AP. Without using pre-trained weights, the GroupYolact algorithm outperforms the original method in all categories, indicating that the prototype grouping and recombining method proposed in GroupYolact is better and has stronger generalization ability than the direct combination of prototypes in Yolact. Although the prototype mask group coefficient prediction branch added during the algorithm improvement process reduces the model's computation speed, the experimental results suggest that this compromise is worthwhile.

[0105]

[0106] Table 1 Comparison of experimental results between Yolact and GroupYolact

[0107] Table 2 shows the performance of three different algorithms on the Cityscapes dataset. The overall model showed a significant improvement in AP after incorporating edge information. Compared to the original Yolact model, the AP improved by 4.4 points after incorporating edge information. Most importantly, this performance improvement did not incur any additional speed loss because the edge information was added during training, not during network inference. Table 4-2 reveals an interesting phenomenon: when the same trained model performs well, its inference speed slightly increases. We analyze this phenomenon as follows: when the overall model optimization is better, some useless weights in the network may be discarded. For example, a 3×3 convolutional kernel may become sparser, thus slightly improving the inference speed. To compare the impact of different edge detection operators on the overall edge information extraction effect, we conducted comparative experiments on the Sobel operator and the Laplacian operator. Table 2 shows that the two operators have similar effects on the model. To compare the differences between different backbone feature extraction networks, we also conducted comparative experiments between ResNet50 and ResNet101. The results in Table 2 show that when ResNet101 is used as the backbone feature extraction network, the model performs better on small instances, but worse on larger instances such as trains. We believe this may be related to the choice of input features for the prototype mask network. Table 2 also shows that all models perform poorly on the rider category. We believe this is because rider instances are small and have complex outlines. We found that the AP (Average Performance) of the ResNet101 model is actually not very good. This may be because the Cityscapes dataset is relatively small, and the number of instances for each category is not very large.

[0108]

[0109] Table 2 Comparison of experimental results for the three algorithms

[0110] Table 3 shows that the network model trained by our proposed algorithm has a significant speed advantage compared to the other two-stage algorithms, and it has great value for engineering applications with simple scenarios and certain requirements for network inference speed.

[0111]

[0112] Table 3 Performance Comparison of Different Algorithms

Claims

1. A single-stage instance segmentation method based on deep learning, characterized in that, Includes the following steps: S1: Based on the YOLACT method, instance segmentation is divided into two parallel tasks: one task is to generate the prototype mask of the current image, and the other task is to predict the mask coefficients of each instance. The prototype mask is generated as follows: k prototype masks are predicted for the entire image through the prototype mask prediction branch in YOLACT. The prototype mask sub-network protonet is implemented as FCN. The last layer of the FCN network has k channels, each channel representing a prototype mask. The input of protonet is the feature layer of the backbone network or the output of the feature layer after the feature fusion network. The mask coefficients are generated as follows: The prediction head of the anchor-based target detector in YOLACT has two branches: One branch predicts the classification confidence of c classes for each anchor box, where c represents the number of classes of the target in a specific dataset; The other branch predicts four regression box coefficients for each anchor box; By adding a third branch to predict the coefficients of k prototype masks, with each prototype mask coefficient corresponding to a prototype mask, Yolac expands the detector's prediction header output from 4+c prediction coefficients to 4+c+k prediction coefficients. The prototype mask has a base of 8, denoted as symbol b, where b is an adjustable hyperparameter. The k=32 prototype masks generated by the prototype mask prediction branch in the YOLACT structure are split along the channel dimension. Each prototype mask group has a corresponding prototype mask prediction coefficient that will be split into g groups from the channel dimension. Based on the Yolact prototype mask coefficient prediction branch, a fourth branch is added in parallel to predict the group coefficients of the prototype mask, expanding the output of the prediction head of the Yolact detector from 4+c+k predicted coefficients to 4+c+k+g predicted coefficients. S2: Perform a linear combination of the prototype mask and mask coefficients, and activate the combination result using an activation function to obtain the prediction mask; The linear combination method is as follows: The k prototype masks are ordered into groups of b in each channel dimension, and the k prototype mask coefficients are also ordered into groups of b in each channel dimension, to obtain inputs Pi and Ci respectively. By combining Pi and Ci, the intermediate prototype mask generated after the combination of each group is obtained; The group coefficient Gi of each group is obtained from the channel dimension of the coefficients of the g prototype mask groups, which is then used as the input for the second fusion process. The fusion result of each group is obtained by combining the intermediate prototype mask and the prototype mask group coefficients; The entire fusion process can be defined by summing the fusion results of each group and generating a prediction mask after activation by the nonlinear function Sigmoid: ； Where σ is the Sigmoid activation function, Pi is an h×w×b matrix for the prototype mask of each group, Ci is n instances retained after NMS and thresholding, the prototype mask coefficients in each group are an n×b matrix, and Gi is an n×1 matrix for the group coefficients of the prototype mask of the n instances. S3: Use the predicted mask as the input to the edge information prediction branch, apply the edge operator to the predicted mask to obtain the prediction result of the instance edge mask, and apply the edge operator to the real mask of the instance to obtain the instance edge mask label. S4: Add a prediction instance edge map branch. The head input of the edge map prediction branch is the prediction result and the matching instance edge mask label. Convolution is performed by the convolution operator to obtain the prediction edge map and the label of the corresponding edge map. Construct the loss function and optimize the edge loss to obtain the final model. S5: Input the instance to be segmented into the final model to obtain the segmentation result.

2. The single-stage instance segmentation method based on deep learning according to claim 1, characterized in that, The method for obtaining instance edge mask labels in S3 is as follows: Let M be the prediction mask for instances in the image. p The mask of n instances in an image is represented as an n×w×h tensor, and the n×w×h tensor is used as the input of the edge information prediction branch. Apply the Sobel or Laplacian edge detection operator to the prediction mask M of the instance. p Then, the prediction result M of the instance edge mask is obtained. pedge ; Accordingly, the edge detection operator is applied to the real mask M of the instance. gt This will give you the label M of the instance edge mask. gtedge ; Let K be the operator for edge detection, and let M be the mask that interacts with K. mask The entire process of extracting the instance edge mask can then be defined as: M edge =K∗M mask 。 3. The single-stage instance segmentation method based on deep learning according to claim 2, characterized in that, In S4, a branch for predicting instance edge maps is added. The input to the edge map prediction head is the predicted mask map M. p and the matching real mask M gt The predicted edge map M is obtained by performing convolution operations using convolution operators respectively. pedge And the corresponding edge map label M gtedge The binary cross-entropy function is chosen to optimize the edge loss, and can be defined as: Loss=L cls +L box +L mask +L edge ; Where L cls For classification loss, L box For bounding box regression loss, L mask For the binary cross-entropy loss of the instance mask, L edge This is the binary cross-entropy loss of the edge graph.

4. The single-stage instance segmentation method based on deep learning according to claim 3, characterized in that, The labels of the instance edge masks are processed using an image smoothing filter, and then the loss is calculated by comparing the labels of the smoothed instance edge masks with the predicted mask.

Images