A method of instance segmentation based on set matching

Abstract

The application discloses an instance segmentation method based on set matching, and specifically comprises the following steps: converting a label mask into a label instance set in the image label mask in an image set; constructing an instance segmentation model; inputting the image of the label mask into the instance segmentation model and training by using a gradient descent method until a cross-entropy loss function converges, thereby completing the training of the instance segmentation model; and inputting an image to be detected into the trained instance segmentation model to predict an instance segmentation result. The application uses a set matching algorithm to pair elements of a predicted instance mask set and a label instance set according to a consistency evaluation index, thereby improving the accuracy and robustness of instance matching.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology in computer vision, specifically, it relates to an instance segmentation method based on set matching. Background Technology

[0002] Image segmentation is subdivided into semantic segmentation and instance segmentation according to the task. Currently, the mainstream solutions for instance segmentation require object detection in the image before pixel-level fine classification, such as Mask-R-CNN and Cascade Mask R-CNN. However, the object detection steps in these solutions are all "two-stage" object detection, and they all require anchor mechanisms, non-maximum suppression (NMS), object box detection and regression, which are relatively complex in structure and slow inference speed.

[0003] While recent innovative methods have improved upon the "two-stage" approach by moving to a "one-instruction-period" approach (as described in the 20230707 document), reducing inference time, they still require bounding boxes, anchor point mechanisms, and NMS as pre-processing steps for predicting instance masks. Currently, methods using instance segmentation in computer vision include:

[0004] (1) Using object detection models such as Faster-RCNN and FCOS to first detect the bounding boxes of the instance, and then segment the bounding boxes into foreground and background, is complex, has a slow inference speed, and can only learn local information of the instance, but cannot learn global information, thus leading to a decrease in segmentation accuracy.

[0005] (2) The post-processing of the output mask for semantic segmentation tasks has a low overall complexity, but practical results show that the accuracy is low in scenarios with a large number of instance targets because the clustering results are related to the number of categories. The multi-instance segmentation effect is poor, and the overall accuracy is low. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides an instance segmentation method based on set matching. Set matching occurs between the instance label mask set and the predicted instance mask during the training of the instance segmentation model. This invention can achieve a one-to-one pairing of instance labels and predicted instance masks through set matching, thereby enabling the model to avoid the problems of excessive algorithm complexity and slow inference speed that exist in the scheme of performing object detection first and then instance segmentation. At the same time, it can also solve the problems of the inability to support multi-instance segmentation and low segmentation accuracy in the scheme of performing semantic segmentation first and then clustering.

[0007] To achieve the above technical objectives, the present invention adopts the following technical solution: an instance segmentation method based on set matching, specifically including the following steps:

[0008] Step S1: In the image set, the image annotation mask is converted into a set of annotation instances;

[0009] Step S2: Construct the instance segmentation model;

[0010] Step S3: Input the image with the labeled mask into the instance segmentation model and train it using the gradient descent method until the cross-entropy loss function converges, thus completing the training of the instance segmentation model.

[0011] Step S4: Input the image to be detected into the trained instance segmentation model to predict the instance segmentation result.

[0012] Furthermore, the set of labeled instances S t ={y1,...,y j ,...,y n ,...y N}, where n represents the number of corresponding target instances in the image, y j Let y represent the binary image of the j-th target instance in the labeled instance set, where N represents the length of the labeled instance set, and y represents the value of y. n+1 ,...y N Both represent empty masks.

[0013] Furthermore, the process of obtaining the binary image is as follows: based on each labeled mask, a binary image with a white target instance and a black background is obtained.

[0014] Furthermore, the instance segmentation model includes a fully convolutional backbone network, a mask output convolutional network, and a set matching network connected in sequence. The fully convolutional backbone network is used to extract feature maps from the image, the mask output convolutional network is used to predict the instance mask set, and the set matching network is used to pair the elements of the predicted instance mask set and the labeled instance set using a consistency evaluation index.

[0015] Furthermore, the fully convolutional backbone network is one of ENet, UNet, or Segnet.

[0016] Furthermore, the number of output channels and the set of labeled instances S of the masked output convolutional network... t The number of elements in the middle is the same.

[0017] Furthermore, step S3 includes the following sub-steps:

[0018] Step S3.1: Input one image from the image set into the fully convolutional backbone network to obtain the feature map;

[0019] Step S3.2: Input the feature map into the mask output convolutional network to obtain the set of predicted instance masks;

[0020] Step S3.3: In the set matching network, the set matching algorithm is used to pair the elements of the predicted instance mask set and the labeled instance set according to the consistency evaluation index.

[0021] Step S3.4: Repeat steps S3.1-S3.3 using gradient descent until the cross-entropy loss function converges, thus completing the training of the instance segmentation model.

[0022] Furthermore, the specific process of step S3.3 is as follows:

[0023] Step S3.3.1: Set the annotation instance set S p initial label and the set of predicted instance masks S t The initial label l(y) j ) = 0, where e ij S represents p The i-th element x i With S t The j-th element y j Consistency evaluation indicators;

[0024] Step S3.3.2: Based on edge set E l ={x i y j |l(x i )+l(y j ) = e ij}, Subgraph G l =(S p ,S tk E l Find a matching M, M∈E, where no two edges in matching M are adjacent, and E represents the set of predicted instance masks S. t and the set of labeled instances S p The edge set formed by the consistency evaluation index among the elements;

[0025] Step S3.3.3: If the set of annotated instances S p If all elements in the output are connected to the matching M, then output the matching M; otherwise, proceed to step S3.3.4.

[0026] Step S3.3.4: In the annotation instance set S p Find an element x0 that is not connected to the matching M, and let A ← {x0}. in, Indicates the empty set;

[0027] Step S3.3.5: If the set of nodes adjacent to nodes in A... and Then update the set of labeled instances S pThe label and the set of predicted instance masks S t Label: Where v represents x i or y j , a represents in S t Except for the current G l The minimum weight among the existing nodes adjacent to node A that is connected to node A. Then update the matching M through step S3.3.2; otherwise, proceed to step S3.3.6.

[0028] Step S3.3.6, Take If y is connected to matching M, find the matching point x of y in M, let A←A∪{x}, B←B∪{y}, and execute step S3.3.5; otherwise, execute step S3.3.7.

[0029] Step S3.3.7: Find the path in E that starts at x0 and ends at y, and the edges of this path appear alternately and do not appear in M, to obtain the augmenting path P from x0 to y. Let M←M+E(P) to expand the matching M and execute step S3.3.3, where E(P) represents the sub-matching formed from x0 to y through the augmenting path P.

[0030] Furthermore, the S p The i-th element x i With S t The j-th element y j Consistency evaluation index e ij for:

[0031]

[0032] Furthermore, the cross-entropy loss function is:

[0033]

[0034] Where N represents the length of the labeled instance set, W represents the width of the image, and H represents the height of the image. S represents the set of predicted instance masks for matching. t The predicted probability value of the nth element with coordinates (w, h). S represents the set of matched labeled instances. p The probability value that the coordinates of the nth element are (w, h).

[0035] Compared with the prior art, the present invention has the following advantages: The instance segmentation method based on set matching of the present invention does not require the step of predicting candidate boxes, but directly predicts the segmentation instances from the feature map. Therefore, it does not require anchor point mechanism and NMS to remove redundant candidate boxes. The process is simple and the computational complexity is low. The present invention uses set matching algorithm to pair the elements of the predicted instance mask set and the labeled instance set according to the consistency evaluation index, which improves the accuracy and robustness of instance matching. Attached Figure Description

[0036] Figure 1 This is a flowchart of the instance segmentation method based on set matching of the present invention;

[0037] Figure 2 This is a schematic diagram illustrating the training of the instance segmentation model in this invention. Detailed Implementation

[0038] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0039] like Figure 1 This is a flowchart of the instance segmentation method based on set matching of the present invention. The instance segmentation method specifically includes the following steps:

[0040] Step S1: The image annotation mask in the image set is converted into a set of annotation instances; in this invention, the annotation instance set S... t ={y1,...,y j ,...,y n ,...y N}, where n represents the number of corresponding target instances in the image, y j Let y represent the binary image of the j-th target instance in the labeled instance set. j The dimensions are (H, W), where H and W are the height and width of the image, respectively. N represents the length of the labeled instance set, y n+1 ,...y N Both indicate that in order to complete set S t An empty mask is added to increase the set length to N, and each empty mask contains no instance foreground information. The process of obtaining the binary image in this invention is as follows: based on each labeled mask, a binary image is obtained in which the target instance is white and the background is black.

[0041] Step S2: Construct an instance segmentation model. Specifically, the instance segmentation model includes a fully convolutional backbone network, a masked output convolutional network, and an set matching network connected in sequence. The fully convolutional backbone network is used to extract feature maps from the image. The fully convolutional backbone network is one of ENet, UNet, or SegNet and should include downsampling and upsampling operations. The masked output convolutional network is used to predict the instance mask set. The number of output channels of the masked output convolutional network is also N, such that the output predicted instance mask set S... p The number of elements and the set of labeled instances S t The number of elements in S is the same. p ={x1,...,x i ,...x N}, x i The dimensions are (H, W), where H and W are the height and width of the image, respectively. Set matching networks are used to predict the pairing of elements between the instance mask set and the labeled instance set based on consistency evaluation metrics.

[0042] Step S3: Input the image with the labeled mask into the instance segmentation model and train it using gradient descent until the cross-entropy loss function converges, thus completing the training of the instance segmentation model; specifically, as follows... Figure 2 This includes the following sub-steps:

[0043] Step S3.1: Input one image from the image set into the fully convolutional backbone network to obtain the feature map;

[0044] Step S3.2: Input the feature map into the mask output convolutional network to obtain the set of predicted instance masks;

[0045] Step S3.3: In the set matching network, the set matching algorithm is used to pair elements of the predicted instance mask set and the labeled instance set according to the consistency evaluation index. This avoids the problems of excessive algorithm complexity and slow inference speed caused by the instance segmentation model performing object detection before instance segmentation. It also solves the problems of poor support for multi-instance segmentation and low segmentation accuracy in the semantic segmentation-then-clustering scheme. Specifically, it includes the following sub-steps:

[0046] Step S3.3.1: Set the annotation instance set S p initial label and the set of predicted instance masks S t The initial label l(y) j ) = 0, where e ij S represents p The i-th element x i With S t The j-th element y j Consistency evaluation indicators

[0047] Step S3.3.2: Based on edge set E l ={x i y j |l(x i )+l(y j ) = e ij}, Subgraph G l =(S p ,S tk E l Find a matching M, M∈E, where no two edges in matching M are adjacent, and E represents the set of predicted instance masks S. t and the set of labeled instances S p The edge set formed by the consistency evaluation index among the elements;

[0048] Step S3.3.3: If the set of annotated instances S p If all elements in the output are connected to the matching M, then output the matching M; otherwise, proceed to step S3.3.4.

[0049] Step S3.3.4: In the annotation instance set S p Find an element x0 that is not connected to the matching M, and let A ← {x0}. in, Indicates the empty set;

[0050] Step S3.3.5: If the set of nodes adjacent to nodes in A... and Then update the set of labeled instances S p The label and the set of predicted instance masks S t Label: Where v represents x i or y j , a represents in S t Except for the current G l The minimum weight among the existing nodes adjacent to node A that is connected to node A. Then update the matching M through step S3.3.2; otherwise, proceed to step S3.3.6.

[0051] Step S3.3.6, Take If y is connected to matching M, find the matching point x of y in M, let A←A∪{x}, B←B∪{y}, and execute step S3.3.5; otherwise, execute step S3.3.7.

[0052] Step S3.3.7: Find the path in E that starts at x0 and ends at y, and the edges of this path appear alternately and do not appear in M, to obtain the augmenting path P from x0 to y. Let M←M+E(P) to expand the matching M and execute step S3.3.3, where E(P) represents the sub-matching formed from x0 to y through the augmenting path P.

[0053] Step S3.4: Repeat steps S3.1-S3.3 using gradient descent until the cross-entropy loss function converges, completing the training of the instance segmentation model; the cross-entropy loss function in this invention is:

[0054]

[0055] Where N represents the length of the labeled instance set, W represents the width of the image, and H represents the height of the image. S represents the set of predicted instance masks for matching. t The predicted probability value of the nth element with coordinates (w, h). S represents the set of matched labeled instances. p The true value of the nth element with coordinates (w, h).

[0056] Step S4: Input the image to be detected into the trained instance segmentation model to predict the instance segmentation result.

[0057] Compared to traditional methods that first detect bounding boxes and then perform segmentation, this invention eliminates complex processes such as predicting candidate boxes, anchoring mechanisms, and Non-Maximum Segmentation (NMS), resulting in lower computational complexity. As the technical solution shows, this method does not require the step of predicting candidate boxes; it directly predicts segmentation instances from the feature map. Therefore, it also eliminates the need for anchoring mechanisms and NMS to remove redundant candidate boxes, simplifying the process and reducing computational complexity. Compared to other post-processing methods that output masks for semantic segmentation tasks, existing technologies are based on clustering, which is an unsupervised approach to instance matching. In contrast, the matching method proposed in this invention introduces a consistency evaluation metric, significantly improving the accuracy of instance matching and demonstrating high robustness.

[0058] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An instance segmentation method based on set matching, characterized in that, Specifically, the steps include the following: Step S1: In the image set, the image annotation mask is converted into a set of annotation instances; Step S2: Construct an instance segmentation model; the instance segmentation model includes a fully convolutional backbone network, a mask output convolutional network, and a set matching network connected in sequence. The fully convolutional backbone network is used to extract feature maps on the image, the mask output convolutional network is used to predict the instance mask set, and the set matching network is used to pair the elements of the predicted instance mask set and the labeled instance set with a consistency evaluation index. Step S3: Input the image with the labeled mask into the instance segmentation model and train it using gradient descent until the cross-entropy loss function converges, thus completing the training of the instance segmentation model; including the following sub-steps: Step S3.1: Input one image from the image set into the fully convolutional backbone network to obtain the feature map; Step S3.2: Input the feature map into the mask output convolutional network to obtain the set of predicted instance masks; Step S3.3: In the set matching network, the set matching algorithm is used to pair the elements of the predicted instance mask set and the labeled instance set according to the consistency evaluation index; the specific process is as follows: Step S3.3.1: Set the annotation instance set S p initial label and the set of predicted instance masks S t The initial label l(y) j ) = 0, where e ij S represents p The i-th element x i With S t The j-th element y j Consistency evaluation indicators; Step S3.3.2: Based on edge set E l ={x i y j |l(x i )+l(y j ) = e ij }, Subgraph G l =(S p ,S tk E l Find a matching M, M∈E, where no two edges in matching M are adjacent, and E represents the set of predicted instance masks S. t and the set of labeled instances S p The edge set formed by the consistency evaluation index among the elements; Step S3.3.3: If the set of annotated instances S p If all elements in the output are connected to the matching M, then output the matching M; otherwise, proceed to step S3.3.

4. Step S3.3.4: In the annotation instance set S p Find an element x0 that is not connected to the matching M, and let A ← {x0}. in, Represents the empty set; Step S3.3.5: If the set of nodes adjacent to nodes in A... and Then update the set of labeled instances S p The label and the set of predicted instance masks S t Label: Where v represents x i or y j , a represents in S t Except for the current G l The minimum weight among the existing nodes adjacent to node A that is connected to node A. Then update the matching M through step S3.3.2; otherwise, proceed to step S3.3.

6. Step S3.3.6, Take If y is connected to matching M, find the matching point x of y in M, let A←A∪{x}, B←B∪{y}, and execute step S3.3.5; otherwise, execute step S3.3.

7. Step S3.3.7: Find the path in E that starts at x0 and ends at y, and the edges of this path appear alternately and do not appear in M, to obtain the augmenting path P from x0 to y. Let M←M+E(P) to expand the matching M and execute step S3.3.3, where E(P) represents the sub-matching formed from x0 to y through the augmenting path P. Step S3.4: Repeat steps S3.1-S3.3 using gradient descent until the cross-entropy loss function converges, thus completing the training of the instance segmentation model; The cross-entropy loss function is: Where N represents the length of the labeled instance set, W represents the width of the image, and H represents the height of the image. S represents the set of predicted instance masks for matching. t The predicted probability value of the nth element with coordinates (w, h). S represents the set of matched labeled instances. p The probability that the coordinates of the nth element are (w, h); Step S4: Input the image to be detected into the trained instance segmentation model to predict the instance segmentation result.

2. The instance segmentation method based on set matching according to claim 1, characterized in that, The set of annotation instances S t ={y1,...,y j ,...,y n ,...y N }, where n represents the number of corresponding target instances in the image, y j Let y represent the binary image of the j-th target instance in the labeled instance set, where N represents the length of the labeled instance set, and y represents the value of y. n+1 ,...y N Both represent empty masks.

3. The instance segmentation method based on set matching according to claim 2, characterized in that, The process of obtaining the binary image is as follows: based on each labeled mask, a binary image with a white target instance and a black background is obtained.

4. The instance segmentation method based on set matching according to claim 1, characterized in that, The fully convolutional backbone network is one of ENet, UNet, or Segnet.

5. The instance segmentation method based on set matching according to claim 1, characterized in that, The number of output channels and the set of labeled instances S of the masked output convolutional network t The number of elements in the middle is the same.

6. The instance segmentation method based on set matching according to claim 1, characterized in that, The S p The i-th element x i With S t The j-th element y j Consistency evaluation index e ij for:

Images