Real-time instance segmentation method

Abstract

The invention provides a real-time instance segmentation method, and relates to the technical field of computer vision, and the method comprises the following steps: 1, cleaning a data set; step 2, preprocessing the cleaned data set; 3, a lightweight target detection model nanoDet is established, a prototype prediction network protonet is added to one layer of a path aggregation network PAN of the nanoDet to obtain a mask prototype, the size of the mask prototype is (1, c, h, w), 1 is the number of predicted pictures, h and w are the length and width of a predicted mask respectively, and c represents the mask for predicting c channels; 4, training the model, and determining parameters of the instance segmentation model; and 5, inputting a picture, and predicting an instance segmentation result. According to the method, the instance segmentation function is added on the basis of the nanoDet model, and the real-time effect equivalent to that of the nanoDet model is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an image segmentation method. Background Art

[0002] In the field of computer vision, instance segmentation is a very important research topic. It plays an important role in supporting application technology in the fields of geographic information systems, medical imaging, autonomous driving, robotics, etc., and has very important research significance.

[0003] Object detection is to identify the content in the image and detect its location, while semantic segmentation is to classify each pixel in the image by labeling it with a category. Instance segmentation is a combination of object detection and semantic segmentation. The object is detected in the image (object detection), and then each pixel is labeled (semantic segmentation). The purpose of instance segmentation is to detect the object in the input image and assign a category label to each pixel of the object. Instance segmentation can distinguish different instances with the same foreground semantic category, which is the biggest difference between it and semantic segmentation. Compared with semantic segmentation, instance segmentation developed later, so the instance segmentation model is mainly based on deep learning technology, but it is also an important component of image segmentation.

[0004] The Chinese patent application with publication number CN111798456A discloses a training method, device, and instance segmentation method for an instance segmentation model, wherein the input data is an RGBD image, and the instance segmentation model is a YOLACT model, on the basis of which a pruning module is added. The YOLACT model is an anchor-based model, and the anchor part requires a lot of calculations. Therefore, although it can be achieved in real time on a GPU, it is difficult to ensure real time on an embedded board with limited computing power. Summary of the invention

[0005] To solve the above technical problems, the present invention proposes a real-time instance segmentation method, comprising the following steps:

[0006] Step 1: Clean the data set;

[0007] Step 2, preprocessing the cleaned data set;

[0008] Step 3: Based on the data set after data preprocessing, a lightweight target detection model nanoDet is established. A prototype prediction network proto_net is added to a layer of the path aggregation network PAN of nanoDet or a layer merged from several layers to obtain a mask prototype. The size of the mask prototype is (1, c, h, w), where 1 is the number of predicted images, h and w are the length and width of the predicted mask, respectively, and c represents the mask of predicted c channels.

[0009] Step 4: training the lightweight instance segmentation model and determining parameters of the instance segmentation model;

[0010] Step 5: Input the image into the trained lightweight instance segmentation model ， Predict instance segmentation results.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1a, sampling images in the data set. If the current image is an image without a target, filter out the current image and resample the images in the data set. If there is a target in the current image, proceed to step 1b.

[0013] Step 1b, filtering out objects in the current image whose length and width are smaller than a predetermined threshold;

[0014] Step 1c: when multiple targets are gathered in an area in the current image, the multiple targets are annotated as a clustered target with one target box in the annotation of the data set, and the clustered targets in the current image are filtered out;

[0015] Step 1d, determine whether there are any remaining targets in the current image. If so, keep the current image; if not, abandon the current image and return to step 1a to resample the image.

[0016] Furthermore, step 2 specifically includes:

[0017] Step 2a: performing at least one of mean-variance normalization and saturation transformation on the pixel values ​​of the image, and performing at least one of random cropping, random mirror transformation and size adjustment on the image;

[0018] Step 2b: performing one of random cropping, random mirroring and resizing of the real labeled data target box and the real labeled data mask of the image;

[0019] Step 2c: Filter out the target boxes whose length and width are smaller than the threshold in the image processed by step 2a, and filter out the labels corresponding to the real annotated data target boxes and the masks corresponding to the real annotated data masks processed by step 2b.

[0020] Furthermore, in step 3, the prototype prediction network includes at least a convolutional layer; a pooling layer and / or an activation layer may be selected as required.

[0021] Furthermore, in step 3, the lightweight target detection model nanoDet also includes: a backbone network and a prediction head.

[0022] Furthermore, the backbone network may select any one of the models ResNet series, ShuffleNet series, and mobileNet series for extracting multi-layer features.

[0023] Furthermore, the prediction head includes a plurality of branches corresponding to the number of feature layers extracted from the path aggregation network, and a mask branch is added to the prediction head of each branch to predict the mask coefficient, and the mask coefficient of each branch is aggregated into a mask coefficient of size (N, c); wherein N is the number of predicted targets and c is the number of channels in the mask prototype.

[0024] Furthermore, BCE loss is used as the loss function, specifically: first, the mask BCE calculated in each target box is divided by the area of ​​the corresponding labeled target box to obtain the normalized mask BCE; then the normalized mask BCE is multiplied by h*w in the mask prototype to obtain the final mask BCE.

[0025] Furthermore, in step 4, an auxiliary training module is used to improve training accuracy and accelerate convergence.

[0026] Furthermore, the step 5 is specifically as follows:

[0027] Step 5a: Input the image into the trained lightweight instance segmentation model to obtain the target box, target category, mask prototype and mask coefficient;

[0028] Step 5b: The mask prototype of each target is multiplied by the mask coefficient to obtain a mask value of h*w, and then the obtained mask value is processed by the logistic regression function to obtain the processed mask value; a threshold is set, and the mask value part greater than the threshold is set as the foreground, otherwise it is the background;

[0029] Step 5c: According to the determined coordinates of the target box in the foreground, the corresponding part of the mask is taken out to obtain an instance.

[0030] The nanoDet model is a lightweight target detection model with a frame rate of up to 97fps on an ARM CPU, which can meet real-time requirements. The present invention adds an instance segmentation function based on the nanoDet model to achieve a real-time effect comparable to that of the nanoDet model.

[0031] The present invention adds a branch during the training process, which can improve the training effect and accelerate convergence. The branch is ignored during the prediction process, so as to achieve the purpose of not affecting the timeliness of the prediction part. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for real-time instance segmentation according to the present invention;

[0033] Figure 2 It is an example segmentation model diagram of the present invention;

[0034] Figure 3 An example structural diagram of a prototype prediction network (proto_net) in an instance segmentation model of the present invention;

[0035] Figure 4 is an example diagram of a mask prototype (proto_type) of the present invention;

[0036] Figure 5 is an example structural diagram of a mask branch in an instance segmentation model of the present invention;

[0037] Figure 6 is a schematic diagram of an auxiliary training module of the present invention;

[0038] Figure 7 This is an example segmentation effect diagram of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] Figure 1 A flowchart of a method for real-time instance segmentation is shown. Public datasets for instance segmentation include pattern analysis, static modeling and computational learning datasets (PASCAL VOC datasets), Microsoft common object datasets (MS COCO datasets), semantic understanding image datasets of urban street scenes (Cityscapes datasets), and 20K-level dense annotation datasets (ADE20K datasets). The present invention uses the COCO dataset to train the model. The COCO dataset is a large and rich dataset for object detection, segmentation and image description, in which there are 80 categories of semantic segmentation, and the images are 3-channel color images with different sizes.

[0041] like Figure 1 As shown, a real-time instance segmentation method of the present invention comprises the following steps:

[0042] Step 1: Clean the data in the dataset.

[0043] Step 1a, sample the images in the data set. If the current image is a picture without a target, filter out the current image and resample the images in the data set. If there is a target in the current image, go to step 1b.

[0044] The COCO dataset contains pictures without objects. The annotations of these pictures are empty. These pictures are filtered out, that is, pictures with empty annotations are resampled when they are encountered.

[0045] Step 1b: Filter out objects in the current image whose length and width are smaller than a predetermined threshold.

[0046] Set a threshold. When the length and width of the target are less than the threshold, filter out the target. After filtering, if there are no remaining targets in the image, resample the image. The threshold can be set to 5 pixels, for example, and of course the threshold can be modified.

[0047] Step 1c: when multiple targets are gathered in an area in the current image, the multiple targets are annotated as a clustered target with one target box in the annotation of the data set, and the clustered targets in the current image are filtered out;

[0048] Step 1d, determine whether there are any remaining targets in the current image. If so, keep the current image; if not, abandon the current image and return to step 1a to resample the image.

[0049] Step 2: Preprocess the data in the cleaned dataset.

[0050] Data preprocessing includes image normalization, random tone adjustment, random cropping, random mirror transformation, resizing, etc.

[0051] Step 2a: performing at least one of mean-variance normalization and saturation transformation on the pixel values ​​of the image, and performing at least one of random cropping, random mirror transformation and size adjustment on the image.

[0052] Normalization can be achieved by dividing the pixel value by 255, and mean-variance normalization can be achieved by subtracting the pixel mean and then dividing by the standard deviation.

[0053] For example, in the present invention, the size of the original picture is adjusted to 416x416.

[0054] Step 2b: Perform one of random cropping, random mirroring and resizing on the ground truth target box and the ground truth mask of the image.

[0055] Step 2c: Filter out the target frames whose length and width are less than the threshold in the image processed by step 2a, and filter out the labels (labels) corresponding to the real annotated data target frames and the masks (masks) corresponding to the real annotated data masks processed by step 2b. The threshold can be set to 5 pixels, for example, and of course the threshold can be modified.

[0056] The target may not be that small at the beginning. After scaling or flipping, part of the target goes out of the image boundary. The target within the effective range is already very small and needs to be filtered again.

[0057] Step 3: Add instance segmentation function to the nanoDet model to implement the instance segmentation model.

[0058] Figure 2 The example segmentation model of the present invention is shown. The present invention adds an example segmentation function based on the nanoDet model to achieve a real-time effect comparable to that of the nanoDet model.

[0059] Step 3a: Model Design

[0060] The nanoDet model is a lightweight target detection model with a frame rate of up to 97fps on an ARM CPU, which can meet real-time requirements. Figure 2 As shown, the nanoDet model mainly includes:

[0061] The backbone network of the nanoDet model can be selected from the ResNet series, ShuffleNet series, mobileNet series, etc. Any model that can extract multi-layer (multi-dimensional) features can be used. The present invention selects the ShuffleNetV2 model.

[0062] The backbone extracts features of multiple dimensions of the image and selects features of three layers of different dimensions as the input of the path aggregation network (PAN). The PAN structure adds a bottom-up path on the basis of the feature pyramid network (FPN). It has both top-down and bottom-up paths, so it is also possible to use FPN here. The present invention adds a prototype prediction network (proto_net) to the bottom layer of the PAN module (which can be a combination of other layers or multiple layers). The structural diagram of the prototype prediction network is shown in Figure 3 , Figure 3This is a rough structure. The actual prototype prediction network can contain any number of 3*3 convolution modules and any number of 1*1 convolution modules. Among them, the 3*3 convolution module contains a convolution layer with a convolution kernel size of 3, padding of 1 and an activation function (which can be ReLU or SiLU or other activation functions), and a pooling layer can also be added. Similarly, the 1*1 convolution module contains a 1*1 convolution layer and an activation function, and a pooling layer can also be added. In the present invention, two 3*3 convolution modules and two 1*1 convolution modules are used, and SiLU is the activation function. The output of the prototype prediction network is the mask prototype (proto_type), and the size of the mask prototype is (1, c, h, w), where 1 is the number of pictures, h and w are the length and width of the predicted mask, respectively, and c represents the predicted mask prototype of c channels. The mask prototype of each channel emphasizes different objects (see Figure 4 ), the mask prototypes of c channels are superimposed together (the weight of each channel during superposition is the mask coefficient predicted by the prediction branch mask branch) to obtain the target mask. The number of channels c and h, w can be defined arbitrarily. In the present invention, c is 32, and h and w are both 104.

[0063] The original prediction heads in the nanoDet model have 4 branches, which predict the target category and target box coordinates under different dimensional features of PAN output, but do not predict the target mask. The present invention adds a mask branch to the prediction head of the 4 branches to predict the mask coefficients. The mask coefficients of the 4 branches are finally summarized into a mask coefficient of size (N, c). N is the number of predicted targets, and c is the number of channels in proto_type. The N targets and mask coefficients will be partially filtered out in the non-maximum suppression (NMS) module of the nanoDet model. Finally, each target has a mask coefficient of (1, c), which is the weight of the c channels of proto_type. When calculating the mask of each target, the mask coefficient is multiplied by the proto_type matrix, which is equivalent to multiplying the proto_type of each channel by a weight and superimposing them to obtain the mask of the target. The newly added module in the nanoDet model in the present invention is Figure 2 The structure of maskbranch can be seen in Figure 5 , which can include any number of 3*3 convolution modules and any number of 1*1 convolutions (1*1 convolutions can also be followed by a pooling layer and an activation function). In the present invention, the mask branch includes two 3*3 convolution modules (including a 3*3 convolution layer, a pooling layer, and an activation function) and one 1*1 convolution. The number of output channels of the last 1*1 convolution is c.

[0064] Step 3b: Design of Mask loss function for training process.

[0065] The loss function uses BCE loss, which calculates the binary cross entropy (BCE) of the mask in the target box, that is, the number of pixels in the target box where the predicted mask and the annotated mask have different values ​​(the annotated mask needs to be scaled to the same size as the predicted mask).

[0066] The BCE loss function is a commonly used loss function. It is a loss function for a binary classification problem. In deep learning, the input data is usually divided into two categories.

[0067] However, directly calculating the mask BCE will face a problem, that is, the larger the target box, the more pixels with different values ​​may be inside, and the greater the loss; conversely, the smaller the target, the smaller the loss may be, which may cause the model to discard large targets. Therefore, the mask BCE is normalized. The annotated target box can best reflect the actual target box size. Therefore, the mask BCE calculated in each target box is divided by the area of ​​the corresponding annotated target box. This eliminates the impact of the target box size on BCE. However, the loss calculated in this way is the normalized loss, which needs to be denormalized. Multiply the normalized mask BCE by the predicted mask area (h*w in proto_type) to get the final mask BCE. That is:

[0068] Mask BCE loss = BCE / annotated target box area * mask area in proto_type.

[0069] Step 4: Train the model and determine the parameters of the target detection model nanoDet.

[0070] Figure 6 FIG. 2 shows a schematic diagram of an auxiliary training module of the present invention. Figure 6As shown, the nanoDet model has an auxiliary training module during training, which uses a module different from the prediction heads to predict the target category and target frame coordinates, and calculates the loss separately. In this way, there is one more branch to calculate the loss, and the gradient will be calculated twice in one training, which accelerates convergence. In addition, there are two branches to train at the same time, which is more accurate than one branch training. The present invention also adopts this strategy when training the mask branch, defining two branches mask branch1 and maskbranch2 (their structures are different) with different structures from the mask branch, and predicting half of the number of channels respectively (for example, the mask coefficient is finally (N, c), then maskbranch1 and maskbranch2 predict (N, c / 2), (N, c / 2), or other ratios, as long as their results are combined to (N, c). Where N is the number of target frames, and c is the number of channels of proto_type). In this way, having one more module to train will accelerate convergence. This auxiliary training module is only used during training and is not used in model reasoning.

[0071] The present invention performs training according to the parameters in the nanoDet model, and the parameters can be adjusted.

[0072] Step 5: Input the image and predict the instance segmentation result.

[0073] Step 5a: Preprocess the image and input it into the network to obtain the target box, target category, proto_type and mask coefficient.

[0074] Step 5b: The proto_type of each target is multiplied by the mask coefficient to obtain a mask value of h*w, and then the obtained mask value is processed by the logistic regression function (sigmoid function), and the processed mask value is between 0 and 1. A threshold is set, and the value of the mask greater than the threshold is set as the foreground, otherwise it is the background. In the present invention, the threshold is set to 0.5.

[0075] Step 5c: Take out the corresponding part of the mask according to the coordinates of the target box to get the instance.

[0076] Figure 7 An example segmentation effect diagram of the present invention is shown.

[0077] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A real-time instance segmentation method, characterized in that: The method comprises the following steps: Step 1: Clean the data set; Step 2, preprocessing the cleaned data set; Step 3: Based on the data set after data preprocessing, a lightweight target detection model nanoDet is established. A prototype prediction network proto_net is added to a layer of the path aggregation network PAN of nanoDet or a layer merged from several layers to obtain a mask prototype. The size of the mask prototype is (1, c, h, w), where 1 is the number of predicted images, h and w are the length and width of the predicted mask, respectively, and c represents the mask of predicted c channels. Step 4: training the lightweight instance segmentation model and determining parameters of the instance segmentation model; Step 5: Input the image into the trained lightweight instance segmentation model ， Predict instance segmentation results.

2. The method for real-time instance segmentation according to claim 1, characterized in that: Step 1 specifically includes: Step 1a, sampling images in the data set. If the current image is an image without a target, filter out the current image and resample the images in the data set. If there is a target in the current image, proceed to step 1b. Step 1b, filtering out objects in the current image whose length and width are smaller than a predetermined threshold; Step 1c: when multiple targets are gathered in an area in the current image, the multiple targets are annotated as a clustered target with one target box in the annotation of the data set, and the clustered targets in the current image are filtered out; Step 1d, determine whether there are any remaining targets in the current image. If so, keep the current image; if not, abandon the current image and return to step 1a to resample the image.

3. The method for real-time instance segmentation according to claim 1 or 2, characterized in that: Step 2 specifically includes: Step 2a: performing at least one of mean-variance normalization and saturation transformation on the pixel values ​​of the image, and performing at least one of random cropping, random mirror transformation and size adjustment on the image; Step 2b: performing one of random cropping, random mirroring and resizing of the real labeled data target box and the real labeled data mask of the image; Step 2c: Filter out the target boxes whose length and width are smaller than the threshold in the image processed by step 2a, and filter out the labels corresponding to the real annotated data target boxes and the masks corresponding to the real annotated data masks processed by step 2b.

4. The method for real-time instance segmentation according to claim 1, characterized in that: In step 3, the prototype prediction network includes at least a convolutional layer; a pooling layer and / or an activation layer may be selected as needed.

5. The method for real-time instance segmentation according to claim 1, characterized in that: In step 3, the lightweight object detection model nanoDet also includes: a backbone network and a prediction head.

6. The method for real-time instance segmentation according to claim 5, characterized in that: The backbone network can select any one of the ResNet series, ShuffleNet series, and mobileNet series models that extract multi-layer features.

7. The method for real-time instance segmentation according to claim 5, characterized in that: The prediction head includes multiple branches corresponding to the number of feature layers extracted from the path aggregation network. A mask branch is added to the prediction head of each branch to predict the mask coefficient. The mask coefficient of each branch is aggregated into a mask coefficient of size (N, c); where N is the number of predicted targets and c is the number of channels in the mask prototype.

8. The method for real-time instance segmentation according to claim 7, characterized in that: BCE loss is used as the loss function. Specifically, the mask BCE calculated in each target box is divided by the area of ​​the corresponding labeled target box to obtain the normalized mask BCE; then the normalized mask BCE is multiplied by h*w in the mask prototype to obtain the final mask BCE.

9. The method for real-time instance segmentation according to claim 1, characterized in that: In step 4, an auxiliary training module is used to improve training accuracy and accelerate convergence.

10. The method for real-time instance segmentation according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5a: Input the image into the trained lightweight instance segmentation model to obtain the target box, target category, mask prototype and mask coefficient; Step 5b: The mask prototype and mask coefficient of each target are multiplied to obtain a mask value of h*w, and then the obtained mask value is processed by the logistic regression function to obtain the processed mask value; Set a threshold, the mask value part greater than the threshold is set as the foreground, otherwise it is the background; Step 5c: According to the determined coordinates of the target box in the foreground, the corresponding part of the mask is taken out to obtain an instance.

Images