Instance segmentation methods, apparatus, electronic devices, and storage media

Abstract

This invention discloses an instance segmentation method, apparatus, electronic device, and storage medium. The instance segmentation method includes: performing object detection and instance segmentation on an image to be segmented to obtain an object detection bounding box and a first instance mask; based on the object detection bounding box, acquiring feature information from a multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, and fusing the acquired feature information to obtain multi-scale shallow and deep features; generating a second instance mask based on the multi-scale shallow and deep features; and outputting the object detection bounding box and the second instance mask. This invention, by fusing multi-scale low-resolution deep features and high-resolution low-resolution features, can better predict the mask details of instances, outputting a more refined instance mask, and improving the accuracy of image instance segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an instance segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Instance segmentation plays a crucial role in supporting applications in fields such as geographic information systems, medical imaging, autonomous driving, and robotics, and is of significant research importance. Instance segmentation combines object detection and semantic segmentation, detecting objects from images and labeling each pixel. Compared to semantic segmentation and object detection, instance segmentation is a relatively recent development, and therefore primarily relies on deep learning techniques. However, the use of deep neural networks leads to low resolution and loss of detail in the deep feature maps, resulting in poor segmentation performance, often inferior to object detection results. Summary of the Invention

[0003] This invention provides an instance segmentation method, apparatus, electronic device, and storage medium to solve the problem in the prior art that the use of deep neural networks leads to low resolution and loss of details in the target on deep feature maps, resulting in poor segmentation performance.

[0004] This invention provides an instance segmentation method, comprising:

[0005] Perform object detection and instance segmentation on the image to be segmented to obtain object detection boxes and the first instance mask;

[0006] Based on the target detection box, feature information is obtained from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, and the obtained feature information is fused to obtain multi-scale shallow and deep features;

[0007] Based on the multi-scale depth features, a second instance mask is generated;

[0008] Output the target detection bounding box and the second instance mask, the second instance mask having more mask details than the first instance mask.

[0009] In some embodiments, the step of performing object detection and instance segmentation on the image to be segmented to obtain object detection boxes and a first instance mask includes:

[0010] Based on a two-stage instance segmentation model, target detection and instance segmentation are performed on the image to be segmented, and the target detection bounding box and the first instance mask output by the two-stage instance segmentation model are obtained.

[0011] The two-stage instance segmentation model includes a backbone network module, a candidate set generation module, and a prediction module. The backbone network module is used to extract features from the image to be segmented and output a multi-scale feature map. The candidate set generation module is used to generate a candidate set based on the multi-scale feature map. The prediction module is used to perform object detection and instance segmentation based on the candidate set to obtain an object detection box and a first instance mask.

[0012] In some embodiments, the backbone network module includes: multiple backbone networks and a feature pyramid network;

[0013] Each of the backbone networks is a Swing Transformer network model containing multiple stages, and the Swing Transformer network model is used for feature extraction.

[0014] The feature pyramid network is used to connect the multiple backbone networks and output multi-scale feature maps.

[0015] In some embodiments, the step of obtaining feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask according to the target detection box, and fusing the obtained feature information to obtain multi-scale shallow and deep features includes:

[0016] The first feature information corresponding to the target detection box is extracted from each feature map of the multi-scale feature map output by the backbone network module in the two-stage instance segmentation model. The extracted first feature information is then convolved and upsampled sequentially to obtain the first feature set.

[0017] The image corresponding to the target detection box is extracted from the image to be segmented, and features are extracted from the image extracted from the image to be segmented to obtain a second feature set;

[0018] The image corresponding to the target detection box is extracted from the first instance mask, and features are extracted from the image extracted from the instance mask to obtain a third feature set;

[0019] Feature fusion is performed on the first feature set, the second feature set, and the third feature set to obtain multi-scale shallow and deep features.

[0020] In some embodiments, the generation of a second instance based on the multi-scale depth features...

[0021] The mask includes:

[0022] 0. The multi-scale shallow and deep features are input into the mask fine-tuning network for alignment and pixel-level classification.

[0023] The class obtains the instance mask of the occluder and the instance mask of the occluded object, and uses the instance mask of the occluder as the second instance mask;

[0024] The mask fine-tuning network includes an alignment module, a fully convolutional network, and two convolutional layers.

[0025] The alignment module is used to align the multi-scale shallow and deep features, and the fully convolutional 5 network and two convolutional branches are used to perform pixel-level alignment based on the aligned multi-scale shallow and deep features.

[0026] Classify the objects to obtain the instance mask of the occluded objects and the instance mask of the occluded objects.

[0027] In some embodiments, the process of determining the mask fine-tuning network includes:

[0028] Obtain multi-scale depth features of image samples and the instance mask corresponding to the image samples.

[0029] Code tags;

[0030] 0. Based on the alignment module, the multi-scale depth features of the image samples are aligned.

[0031] Obtain multi-scale depth and shallow features of aligned image samples;

[0032] The multi-scale depth features of the aligned image samples are input into the fully convolutional network and two convolutional branches for pixel-by-pixel classification to obtain the predicted instance mask for each pixel.

[0033] 5. Based on the predicted instance mask corresponding to each pixel and the instance corresponding to the image sample

[0034] The mask label is used to calculate the loss function value pixel by pixel, and the alignment module, the fully convolutional network, and the two convolutional branches are iterated based on the loss function value to obtain the mask fine-tuning network.

[0035] In some embodiments, obtaining the multi-scale depth features of image samples includes:

[0036] The image samples are subjected to target detection and instance segmentation to obtain the target detection bounding boxes and instance masks corresponding to the image samples.

[0037] The target detection boxes corresponding to the image samples are randomly perturbed to obtain the processed target detection boxes;

[0038] Based on the processed target detection box, feature information is obtained from the multi-scale feature map corresponding to the image sample, the image sample, and the instance mask corresponding to the image sample, respectively. The obtained feature information is then fused to obtain the multi-scale depth features of the image sample.

[0039] The present invention also provides an instance segmentation apparatus, comprising:

[0040] The processing unit is used to perform object detection and instance segmentation on the image to be segmented, and to obtain the object detection box and the first instance mask;

[0041] The multi-scale feature extraction and fusion unit is used to obtain feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask according to the target detection box, and to fuse the obtained feature information to obtain multi-scale shallow and deep features.

[0042] The mask fine-tuning unit is used to generate a second instance mask based on the multi-scale depth features;

[0043] The output unit is used to output the target detection box and the second instance mask, wherein the second instance mask has more mask details than the first instance mask.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the instance segmentation method as described above.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the instance segmentation method as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the instance segmentation method as described above.

[0047] The instance segmentation method, apparatus, electronic device, and storage medium provided by this invention perform target detection and instance segmentation on the image to be segmented to obtain a target detection box and a first instance mask. Then, based on the target detection box, feature information is obtained from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively. The obtained feature information is fused to obtain multi-scale shallow and deep features. Then, the mask is fine-tuned based on the multi-scale shallow and deep features to generate a second instance mask. Finally, the target detection box and the second instance mask are output. Because multi-scale low-resolution deep features and high-resolution low-level features are fused, the mask details of the instance can be better predicted, and a more refined instance mask can be output, thereby improving the accuracy of image instance segmentation. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the instance segmentation method provided in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a two-stage instance segmentation model provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a backbone network module provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram illustrating the acquisition of multi-scale depth features according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the mask fine-tuning network provided in an embodiment of the present invention;

[0054] Figure 6 A flowchart illustrating the process of determining the mask fine-tuning network provided in an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the structure of the instance segmentation device provided in an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0059] Current instance segmentation methods can be broadly categorized into two types: two-stage instance segmentation methods and single-stage instance segmentation methods. Two-stage instance segmentation methods are further divided into two sub-types: bottom-up semantic segmentation-based methods and top-down detection-based methods. Bottom-up semantic segmentation methods first locate the region containing the instance using object detection, then perform semantic segmentation within the detection bounding box. Each segmentation result can be output as an instance. The drawback of this type of method is that due to the use of deeper neural networks, the target's resolution in the deep feature maps is too low, resulting in loss of detail and poor segmentation performance. Additionally, the predicted instance mask usually has low resolution, losing much detail. Top-down detection-based methods first perform pixel-level semantic segmentation, then distinguish different instances through clustering, metric learning, etc. It preserves better low-level features (detailed information and location information), but its generalization performance is poor, making it unable to handle scenarios with many categories, and it has not been extensively studied. Single-stage instance segmentation methods, mainly influenced by single-stage object detection, offer faster speeds, but their accuracy cannot yet match that of two-stage methods. However, regardless of whether it's a two-stage or single-stage approach, the use of deep neural networks results in low resolution and loss of detail in the deep feature maps, leading to poor segmentation performance, which is often worse than object detection results. Furthermore, occlusion causes a loss of object information, especially when both the occluded and occluded objects belong to the same category or have similar textures and colors; segmentation of mutually occluded instances is also poor.

[0060] To address this, embodiments of the present invention provide an instance segmentation method, apparatus, electronic device, and storage medium. By performing target detection and instance segmentation on the image to be segmented, a target detection bounding box and a first instance mask are obtained. Then, based on the target detection bounding box, feature information is obtained from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively. The obtained feature information is then fused to obtain multi-scale shallow and deep features. Next, the mask is fine-tuned based on the multi-scale shallow and deep features to generate a second instance mask. Finally, the target detection bounding box and the second instance mask are output. Because multi-scale low-resolution deep features and high-resolution low-resolution features are fused, the mask details of instances can be better predicted, resulting in a more refined instance mask and improving the accuracy of image instance segmentation.

[0061] Figure 1 This is a flowchart illustrating the instance segmentation method provided in an embodiment of the present invention. Figure 1 As shown, an instance segmentation method is provided, which includes the following steps: step 110, step 120, step 130, and step 140. These method steps are merely one possible implementation of the present invention.

[0062] Step 110: Perform object detection and instance segmentation on the image to be segmented to obtain the object detection box and the first instance mask.

[0063] Optionally, an instance segmentation model can be used for image instance segmentation. An instance segmentation model generally consists of three parts: image input, instance segmentation processing, and segmentation result output. After the image to be segmented is input, image features can be extracted using a convolutional neural network (CNN) or a backbone network such as a Transformer. Then, the instance segmentation model can be used for processing. This can involve first determining the location and category of the target instance through two-stage object detection, and then segmenting the selected region. Alternatively, a semantic segmentation task can be performed first, and then different instances can be distinguished. Finally, the instance segmentation result is output, which yields the instance's bounding box (Bbox) and the first instance mask.

[0064] Step 120: Based on the target detection box, obtain feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, and fuse the obtained feature information to obtain multi-scale shallow and deep features.

[0065] Features extracted by deep networks are closer to the output and contain more abstract information, namely semantic information. They are mainly coarse-grained information, meaning that the features extracted by deep networks have low resolution. These features can be called high-level features.

[0066] The features extracted by shallow networks are closer to the input and contain more pixel information, mainly fine-grained information such as color, texture, edges, and corners. In other words, the features extracted by shallow networks have high resolution and can be called low-level features.

[0067] Existing image instance segmentation methods mostly focus on improving segmentation performance using two-stage deep learning-based segmentation networks, neglecting the issues of low target resolution and missing details in deep features. A few methods employ aligning target detection boxes extracted from the original image for instance mask prediction, but these fail to utilize the deep, low-resolution features extracted in the first stage, thus failing to achieve feature fusion between deep and low-resolution features. This invention proposes a feature fusion approach that integrates deep low-resolution features with low-resolution high-resolution features to address these problems.

[0068] Among them, the multi-scale deep and shallow features are features obtained by fusing deep low-resolution features and shallow high-resolution features at multiple scales.

[0069] Specifically, low-level features are obtained from the original image (the image to be segmented) using the location information of the target detection boxes. High-level features are extracted from the multi-scale feature map corresponding to the image to be segmented using the location information of the target detection boxes. Feature information is also obtained from the first instance mask. The obtained low-level features, high-level features, and mask features are then fused to obtain multi-scale shallow and deep features. It can be understood that multi-scale shallow and deep features fully integrate deep low-resolution features and low-resolution high-resolution features, which is beneficial for inferring more refined mask information.

[0070] The multi-scale feature map of the image to be segmented can be the output of the backbone network of the instance segmentation model in step 110. The multi-scale feature map includes multiple feature maps with different resolutions.

[0071] Step 130: Generate a second instance mask based on the multi-scale depth and shallow features.

[0072] This invention further refines the prediction of instance masks based on multi-scale shallow and deep features. Since multi-scale shallow and deep features are obtained by deep fusion of deep and low-level features, the second instance mask generated based on multi-scale shallow and deep features has higher resolution and more mask details compared to the first instance mask.

[0073] Optionally, the multi-scale shallow and deep features are input into the mask fine-tuning network for mask fine-tuning to obtain the second instance mask output by the mask fine-tuning network.

[0074] The mask fine-tuning network is used to predict the mask of the input features. It uses multi-scale, shallow and high-resolution features as input, and after alignment, generates a mask that is close to the size of the original image.

[0075] Step 140: Output the target detection bounding box and the second instance mask.

[0076] The present invention obtains the target detection box and the first instance mask of the instance through step 110, and then obtains multi-scale shallow and deep features. The multi-scale shallow and deep features are input into the mask fine-tuning network, thereby realizing further fine-tuning of the first instance mask obtained in step 110 and generating a second instance mask. Since it integrates multi-scale low-resolution deep features and high-resolution low-level features, it can better predict the mask details of the instance and output a more refined instance mask.

[0077] The instance segmentation method provided in this invention performs target detection and instance segmentation on the image to be segmented to obtain a target detection box and a first instance mask. Then, based on the target detection box, feature information is obtained from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively. The obtained feature information is fused to obtain multi-scale shallow and deep features. Then, the mask is fine-tuned based on the multi-scale shallow and deep features to generate a second instance mask. Finally, the target detection box and the second instance mask are output. Because multi-scale low-resolution deep features and high-resolution low-level features are fused, the mask details of the instance can be better predicted, and a more refined instance mask can be output, thus improving the accuracy of image instance segmentation.

[0078] It should be noted that each embodiment of the present invention can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0079] In some embodiments, the step of performing object detection and instance segmentation on the image to be segmented to obtain object detection boxes and a first instance mask includes:

[0080] Based on a two-stage instance segmentation model, target detection and instance segmentation are performed on the image to be segmented, and the target detection bounding box and the first instance mask output by the two-stage instance segmentation model are obtained.

[0081] Figure 2 This is a schematic diagram of a two-stage instance segmentation model provided in an embodiment of the present invention. Figure 2 As shown, the two-stage instance segmentation model includes: a backbone network module, a candidate set generation module, and a prediction module.

[0082] The backbone network module is used to extract features from the image to be segmented and output multi-scale feature maps.

[0083] Figure 3 This is a schematic diagram of a backbone network module provided in an embodiment of the present invention. Figure 3 As shown, in some embodiments, the backbone network module includes: multiple backbone networks and a feature pyramid network.

[0084] Multiple backbone networks are based on model fusion in CBNetV2. CBNetV2 uses multiple (usually two) SwingTransformer network models as backbone networks, fusing high-level and low-level features from multiple backbone networks and gradually expanding the receptive domain to more effectively perform object detection and instance segmentation.

[0085] The Swin Transformer network model is an attention-based backbone network model that incorporates sliding window operations and a hierarchical design for feature extraction from input images. It mainly consists of four stages, each employing local windows and local cross windows. Each stage also reduces the resolution of the input feature map, expanding the receptive field layer by layer, much like a convolutional neural network.

[0086] Feature Pyramid Networks (FPNs) are used to connect the multiple backbone networks and output multi-scale feature maps.

[0087] The Feature Pyramid Network primarily addresses the multi-scale problem in object detection. It introduces a feature map scaled by a factor of two from the feature map of the next resolution at each resolution, performing element-wise addition. Through this connection, each prediction layer's feature map fuses features of different resolutions and semantic strengths, and these fused feature maps at different resolutions are used for object detection at their respective resolutions. This ensures that each layer has appropriate resolution and strong semantic features. Furthermore, since this method only adds extra cross-layer connections to the original network, it incurs almost no additional time or computational cost in practical applications.

[0088] By constructing feature pyramids at different scales using image pyramids and modifying network connections, it is possible to fuse low-resolution feature maps with strong semantic information and high-resolution feature maps with weak semantic information but rich spatial information with minimal increase in computation, significantly improving the performance of small object detection. This is achieved through upsampling of high-level features and top-down connections between low-level features, with prediction performed at each layer. This approach demonstrates significant performance improvements in instance segmentation and object detection tasks, particularly for small object prediction.

[0089] The output of the backbone network module is connected to the candidate set generation module, which is used to generate a candidate set based on the multi-scale feature map.

[0090] The output of the candidate set generation module is connected to the prediction module, which performs object detection and instance segmentation based on the candidate set to obtain object detection boxes and a first instance mask. The prediction module is a multi-stage model of a multi-level hybrid task (HTC).

[0091] like Figure 2 As shown, the prediction module includes convolutional and fully connected networks, as well as a fully convolutional network. The fully convolutional network is used for instance mask prediction, while the convolutional and fully connected networks are used for object detection box generation and type information prediction.

[0092] In this embodiment of the invention, based on a two-stage instance segmentation model, target detection and instance segmentation are performed on the image to be segmented, which can effectively perform target detection and instance segmentation.

[0093] However, instance masks inferred using a two-stage instance segmentation model typically have low resolution (e.g., the instance mask inferred by HTC is 28x28), losing many detailed features. Therefore, this invention proposes to obtain feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask output by the two-stage instance segmentation model, based on the target detection box output by the two-stage instance segmentation model. The obtained feature information is then fused to obtain multi-scale shallow and deep features. A more refined instance mask can then be generated based on these multi-scale shallow and deep features.

[0094] In some embodiments, step 120 includes:

[0095] The first feature information corresponding to the target detection box is extracted from each feature map of the multi-scale feature map output by the backbone network module in the two-stage instance segmentation model. The extracted first feature information is then convolved and upsampled sequentially to obtain the first feature set.

[0096] The image corresponding to the target detection box is extracted from the image to be segmented, and features are extracted from the image extracted from the image to be segmented to obtain a second feature set;

[0097] The image corresponding to the target detection box is extracted from the first instance mask, and features are extracted from the image extracted from the instance mask to obtain a third feature set;

[0098] Feature fusion is performed on the first feature set, the second feature set, and the third feature set to obtain multi-scale shallow and deep features.

[0099] Figure 4 This is a schematic diagram illustrating the acquisition of multi-scale depth features according to an embodiment of the present invention. (Reference) Figure 4The multi-scale feature map extraction and fusion method proposed in this invention can fully integrate low-resolution deep features and high-resolution shallow features, resulting in better performance for downstream tasks. During the training phase, after obtaining the predicted target detection bounding boxes (Bboxes) and instance masks (M) from step 110, the target detection bounding boxes are first randomly perturbed (using a perturbation amplitude conforming to a normal distribution) to perform sample augmentation, increase noise, and simulate the target detection bounding boxes from the inference stage, thereby improving the model's generalization ability. Based on the processed target detection bounding boxes, corresponding feature information is extracted from each feature map of the high-level feature maps. These features are then reduced in dimensionality using 1x1 convolutions, upsampled to the size of the target detection bounding boxes, and finally fused together to obtain the first feature set F1. Simultaneously, based on the processed target detection bounding boxes, corresponding images are extracted from the original image and input into a simple feature extraction network to obtain the second feature set F2 with unchanged size. Finally, based on the processed target detection bounding boxes, corresponding images are extracted from the inferred instance mask M and input into a simple feature extraction network to obtain the third feature set F3. Finally, the feature sets F1, F2, and F3 are fused to obtain the fused feature set Fmerge, which is a multi-scale shallow and deep feature set. During the inference stage, the target detection boxes extracted by the two-stage instance segmentation network are used directly without random perturbation.

[0100] In this embodiment of the invention, based on the target detection box output by the two-stage instance segmentation model, feature information is obtained from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask output by the two-stage instance segmentation model, respectively. The obtained feature information is then fused to obtain multi-scale shallow and deep features, which can achieve more refined mask prediction.

[0101] In some embodiments, generating a second instance mask based on the multi-scale depth features includes:

[0102] The multi-scale shallow and deep features are input into the mask fine-tuning network for alignment and pixel-level classification to obtain the instance mask of the occluded object and the instance mask of the occluded object. The instance mask of the occluded object is used as the second instance mask.

[0103] The mask fine-tuning network includes an alignment module, a fully convolutional network, and two convolutional branches. The alignment module is used to align the multi-scale shallow and deep features. The fully convolutional network and the two convolutional branches are used to perform pixel-level classification based on the aligned multi-scale shallow and deep features to obtain the instance mask of the occluded object and the instance mask of the occluded object.

[0104] It should be noted that the mask fine-tuning network in this embodiment of the invention uses multi-scale shallow and deep features that integrate high and low resolution as input. After alignment, pixel-level classification is performed, which can generate a mask that is close to the size of the original image. The mask generated in this way is more refined and has less missing details.

[0105] In practice, instances within a single object detection bounding box often occlude each other. Many models and methods address this issue by designing stronger backbone networks or more complex multi-stage mask inference networks. However, they often fail to learn edge differences between occluded and occluded objects. To address this, the Mask Refine Network in this embodiment of the invention can simultaneously output two masks, representing the instance mask of the occluded object and the instance mask of the occluded object, respectively.

[0106] Figure 5 This is a schematic diagram of the mask fine-tuning network provided in an embodiment of the present invention. Figure 5 As shown, specifically, embodiments of the present invention use fully convolutional networks (FCNs) and two convolutional branches to predict the instance mask of the occluder and the instance mask of the occluded object.

[0107] Fully convolutional networks (FCNs) perform pixel-level image classification, thus solving the semantic segmentation problem. The FCN network structure mainly consists of two parts: a fully convolutional part and a deconvolutional part. The fully convolutional part uses classic CNN networks (such as VGG and ResNet) to extract features; the deconvolutional part obtains the original-size semantic segmented image through upsampling.

[0108] Unlike classic convolutional neural networks that use fully connected layers after convolutional layers to obtain fixed-length feature vectors for classification (fully connected layer + softmax output), FCN can accept input images of any size. It uses deconvolutional layers to upsample the feature map of the last convolutional layer, restoring it to the same size as the input image. This allows it to generate a prediction for each pixel while preserving the spatial information in the original input image.

[0109] This invention provides an embodiment of the invention that aligns and classifies multi-scale shallow and deep features into a mask fine-tuning network to obtain instance masks of occluders and instances of occluded objects. This enables more refined prediction of instance masks. Furthermore, by simultaneously inferring the occluder and occluded object masks, the edges of the instance masks can be fine-tuned and refined, significantly improving the prediction accuracy of mutually occluded instances.

[0110] Figure 6This is a flowchart illustrating the process of determining the mask fine-tuning network provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the process of determining the mask fine-tuning network includes:

[0111] Step 610: Obtain the multi-scale depth features of the image sample and the instance mask label corresponding to the image sample;

[0112] Step 620: Based on the alignment module, align the multi-scale depth features of the image sample to obtain the aligned multi-scale depth features of the image sample;

[0113] Step 630: Input the multi-scale depth features of the aligned image samples into the fully convolutional network and two convolutional branches for pixel-by-pixel classification to obtain the prediction instance mask corresponding to each pixel;

[0114] Step 640: Based on the predicted instance mask corresponding to each pixel and the instance mask label corresponding to the image sample, calculate the loss function value pixel by pixel, and perform parameter iteration on the alignment module, the fully convolutional network and the two convolutional branches based on the loss function value to obtain the mask fine-tuning network.

[0115] Specifically, based on the predicted instance mask corresponding to each pixel and the mask label corresponding to each pixel, the loss function value corresponding to each pixel is calculated. Based on the loss function corresponding to each pixel, the alignment module, the fully convolutional network and the two convolutional branches are iterated. After the iteration is completed, the mask fine-tuning network is obtained.

[0116] The mask label corresponding to each pixel is determined based on the instance mask label corresponding to the image sample.

[0117] In some embodiments, obtaining multi-scale depth features of image samples in step 610 includes:

[0118] The image samples are subjected to target detection and instance segmentation to obtain the target detection bounding boxes and instance masks corresponding to the image samples.

[0119] The target detection boxes corresponding to the image samples are randomly perturbed to obtain the processed target detection boxes;

[0120] Based on the processed target detection box, feature information is obtained from the multi-scale feature map corresponding to the image sample, the image sample, and the instance mask corresponding to the image sample, respectively. The obtained feature information is then fused to obtain the multi-scale depth features of the image sample.

[0121] Optionally, a two-stage instance segmentation network can be used to perform object detection and instance segmentation on the image samples to obtain the object detection bounding boxes and instance masks corresponding to the image samples.

[0122] Randomly perturb the target detection boxes corresponding to the image samples. The perturbation amplitude can be in accordance with a normal distribution to achieve sample enhancement, increase noise, and simulate the target detection boxes during the inference stage, thereby improving the generalization ability of the model.

[0123] Specifically, based on the processed target detection box, feature information is obtained from the multi-scale feature map corresponding to the image sample, the image sample itself, and the instance mask corresponding to the image sample, respectively. The obtained feature information is then fused to obtain the multi-scale depth features of the image sample, including:

[0124] Based on the processed target detection box, the corresponding second feature information is extracted from each feature map of the multi-scale feature map corresponding to the image sample. The extracted second feature information is then convolved and upsampled sequentially to obtain the fourth feature set.

[0125] Based on the processed target detection box, a corresponding image is cropped from the image sample, and feature extraction is performed on the cropped image to obtain a fifth feature set;

[0126] Based on the processed target detection box, the corresponding image is extracted from the instance mask corresponding to the image sample, and feature extraction is performed on the image extracted from the instance mask corresponding to the image sample to obtain the sixth feature set;

[0127] Feature fusion is performed on the fourth feature set, the fifth feature set, and the sixth feature set to obtain the multi-scale depth features of the image sample.

[0128] In this embodiment of the invention, by training a mask fine-tuning network, the problem of insufficient instance mask resolution and loss of details obtained by the two-stage instance segmentation network can be solved, enabling more refined instance mask prediction and significantly improving the prediction accuracy of mutually occluded instances.

[0129] The instance segmentation apparatus provided in the embodiments of the present invention is described below. The instance segmentation apparatus described below and the instance segmentation method described above can be referred to in correspondence.

[0130] Figure 7 This is a schematic diagram of the structure of the instance segmentation device provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the instance segmentation device 700 includes:

[0131] Processing unit 710 is used to perform object detection and instance segmentation on the image to be segmented, and obtain object detection boxes and first instance masks;

[0132] The multi-scale feature extraction and fusion unit 720 is used to obtain feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask according to the target detection box, and to fuse the obtained feature information to obtain multi-scale shallow and deep features.

[0133] The mask fine-tuning unit 730 is used to generate a second instance mask based on the multi-scale depth features;

[0134] The output unit 740 is used to output the target detection box and the second instance mask, the second instance mask having more mask details than the first instance mask.

[0135] In some embodiments, the step of performing object detection and instance segmentation on the image to be segmented to obtain object detection boxes and a first instance mask includes:

[0136] Based on a two-stage instance segmentation model, target detection and instance segmentation are performed on the image to be segmented, and the target detection bounding box and the first instance mask output by the two-stage instance segmentation model are obtained.

[0137] The two-stage instance segmentation model includes a backbone network module, a candidate set generation module, and a prediction module. The backbone network module is used to extract features from the image to be segmented and output a multi-scale feature map. The candidate set generation module is used to generate a candidate set based on the multi-scale feature map. The prediction module is used to perform object detection and instance segmentation based on the candidate set to obtain an object detection box and a first instance mask.

[0138] In some embodiments, the backbone network module includes: multiple backbone networks and a feature pyramid network;

[0139] Each of the backbone networks is a Swing Transformer network model containing multiple stages, and the Swing Transformer network model is used for feature extraction.

[0140] The feature pyramid network is used to connect the multiple backbone networks and output multi-scale feature maps.

[0141] In some embodiments, the step of obtaining feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask according to the target detection box, and fusing the obtained feature information to obtain multi-scale shallow and deep features includes:

[0142] The first feature information corresponding to the target detection box is extracted from each feature map of the multi-scale feature map output by the backbone network module in the two-stage instance segmentation model. The extracted first feature information is then convolved and upsampled sequentially to obtain the first feature set.

[0143] The image corresponding to the target detection box is extracted from the image to be segmented, and features are extracted from the image extracted from the image to be segmented to obtain a second feature set;

[0144] The image corresponding to the target detection box is extracted from the first instance mask, and features are extracted from the image extracted from the instance mask to obtain a third feature set;

[0145] Feature fusion is performed on the first feature set, the second feature set, and the third feature set to obtain multi-scale shallow and deep features.

[0146] In some embodiments, generating a second instance mask based on the multi-scale depth features includes:

[0147] The multi-scale shallow and deep features are input into the mask fine-tuning network for alignment and pixel-level classification to obtain the instance mask of the occluded object and the instance mask of the occluded object. The instance mask of the occluded object is used as the second instance mask.

[0148] The mask fine-tuning network includes an alignment module, a fully convolutional network, and two convolutional branches. The alignment module is used to align the multi-scale shallow and deep features. The fully convolutional network and the two convolutional branches are used to perform pixel-level classification based on the aligned multi-scale shallow and deep features to obtain the instance mask of the occluded object and the instance mask of the occluded object.

[0149] In some embodiments, the process of determining the mask fine-tuning network includes:

[0150] Obtain multi-scale depth features of image samples and instance mask labels corresponding to the image samples;

[0151] Based on the alignment module, the multi-scale depth features of the image sample are aligned to obtain the multi-scale depth features of the aligned image sample.

[0152] The multi-scale depth features of the aligned image samples are input into the fully convolutional network and two convolutional branches for pixel-by-pixel classification to obtain the predicted instance mask for each pixel.

[0153] Based on the predicted instance mask corresponding to each pixel and the instance mask label corresponding to the image sample, the loss function value is calculated pixel by pixel, and the alignment module, the fully convolutional network and the two convolutional branches are iterated based on the loss function value to obtain the mask fine-tuning network.

[0154] In some embodiments, obtaining the multi-scale depth features of image samples includes:

[0155] The image samples are subjected to target detection and instance segmentation to obtain the target detection bounding boxes and instance masks corresponding to the image samples.

[0156] The target detection boxes corresponding to the image samples are randomly perturbed to obtain the processed target detection boxes;

[0157] Based on the processed target detection box, feature information is obtained from the multi-scale feature map corresponding to the image sample, the image sample, and the instance mask corresponding to the image sample, respectively. The obtained feature information is then fused to obtain the multi-scale depth features of the image sample.

[0158] It should be noted that the instance segmentation device provided in this embodiment of the invention can implement all the method steps implemented in the above-described instance segmentation method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0159] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an instance segmentation method, which includes: performing object detection and instance segmentation on the image to be segmented to obtain an object detection box and a first instance mask; based on the object detection box, obtaining feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, and fusing the obtained feature information to obtain multi-scale shallow and deep features; generating a second instance mask based on the multi-scale shallow and deep features; and outputting the object detection box and the second instance mask, wherein the second instance mask has more mask details than the first instance mask.

[0160] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the instance segmentation method provided in the above-described method embodiments. The method includes: performing object detection and instance segmentation on the image to be segmented to obtain an object detection box and a first instance mask; obtaining feature information from a multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask according to the object detection box, and fusing the obtained feature information to obtain multi-scale shallow and deep features; generating a second instance mask based on the multi-scale shallow and deep features; and outputting the object detection box and the second instance mask, wherein the second instance mask has more mask details than the first instance mask.

[0162] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the instance segmentation method provided in the above-described method embodiments. The method includes: performing object detection and instance segmentation on an image to be segmented to obtain an object detection box and a first instance mask; acquiring feature information from a multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, based on the object detection box, and fusing the acquired feature information to obtain multi-scale shallow and deep features; generating a second instance mask based on the multi-scale shallow and deep features; and outputting the object detection box and the second instance mask, wherein the second instance mask has more mask details than the first instance mask.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An instance segmentation method, characterized by, include: Perform object detection and instance segmentation on the image to be segmented to obtain object detection boxes and the first instance mask; Based on the target detection box, feature information is obtained from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, and the obtained feature information is fused to obtain multi-scale shallow and deep features; Based on the multi-scale depth features, a second instance mask is generated; Output the target detection bounding box and the second instance mask, where the second instance mask has more mask details than the first instance mask; The multi-scale feature map is a set of feature maps with different resolutions output by the backbone network module for feature extraction. The backbone network module includes multiple backbone networks and a feature pyramid network. Each backbone network contains a multi-stage Swin Transformer network model for feature extraction, and the feature pyramid network is used to connect the multiple backbone networks to output the multi-scale feature map. The generation of the second instance mask based on the multi-scale depth features includes: The multi-scale shallow and deep features are input into the mask fine-tuning network for mask fine-tuning to obtain the second instance mask; The mask fine-tuning network includes an alignment module, a fully convolutional network, and two convolutional branches. The alignment module is connected to the fully convolutional network, and the fully convolutional network is connected to the two convolutional branches respectively.

2. The instance segmentation method according to claim 1, characterized in that, The process of performing object detection and instance segmentation on the image to be segmented to obtain object detection boxes and a first instance mask includes: Based on a two-stage instance segmentation model, target detection and instance segmentation are performed on the image to be segmented, and the target detection bounding box and the first instance mask output by the two-stage instance segmentation model are obtained. The two-stage instance segmentation model includes a backbone network module, a candidate set generation module, and a prediction module. The backbone network module is used to extract features from the image to be segmented and output a multi-scale feature map. The candidate set generation module is used to generate a candidate set based on the multi-scale feature map. The prediction module is used to perform object detection and instance segmentation based on the candidate set to obtain an object detection box and a first instance mask.

3. The instance segmentation method according to claim 2, characterized in that, The step involves obtaining feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask, respectively, based on the target detection box, and fusing the obtained feature information to obtain multi-scale shallow and deep features, including: The first feature information corresponding to the target detection box is extracted from each feature map of the multi-scale feature map output by the backbone network module in the two-stage instance segmentation model. The extracted first feature information is then convolved and upsampled sequentially to obtain the first feature set. The image corresponding to the target detection box is extracted from the image to be segmented, and features are extracted from the image extracted from the image to be segmented to obtain a second feature set; The image corresponding to the target detection box is extracted from the first instance mask, and features are extracted from the image extracted from the instance mask to obtain a third feature set; Feature fusion is performed on the first feature set, the second feature set, and the third feature set to obtain multi-scale shallow and deep features.

4. The instance segmentation method according to any one of claims 1-2, characterized in that, The step of inputting the multi-scale shallow and deep features into the mask fine-tuning network for mask fine-tuning to obtain the second instance mask includes: The multi-scale depth features are input into the mask fine-tuning network for alignment and pixel-level classification to obtain the instance mask of the occluder and the instance mask of the occluded object. The instance mask of the occluder is used as the second instance mask. The alignment module is used to align the multi-scale shallow and deep features, and the fully convolutional network and two convolutional branches are used to perform pixel-level classification based on the aligned multi-scale shallow and deep features to obtain the instance mask of the occluded object and the instance mask of the occluded object.

5. The instance segmentation method according to claim 4, characterized in that, The process of determining the mask fine-tuning network includes: Obtain multi-scale depth features of image samples and instance mask labels corresponding to the image samples; Based on the alignment module, the multi-scale depth features of the image sample are aligned to obtain the multi-scale depth features of the aligned image sample. The multi-scale depth features of the aligned image samples are input into the fully convolutional network and two convolutional branches for pixel-by-pixel classification to obtain the predicted instance mask for each pixel. Based on the predicted instance mask corresponding to each pixel and the instance mask label corresponding to the image sample, the loss function value is calculated pixel by pixel, and the alignment module, the fully convolutional network and the two convolutional branches are iterated based on the loss function value to obtain the mask fine-tuning network.

6. The instance segmentation method according to claim 5, characterized in that, The acquisition of multi-scale depth features of image samples includes: The image samples are subjected to target detection and instance segmentation to obtain the target detection bounding boxes and instance masks corresponding to the image samples. The target detection boxes corresponding to the image samples are randomly perturbed to obtain the processed target detection boxes; Based on the processed target detection box, feature information is obtained from the multi-scale feature map corresponding to the image sample, the image sample, and the instance mask corresponding to the image sample, respectively. The obtained feature information is then fused to obtain the multi-scale depth features of the image sample.

7. An instance segmentation device, characterized in that, include: The processing unit is used to perform object detection and instance segmentation on the image to be segmented, and to obtain the object detection box and the first instance mask; The multi-scale feature extraction and fusion unit is used to obtain feature information from the multi-scale feature map corresponding to the image to be segmented, the image to be segmented, and the first instance mask according to the target detection box, and to fuse the obtained feature information to obtain multi-scale shallow and deep features. The mask fine-tuning unit is used to generate a second instance mask based on the multi-scale depth features; The output unit is used to output the target detection box and the second instance mask, wherein the second instance mask has more mask details than the first instance mask; The multi-scale feature map is a set of feature maps with different resolutions output by the backbone network module for feature extraction. The backbone network module includes multiple backbone networks and a feature pyramid network. Each backbone network contains a multi-stage Swin Transformer network model for feature extraction, and the feature pyramid network is used to connect the multiple backbone networks to output the multi-scale feature map. The generation of the second instance mask based on the multi-scale depth features includes: The multi-scale shallow and deep features are input into the mask fine-tuning network for mask fine-tuning to obtain the second instance mask; The mask fine-tuning network includes an alignment module, a fully convolutional network, and two convolutional branches. The alignment module is connected to the fully convolutional network, and the fully convolutional network is connected to the two convolutional branches respectively.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the instance segmentation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the instance segmentation method as described in any one of claims 1 to 6.

Images