Long-tail scene instance segmentation method and device, equipment and medium

Abstract

The invention relates to the technical field of image processing, in particular to a long-tail scene instance segmentation method and device, equipment and a medium. The method comprises the steps of obtaining a long-tail distribution data set, and performing data enhancement on the long-tail distribution data set to obtain a tail extension data set; obtaining a plurality of head class images and a plurality of tail class images according to the long-tail distribution data set; splicing any tail class image on each head class image to generate a composite image corresponding to each head class image; training a preset global teacher network according to each head class image and the composite image corresponding to each head class image, and generating a pseudo tag corresponding to each head class image; and performing cooperative training on a preset local teacher network and a preset student segmentation network according to the pseudo tag and the tail extension data set corresponding to each head image to obtain a target segmentation network, and performing instance segmentation on the to-be-segmented image based on the target segmentation network.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and medium for segmenting long-tailed scene instances. Background Technology

[0002] Instance segmentation aims to associate each pixel in an image with its corresponding semantic category and instance. It combines the characteristics of semantic segmentation, requiring pixel-level classification, with the characteristics of object detection, requiring the localization of different instances. However, the long-tailed distribution problem has been a persistent challenge in instance segmentation. A long-tailed distribution refers to a dataset where some categories have very few samples, while others have a relatively large number. This imbalanced distribution prevents the model from fully learning the features of the tail categories during training, leading to poor performance in identifying these rare categories. This not only limits the model's ability to extract features from these categories but may also result in insufficient generalization ability when faced with new samples, or even misclassification.

[0003] In recent years, the rapid development of object detection methods has greatly facilitated the emergence of many successful approaches for efficient instance segmentation. Recent instance segmentation methods, through fine-grained human annotation, have achieved impressive results on the challenging COCO dataset. This field has seen unprecedented breakthroughs thanks to deep neural networks and large-scale datasets for common objects such as people and cars. However, when dealing with rare and less common objects, performance drops sharply due to insufficient training examples.

[0004] Currently, most methods address the "long-tail" distribution problem during model training by developing training objectives or algorithms. Typically, the model is first pre-trained in a traditional manner using all or only the head class data, and then fine-tuned on the entire long-tail dataset using resampling or cost-sensitive learning. However, current methods still perform poorly in long-tail instance segmentation. Therefore, a method for instance segmentation that can effectively address the long-tail distribution problem is urgently needed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, apparatus, device and medium for segmenting long-tail scene instances, so as to solve the above-mentioned technical problem.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a long-tail scene instance segmentation method, comprising: acquiring a long-tail distribution dataset; performing data augmentation on the long-tail distribution dataset to obtain a tail-extended dataset; acquiring multiple head class images and multiple tail class images based on the long-tail distribution dataset; stitching any tail class image onto each head class image to generate a synthetic image corresponding to each head class image; training a preset global teacher network based on each head class image and the synthetic image corresponding to each head class image to generate pseudo-labels corresponding to each head class image; and co-training a preset local teacher network and a preset student segmentation network based on the pseudo-labels corresponding to each head class image and the tail-extended dataset to obtain a target segmentation network, and performing instance segmentation on the image to be segmented based on the target segmentation network.

[0007] The beneficial effects of this invention are as follows: By augmenting the long-tailed distribution dataset, this invention solves the problem of insufficient tail class samples. By introducing additional instance information of the tail class image category through synthetic images, the activation regions of the current head class images are weakened, thereby avoiding the model's over-reliance on head class features and enhancing the model's ability to perceive tail classes. Utilizing a teacher-student-teacher contrastive learning structure, a global teacher network provides global semantic information to help the student segmentation network distinguish between positive and negative samples of different categories, while a local teacher network provides fine-grained semantic information to optimize positive sample pairs within the same category. This dual mechanism improves the model's ability to learn features of long-tailed categories and reduces misclassification caused by class imbalance.

[0008] Based on the above technical solution, the present invention can be further improved as follows.

[0009] Further, the step of data augmentation of the long-tailed distribution dataset to obtain a tail-extended dataset includes: acquiring an image dataset, the image dataset including multiple first categories, each first category containing multiple object images; for each first category, segmenting each object image in the first category using a co-segmentation algorithm to obtain a target object mask fragment corresponding to each object image in the first category; for each first category, acquiring a pseudo-boundary box corresponding to each object image in the first category; determining multiple target mask fragments based on the pseudo-boundary boxes and target object mask fragments corresponding to each object image in each first category; and pasting at least one target mask fragment onto each image in the long-tailed distribution dataset to generate the tail-extended dataset.

[0010] Furthermore, for each object image in each of the first categories, the pseudo-boundary box corresponding to the object image is obtained in the following manner: S11. According to the preset removal order, remove a portion of the object image to obtain a removed image; S12. Classify the removed image using a pre-trained classifier to obtain the second category corresponding to the removed image; S13. Determine whether the second category corresponding to the removed image is the same as the first category corresponding to the object image. If they are the same, then execute S11; otherwise, determine the pseudo bounding box corresponding to the object image based on the current partial region.

[0011] Further, determining multiple target mask segments based on the pseudo-boundary boxes and target object mask segments corresponding to each object image in each of the first categories includes: for each first category, calculating the overlap between the target object mask segment and the pseudo-boundary box corresponding to each object image in the first category to obtain the overlap corresponding to each object image in the first category; for each first category, obtaining the target mask segment in the first category based on the magnitude of the overlap corresponding to each object image in the first category; and obtaining multiple target mask segments based on the target mask segments in each of the first categories.

[0012] Furthermore, the preset global teacher network, the preset local teacher network, and the preset student segmentation network share a backbone network, which is a ResNet-38 backbone network.

[0013] Further, the step of training a preset global teacher network based on each of the head class images and the corresponding synthetic images to generate pseudo-labels for each head class image includes: obtaining a category label for each head class image; for each head class image, performing feature extraction through the preset global teacher network based on the head class image and the corresponding synthetic image to obtain synthetic class activation map features and anchor point class activation map features for the head class image; for each head class image, performing occlusion processing on the head class image based on the anchor point class activation map features to generate an occluded image for the head class image; for each head class image, performing feature extraction through the preset global teacher network based on the occluded image to obtain occlusion class activation map features for the head class image; and for each head class image, generating image category knowledge and pseudo-labels for the head class image based on the category label, synthetic class activation map features, anchor point class activation map features, and occlusion class activation map features.

[0014] Furthermore, the step of co-training a preset local teacher network and a preset student segmentation network to obtain a target segmentation network based on the pseudo-labels corresponding to each of the head class images and the tail extended dataset includes: for each image in the tail extended dataset, extracting local semantic information through the preset local teacher network to generate local semantic knowledge corresponding to each image in the tail extended dataset; and training a preset student segmentation network based on a label correction mechanism to obtain the target segmentation network based on the pseudo-labels corresponding to each of the head class images, the tail extended dataset, the image category knowledge corresponding to each of the head class images, and the local semantic knowledge corresponding to each image in the tail extended dataset.

[0015] To address the aforementioned technical problems, the present invention also provides a long-tail scene instance segmentation device, comprising: The data acquisition module is used to acquire a long-tailed distributed dataset, perform data augmentation on the long-tailed distributed dataset, and obtain a tail-extended dataset. The image acquisition module is used to acquire multiple head class images and multiple tail class images based on the long-tail distribution dataset; An image synthesis module is used to stitch any of the tail class images onto each of the head class images to generate a synthesized image corresponding to each of the head class images; The first network training module is used to train a preset global teacher network based on each of the head class images and the corresponding synthetic images of each of the head class images, and to generate pseudo-labels corresponding to each of the head class images. The second network training module is used to collaboratively train a preset local teacher network and a preset student segmentation network based on the pseudo-labels corresponding to each head class image and the tail extended dataset to obtain a target segmentation network, so as to perform instance segmentation on the image to be segmented based on the target segmentation network.

[0016] To address the aforementioned technical problems, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the long-tail scene instance segmentation method as described above.

[0017] To address the aforementioned technical problems, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the long-tail scene instance segmentation method described above. Attached Figure Description

[0018] Figure 1 A bar chart showing the distribution of the number of instances of different frequency categories in the long-tail dataset; Figure 2 This is a flowchart of a long-tail scene instance segmentation method according to the present invention; Figure 3 This is a schematic diagram of the overall network structure of a long-tail scene instance segmentation method according to the present invention; Figure 4 The original long-tailed scene image to be segmented; Figure 5 The image is generated after the original long-tailed scene image to be segmented is processed by a long-tailed scene instance segmentation method of the present invention. Figure 6 This is a schematic diagram of a long-tail scene instance segmentation device according to the present invention; Figure 7 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation

[0019] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0020] Long-tail distributions typically exhibit a pattern where some classes have very few samples (tail classes), while other classes have relatively more samples (head classes). Figure 1 As shown. This imbalanced distribution prevents the model from fully learning the features of the tail categories during training, resulting in poor performance in identifying these rare categories. This not only limits the model's ability to extract features from these categories but may also lead to insufficient generalization ability when faced with new samples, or even misclassification.

[0021] To address the aforementioned issues, in addition to learning models directly from long-tail data, existing work has also explored data augmentation techniques to improve the performance of long-tail object detection and instance segmentation. Data augmentation is a method that increases the diversity and scale of a dataset by generating new training samples. It can help the model learn a broader feature distribution, thereby improving its ability to identify tail categories.

[0022] Furthermore, contrastive learning, as an emerging technique, is gradually demonstrating its unique advantages and application potential. By constructing pairs of positive and negative samples, contrastive learning learns the similarities and differences between samples, thereby improving the model's robustness to features and its generalization ability. In recent years, the application of contrastive learning in long-tail problems has gradually attracted attention. For example, KCL proposed a k-positive contrastive loss, which alleviates the class imbalance problem and improves the model's ability to identify rare classes by learning a balanced feature space. In addition, parametric contrastive learning (PaCo) further optimizes the effect of supervised contrastive learning by introducing learnable class centers.

[0023] Example 1 Based on this, such as Figure 2 As shown, this embodiment provides a method for segmenting long-tail scene instances, including: S101. Obtain the long-tailed distribution dataset, perform data augmentation on the long-tailed distribution dataset, and obtain the tail-extended dataset.

[0024] In this embodiment, the long-tailed distribution dataset uses the existing LVIS dataset. The categories in the LVIS dataset exhibit a natural long-tailed distribution, containing some common categories and a large number of rare categories with few training samples.

[0025] Furthermore, based on the LVIS dataset, data augmentation was performed to generate a tail-extended dataset to address the problem of insufficient tail class samples.

[0026] S102. Based on the long-tail distribution dataset, obtain multiple head class images and multiple tail class images.

[0027] Tail class images refer to images in the LVIS dataset that represent most classes and have a small number of images in each class, while head class images refer to images in a few classes and have a large number of images in each class.

[0028] S103. Concatenate any tail class image onto each head class image to generate a composite image corresponding to each head class image.

[0029] Specifically, the synthesized image is obtained by stitching a head class image top-to-bottom with any tail class image, and the size of the synthesized image is the same as the original head class image. By introducing additional instance information from the tail class image, the activation region of the current head class image is weakened, thereby avoiding the model's over-reliance on head class features and enhancing the model's ability to perceive tail classes.

[0030] S104. Based on each head class image and the corresponding synthetic and occluded images, train the preset global teacher network to generate pseudo-labels for each head class image.

[0031] S105. Based on the pseudo-labels and tail extended datasets corresponding to each head class image, the preset local teacher network and the preset student segmentation network are trained collaboratively to obtain the target segmentation network, which is then used to perform instance segmentation on the image to be segmented.

[0032] This method addresses the long-tailed data distribution problem in instance segmentation tasks by constructing a global teacher-student-local teacher instance segmentation framework based on data augmentation and contrastive learning. By augmenting the long-tailed dataset, the problem of insufficient tail-class samples is solved. This method utilizes simulated erasure to introduce additional instance information of the tail-class image category through synthetic images, weakening the activation regions of the current head-class images. This avoids the model's over-reliance on head-class features and enhances the model's ability to perceive tail-classes. Using a teacher-student-teacher contrastive learning structure, the global teacher network provides global semantic information to help the student segmentation network distinguish between positive and negative samples of different categories, while the local teacher network provides fine-grained semantic information to optimize positive sample pairs within the same category. This dual mechanism improves the model's ability to learn features of long-tailed categories and reduces misclassification caused by class imbalance. In summary, this method is suitable for long-tailed scenarios and can significantly improve the model's instance segmentation performance in complex environments.

[0033] Optionally, in an embodiment, data augmentation is performed on the long-tailed distribution dataset to obtain a tail-extended dataset, including: acquiring an image dataset, which includes multiple first categories, each containing multiple object images; for each first category, segmenting each object image in the first category using a co-segmentation algorithm to obtain a target object mask fragment corresponding to each object image in the first category; for each first category, acquiring a pseudo-boundary box corresponding to each object image in the first category; determining multiple target mask fragments based on the pseudo-boundary boxes and target object mask fragments corresponding to each object image in each first category; and pasting at least one target mask fragment onto each image in the long-tailed distribution dataset to generate a tail-extended dataset.

[0034] In this embodiment, the image dataset used is the existing ImageNet dataset. The ImageNet dataset has 1000 categories, with 1000 images in each category. Each image contains only one salient object, and the background is relatively simple.

[0035] Optionally, in an embodiment, for each object image in each first category, the pseudo-boundary box corresponding to the object image is obtained in the following manner: S11. According to the preset removal order, remove a portion of the object image to obtain the removed image; S12. Classify the removed image using a pre-trained classifier to obtain the second category corresponding to the removed image; S13. Determine whether the second category corresponding to the removed image is the same as the first category corresponding to the object image. If they are the same, execute S11; otherwise, determine the pseudo bounding box corresponding to the object image based on the current partial region.

[0036] Images in the ImageNet dataset all have corresponding class labels. A classifier is trained using existing ImageNet dataset images. By learning the image features of different classes, the classifier can classify the input image and determine which class it belongs to.

[0037] For each image, pseudo-boundary boxes are determined using the "Localization by Region Removal" (LORE) method. Specifically, for each image, different regions are removed sequentially. After each region is removed, the remaining image portion is fed into a trained classifier to check if the classifier can still correctly classify the image. If the classifier cannot correctly classify the image after removing a region, it is considered that the removed region may contain key features of the target object. In this way, pseudo-boundary boxes containing the target object are gradually determined.

[0038] Optionally, in an embodiment, determining multiple target mask segments based on the pseudo-boundary boxes and target object mask segments corresponding to each object image in each first category includes: for each first category, calculating the overlap between the target object mask segment and the pseudo-boundary box corresponding to each object image in the first category to obtain the overlap corresponding to each object image in the first category; for each first category, obtaining the target mask segment in the first category based on the magnitude of the overlap corresponding to each object image in the first category; and obtaining multiple target mask segments based on the target mask segments in each first category.

[0039] For each category in the ImageNet dataset, the images of that category are processed using a co-segmentation algorithm to extract the foreground salient object mask fragments from each image.

[0040] For each mask fragment obtained from the co-segmentation, it is compared with the pseudo-boundary box. If the mask highly overlaps with the pseudo-boundary box, then the mask is more likely to cover the target object and is therefore considered high quality, and thus the mask fragment is identified as the target mask fragment.

[0041] Within each category, multiple target mask fragments can be selected. When determining target mask fragments based on the degree of overlap for each category, the overlap can be sorted from largest to smallest, and the target object mask fragments with the highest overlap can be selected as target mask fragments; alternatively, an overlap threshold can be set, and target object mask fragments with an overlap greater than the preset overlap threshold can be selected as target mask fragments.

[0042] Based on the above processing method, the images in each category are processed separately to obtain the target mask fragments corresponding to each category.

[0043] For each image in the LVIS dataset, at least one target mask fragment is randomly selected from the resulting multiple target mask fragments and copied and pasted onto the original image to form a new image. This method is used to process each image in the LVIS dataset, resulting in a tail-expanded dataset. This approach achieves data augmentation, thus addressing the problem of insufficient tail class samples.

[0044] Optionally, in the embodiments, the preset global teacher network, the preset local teacher network, and the preset student segmentation network share a backbone network, which is a ResNet-38 backbone network.

[0045] Optionally, in an embodiment, the step of training a preset global teacher network based on each of the head class images and the corresponding synthetic images to generate pseudo-labels for each head class image includes: obtaining a category label for each head class image; for each head class image, performing feature extraction through the preset global teacher network based on the head class image and the corresponding synthetic image to obtain synthetic activation map features and anchor activation map features corresponding to the head class image; for each head class image, performing occlusion processing on the head class image based on the anchor activation map features corresponding to the head class image to generate an occluded image corresponding to the head class image; for each head class image, performing feature extraction through the preset global teacher network based on the occluded image corresponding to the head class image to obtain occlusion activation map features corresponding to the head class image; and for each head class image, generating image category knowledge and pseudo-labels corresponding to the head class image based on the category label, synthetic activation map features, anchor activation map features, and occlusion activation map features corresponding to the head class image.

[0046] Specifically, such as Figure 3As shown, the global teacher network processes the head class image (anchor image in the figure) and the synthetic image through the shared ResNet-38 backbone network to generate synthetic class activation map features (synthetic CAM features Fa) and anchor class activation map features (anchor CAM features Fs).

[0047] Based on the category labels corresponding to the head-class images, the synthetic CAM features, and the anchor CAM features, class feature extraction is performed. Features related to the categories present in the images are extracted from the synthetic and anchor CAM features, resulting in class feature vector maps for both the synthetic and head-class images. Furthermore, the knowledge transfer loss is calculated based on these class feature vectors. This enhances the network's ability to locate objects.

[0048] Specifically, after generating anchor point CAM features, a masking operation is applied to the head class image based on these features. By masking the salient regions of the head class, a masked image is generated. The masked image is then processed through a shared backbone network to generate masked class activation map features (masked CAM features Fm). Furthermore, the features of the masked image are aligned with the features of the anchor point image to gently enlarge the object activation region. The model is forced to focus on instances that may contain tail classes, while avoiding the introduction of excessive background noise.

[0049] By transferring knowledge, the global teacher network's attention to head-class images is partially shifted to tail-class images in the synthesized image, reducing the network's over-attention to head-class images (which may identify background areas of the target in the image). This allows for more accurate activation of head-class images, resulting in a coarse class activation map. The class activation map is then refined to obtain a high-quality pseudo-mask, i.e., a pseudo-label.

[0050] Optionally, in an embodiment, based on the pseudo-labels corresponding to each head class image and the tail extension dataset, a preset local teacher network and a preset student segmentation network are co-trained to obtain a target segmentation network. This includes: for each image in the tail extension dataset, local semantic information is extracted through the preset local teacher network to generate local semantic knowledge corresponding to each image in the tail extension dataset; based on the pseudo-labels corresponding to each head class image, the tail extension dataset, the image category knowledge corresponding to each head class image, and the local semantic knowledge corresponding to each image in the tail extension dataset, the preset student segmentation network is trained using a label correction mechanism to obtain the target segmentation network.

[0051] The student segmentation network learns from the tail-expanded dataset and high-quality pseudo-labels generated by the global teacher network, allowing it to inherit the performance of the global teacher network. During training, the student segmentation network receives pseudo-labels from the global teacher network and optimizes based on these labels. Tail-expanded datasets further enhance its robustness to variations in input data. This also reduces computational cost and improves the model's generalization ability. After training, the student segmentation network becomes the target segmentation network, which is used to perform instance segmentation on the image to be segmented, such as... Figure 4 and Figure 5 As shown.

[0052] In long-tail instance segmentation, head categories typically occupy a large portion of the image area, while tail category instances may be overwhelmed by background noise. Extracting global semantic information through a global teacher network allows for better separation of head and tail semantics. This separation helps the student segmentation network focus on the feature representations of long-tail categories, thereby improving the detection capability of rare categories. Due to the introduction of additional discriminative object information, fewer object regions are highlighted in the anchor portion of the synthesized image. Therefore, transferring more complete object knowledge from the original anchor CAM to the network enhances its object localization capability, resulting in less activation in the anchor portion of the synthesized image.

[0053] Knowledge transfer loss during image erasure It can be formalized as follows: ; ; in, and These represent the CAM features of the anchor branch and the simulated branch, respectively. CFE stands for Class Feature Extraction, which extracts feature channels related to the class present in the image. The ReLU operation indicates that alignment focuses on more prominent object regions. Additional object information is introduced only from the tail class image on the synthesized image, and other variables are controlled to ensure that the activation of the anchor part is reduced. Knowledge transfer can effectively enhance the network's localization ability to alleviate the problem of underactivation, especially when there are multiple object instances in the image.

[0054] To improve CAM quality by capturing both global context and local details of the image, a gated pyramid pooling layer is used as the final pool head. Classification logic generated by the pool head is then utilized. A global teacher network was trained using multi-label soft marginal loss, with classification loss applied. As shown below: .

[0055] To avoid introducing excessive background noise, a multi-granularity alignment module is proposed to gently expand object activation, encouraging CAM expansion of the erased image, and then the learned object knowledge is passed back to the anchor branch. First, the anchor features Fa and occlusion features Fs are input into the class feature extraction module to obtain the corresponding features of the existing target categories. Experiments show that rigid classification supervision of the mask branch often leads to overexpansion in adversarial erasure; therefore, soft class confidence guidance from the anchor branch to the mask branch is used. Specifically, a global average pooling (GAP) operation is directly applied to obtain the final class confidence for each branch, and a global alignment loss is applied. This can be expressed by the following formula: ; .

[0056] Another difference from strict classification supervision is that it benefits from global alignment loss derived from class feature extraction. By focusing only on the logit confidence of the classes present in the image, more efficient and effective gradient propagation is achieved. Experiments show that direct alignment with CAM features leads to more promising performance. Local alignment loss is further utilized by leveraging class information learned from the anchor branch. This is used to transfer information about newly discovered objects from the erased image to guide anchor branch learning. Local alignment loss. This can be expressed by the following formula: .

[0057] A global loss was designed using the global image and the local image q. To guide students in online training. In the filtered local semantics... and knowledge of the occurrence category A diversity comparison was made between them, among which This refers to the number of patches. Global loss. It is achieved in the following way: ; in, Calculate the number of pairs between patches and prototypes. Is with Belonging to the same category The original prototype, It is a temperature factor.

[0058] The local teacher network can extract local semantic information and extract the initial feature map Fx from the backbone network. Multi-scale bidirectional convolutions capture multi-scale global contextual information through convolutional kernels of different sizes (k1, k2, k3, k4). Each branch uses parallel row and column convolutions, followed by element-wise addition to fuse the extracted features. Channel attention (CA) is then used to enhance object-related feature channels and suppress unimportant feature channels. To help the student model better understand the detailed features of long-tailed instances, a local loss is designed using category labels and historical patch semantics. Semantics with the same category labels are considered positive pairs, while noise patch semantics are excluded. This utilizes query semantics (q) and local embeddings. The reservoir paired with the tag is defined as: ; The current tagged queries and key embeddings are represented as follows: and The earliest ones are respectively and Among them, queues It is the historical local semantics paired with tags in the library, and N is the library capacity.

[0059] Using loss to monitor the above process, the local loss is expressed as: ; in It is a correction mask that excludes noise patches. N is the number of pairs, and N is the number of patches. It is a temperature factor.

[0060] In this embodiment, a knowledge transfer loss function is constructed between the synthetic image and the class feature vector map extracted from the head class image. The classification loss function is calculated from the image-level labels and classification scores. A global alignment loss function is calculated between the class feature vector maps extracted from the occluded image and the head class image. and local alignment loss function A global loss function is constructed between the global teacher network and the student segmentation network. A local loss function is constructed between the local teacher network and the student segmentation network. And the final segmentation loss function of the student segmentation network. Network performance is optimized through feedback. The total loss is expressed as: .

[0061] During training, the label correction mechanism can be used to correct noisy labels, thereby reducing the impact of noise on model training and optimizing network performance.

[0062] Example 2 like Figure 6 As shown, this embodiment provides a long-tail scene instance segmentation device 200, including: Data acquisition module 201 is used to acquire long-tailed distributed datasets, perform data augmentation on the long-tailed distributed datasets, and obtain tail-extended datasets. Image acquisition module 202 is used to acquire multiple head class images and multiple tail class images based on a long-tail distribution dataset; The image synthesis module 203 is used to stitch any tail class image onto each head class image to generate a synthesized image corresponding to each head class image; The first network training module 205 is used to train the preset global teacher network based on each head class image and the corresponding synthetic image of each head class image, and generate pseudo-labels corresponding to each head class image. The second network training module 206 is used to collaboratively train a preset local teacher network and a preset student segmentation network based on the pseudo-labels and tail extended datasets corresponding to each head class image to obtain a target segmentation network, which is then used to perform instance segmentation on the image to be segmented.

[0063] Optionally, in an embodiment, the data acquisition module 201 includes: The dataset acquisition unit is used to acquire an image dataset, which includes multiple first categories, and each first category contains multiple object images. The mask fragment extraction unit is used to segment each object image in the first category using a co-segmentation algorithm for each first category, and obtain the target object mask fragment corresponding to each object image in the first category. The pseudo bounding box generation unit is used to obtain the pseudo bounding box corresponding to each object image in the first category for each first category; The target fragment acquisition unit is used to determine multiple target mask fragments based on the pseudo bounding boxes and target object mask fragments corresponding to each object image in each first category. The data expansion unit is used to paste at least one target mask fragment onto each image in the long-tailed distribution dataset to generate a tail-expanded dataset.

[0064] Optionally, in the embodiments, for each object image in each first category, the pseudo-boundary box corresponding to the object image is obtained in the following manner: S11. According to the preset removal order, remove a portion of the object image to obtain the removed image; S12. Classify the removed image using a pre-trained classifier to obtain the second category corresponding to the removed image; S13. Determine whether the second category corresponding to the removed image is the same as the first category corresponding to the object image. If they are the same, execute S11; otherwise, determine the pseudo bounding box corresponding to the object image based on the current partial region.

[0065] Optionally, in an embodiment, the target fragment acquisition unit includes: The overlap calculation subunit is used to calculate the overlap between the target object mask fragment and the pseudo bounding box corresponding to each object image in the first category for each first category, so as to obtain the overlap corresponding to each object image in the first category. The segment selection subunit is used to obtain the target mask segment in the first category for each first category based on the degree of overlap of each object image in the first category; The target fragment acquisition subunit is used to obtain multiple target mask fragments based on the target mask fragments in each first category.

[0066] Optionally, in the embodiments, the preset global teacher network, the preset local teacher network, and the preset student segmentation network share a backbone network, which is a ResNet-38 backbone network.

[0067] Optionally, in an embodiment, the first network training module 205 includes: The category label acquisition unit is used to acquire the category label corresponding to each of the head class images; The first feature extraction module is used to extract features for each head class image by means of the preset global teacher network based on the head class image and the corresponding synthetic image, so as to obtain the synthetic class activation map features and anchor point class activation map features corresponding to the head class image. The occlusion processing module is used to perform occlusion processing on each head class image according to the anchor point class activation map features corresponding to the head class image, and generate an occlusion image corresponding to the head class image. The second feature extraction module is used to extract features for each head class image by using the preset global teacher network based on the occlusion image corresponding to the head class image; The information generation unit is used to generate image category knowledge and pseudo-labels corresponding to each head class image based on the category label, synthetic activation map features, anchor activation map features and occlusion activation map features corresponding to the head class image.

[0068] Optionally, in an embodiment, the second network training module 206 includes: The local information extraction unit is used to extract local semantic information for each image in the tail extension dataset through a pre-set local teacher network, and generate local semantic knowledge corresponding to each image in the tail extension dataset. The network training unit is used to train the pre-set student segmentation network based on the pseudo-labels corresponding to each head class image, the tail extended dataset, the image category knowledge corresponding to each head class image, and the local semantic knowledge corresponding to each image in the tail extended dataset, to obtain the target segmentation network.

[0069] In some embodiments, the long-tail scene instance segmentation device 200 of the present invention can be implemented in a combination of hardware and software. As an example, the long-tail scene instance segmentation device 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the long-tail scene instance segmentation method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0070] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0071] Example 3 like Figure 7 As shown, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a long-tail scene instance segmentation method as described in Embodiment 1.

[0072] In other words, an electronic device according to an embodiment of the present invention may include, but is not limited to, a processor and a memory; the memory is used to store a computer program; the processor is used to execute a long-tail scene instance segmentation method shown in any embodiment of the present invention by calling the computer program.

[0073] In one alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may further include a transceiver 304, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present invention.

[0074] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0075] Bus 302 may include a path for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus 302 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0076] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0077] The memory 303 is used to store application code (computer program) for executing the present invention, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0078] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0079] It should be noted that, Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0080] Example 4 This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to execute a long-tail scene instance segmentation method as described in Embodiment 1.

[0081] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0082] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned long-tail scene instance segmentation method.

[0083] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0086] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0087] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0088] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0089] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0090] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for segmenting long-tail scene instances, characterized in that, include: Obtain a long-tailed distribution dataset, perform data augmentation on the long-tailed distribution dataset, and obtain a tail-expanded dataset; Based on the long-tail distribution dataset, obtain multiple head class images and multiple tail class images; Each of the head class images is stitched together with any of the tail class images to generate a composite image corresponding to each of the head class images; Based on each of the head class images and the corresponding synthesized images, a preset global teacher network is trained to generate pseudo-labels for each of the head class images. Based on the pseudo-labels corresponding to each head class image and the tail extended dataset, a preset local teacher network and a preset student segmentation network are co-trained to obtain a target segmentation network, which is then used to perform instance segmentation on the image to be segmented.

2. The long-tail scene instance segmentation method according to claim 1, characterized in that, The process of augmenting the long-tailed distribution dataset to obtain a tail-expanded dataset includes: Obtain an image dataset, which includes multiple first categories, and each first category contains multiple object images; For each of the first categories, the object images in the first category are segmented using a co-segmentation algorithm to obtain the target object mask fragment corresponding to each object image in the first category; For each of the first categories, obtain the pseudo bounding box corresponding to each object image in the first category; Based on the pseudo bounding box and target object mask fragment corresponding to each object image in each of the first categories, multiple target mask fragments are determined; At least one target mask fragment is pasted onto each image in the long-tailed distribution dataset to generate the tail-extended dataset.

3. The long-tail scene instance segmentation method according to claim 2, characterized in that, For each object image in each of the first categories, the pseudo-boundary box corresponding to the object image is obtained in the following manner: S11. According to the preset removal order, remove a portion of the object image to obtain a removed image; S12. Classify the removed image using a pre-trained classifier to obtain the second category corresponding to the removed image; S13. Determine whether the second category corresponding to the removed image is the same as the first category corresponding to the object image. If they are the same, then execute S11. Otherwise, the pseudo bounding box corresponding to the object image is determined based on the current partial region.

4. The long-tail scene instance segmentation method according to claim 2, characterized in that, The step of determining multiple target mask fragments based on the pseudo bounding boxes and target object mask fragments corresponding to each of the object images in each of the first categories includes: For each of the first categories, the overlap between the target object mask fragment and the pseudo bounding box corresponding to each object image in the first category is calculated to obtain the overlap corresponding to each object image in the first category; For each of the first categories, the target mask fragment in the first category is obtained according to the degree of overlap corresponding to each object image in the first category; Multiple target mask fragments are obtained based on the target mask fragments in each of the first categories.

5. The method for segmenting long-tail scene instances according to claim 1, characterized in that, The preset global teacher network, the preset local teacher network, and the preset student segmentation network share a backbone network, which is a ResNet-38 backbone network.

6. The method for segmenting long-tail scene instances according to claim 1, characterized in that, The step of training a preset global teacher network based on each of the head class images and the corresponding synthesized images to generate pseudo-labels for each of the head class images includes: Obtain the category label corresponding to each of the head class images; For each head class image, based on the head class image and the corresponding synthetic image, feature extraction is performed through the preset global teacher network to obtain the synthetic class activation map features and anchor class activation map features corresponding to the head class image. For each head-type image, the head-type image is masked according to the anchor point class activation map features corresponding to the head-type image to generate a masked image corresponding to the head-type image. For each head class image, based on the occlusion image corresponding to the head class image, feature extraction is performed through the preset global teacher network to obtain the occlusion class activation map features corresponding to the head class image; For each head-type image, image category knowledge and pseudo-labels are generated based on the category label, synthetic activation map features, anchor activation map features, and occlusion activation map features corresponding to the head-type image.

7. The method for segmenting long-tail scene instances according to claim 6, characterized in that, The step of co-training a preset local teacher network and a preset student segmentation network based on the pseudo-labels corresponding to each of the head class images and the tail extended dataset to obtain the target segmentation network includes: For each image in the tail-extended dataset, local semantic information is extracted through the preset local teacher network to generate local semantic knowledge corresponding to each image in the tail-extended dataset. Based on the pseudo-labels corresponding to each head class image, the tail extended dataset, the image category knowledge corresponding to each head class image, and the local semantic knowledge corresponding to each image in the tail extended dataset, the preset student segmentation network is trained using a label correction mechanism to obtain the target segmentation network.

8. A long-tail scene instance segmentation device, characterized in that, include: The data acquisition module is used to acquire a long-tailed distributed dataset, perform data augmentation on the long-tailed distributed dataset, and obtain a tail-extended dataset. The image acquisition module is used to acquire multiple head class images and multiple tail class images based on the long-tail distribution dataset; An image synthesis module is used to stitch any of the tail class images onto each of the head class images to generate a synthesized image corresponding to each of the head class images; The first network training module is used to train a preset global teacher network based on each of the head class images and the corresponding synthetic images of each of the head class images, and to generate pseudo-labels corresponding to each of the head class images. The second network training module is used to collaboratively train a preset local teacher network and a preset student segmentation network based on the pseudo-labels corresponding to each head class image and the tail extended dataset to obtain a target segmentation network, so as to perform instance segmentation on the image to be segmented based on the target segmentation network.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a long-tail scene instance segmentation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the long-tail scene instance segmentation method according to any one of claims 1 to 7.

Images