Incremental small sample instance segmentation method based on transfer learning

Abstract

The application discloses an incremental small sample instance segmentation method based on transfer learning, comprising the following steps: constructing an incremental small sample instance segmentation model, wherein the incremental small sample instance segmentation model comprises a feature extractor, a Transformer encoder, a Transformer decoder and two prediction heads connected in sequence; training the incremental small sample instance segmentation model; and using the trained incremental small sample segmentation model to segment foreground objects of base classes and new classes in an image to obtain the class and segmentation mask of each foreground object; the application adds a lightweight adapter and adopts a knowledge distillation strategy, aiming to overcome the problems that previous methods cannot fully utilize base class knowledge and the performance of old classes decreases when fine-tuning the model, and better integrate base class knowledge and new class knowledge, and better balance the stability-plasticity dilemma.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of instance segmentation, and particularly relates to an incremental few-shot instance segmentation method based on transfer learning. BACKGROUND

[0002] Instance segmentation is one of the basic tasks of computer vision, which combines the core elements of object detection and semantic segmentation. Object detection aims to predict the class and location of each foreground object in an image, where the location is given in the form of a bounding box. In contrast, semantic segmentation focuses on assigning a semantic class to each pixel in an image. Instance segmentation is more complex and challenging than both object detection and semantic segmentation. It aims to segment each foreground object in an image and distinguish different objects belonging to the same class. The output of instance segmentation typically includes the class and segmentation mask of each foreground object.

[0003] With the continuous progress of deep learning, deep learning-based instance segmentation models have achieved significant performance improvements and have become the mainstream solution for instance segmentation tasks. However, when applying these models in real-world complex scenarios, there are still many challenges. First, the excellent performance of deep learning-based instance segmentation models relies on training on high-precision, large-scale annotated image datasets. However, in the real world, the difficulty of obtaining large-scale image datasets and the high cost of manual annotation limit the further application of these models. Second, data in the real world often comes in a continuous stream, requiring models to have the ability to continuously learn new knowledge without accessing past data. Unfortunately, without the ability to access previous data for unified training, current instance segmentation models often forget previously learned old knowledge while learning new knowledge, leading to catastrophic forgetting of old classes and a sharp deterioration in segmentation performance on old classes. To address these challenges, researchers have proposed incremental few-shot instance segmentation.

[0004] Incremental few-shot instance segmentation aims to quickly learn how to segment new classes using a small number of new class samples without accessing previous samples, while maintaining the performance of the model on old classes to prevent catastrophic forgetting of old classes. The incremental few-shot instance segmentation task has three main challenges: (1) catastrophic forgetting on old classes; (2) overfitting caused by a small number of new class samples; (3) the model needs to balance between maintaining old class performance and learning new classes, which is known as the stability-plasticity dilemma, where the ability to maintain old class performance is called stability, and the ability to learn new classes is called plasticity.

[0005] In the incremental few-shot instance segmentation task, the entire dataset class C is usually divided into two disjoint sets: base classes C base and new classes C noveli.e.C base ∪C novel =C, where each class in the base classes has a large number of labeled samples available for training, while each class in the new classes has only a small number of K labeled samples available for training (usually K = 1, 5, 10), and the base classes C base consisting of samples are denoted as the base dataset D base , and the new classes C novel consisting of samples are denoted as the new dataset D novel . The task of incremental few-shot instance segmentation is to train a model that first trains on D base containing a large number of labeled samples, and then trains on D base without touching the samples in D nopvel containing only a small number of samples, and finally the model performs well on C test =C base ∪C novel .

[0006] Existing incremental few-shot instance segmentation methods can be mainly divided into two categories, namely the transfer learning-based method and the meta-learning-based method. The transfer learning-based method, such as iMTFA and iFS-RCNN, usually adopts the “pre-training-fine-tuning” method, which trains the model on the base dataset with a large number of labeled samples, and then trains the model on the new dataset by freezing the model class-independent components and fine-tuning the model class-specific components to make the model learn new knowledge while preventing the model from catastrophic forgetting on the old classes. The meta-learning-based method, such as Reft, uses the meta-learning method to make the model quickly adapt to new classes. This method usually adopts the scenario training method during training, which sets a series of scenarios E i =(I i , S i ), each of which contains a training set called support set and a test set called query set. By this way, the model can quickly learn new class knowledge on the new dataset by simulating the small sample situation on the new classes during training.

[0007] However, these methods still have some limitations. In the transfer learning-based method, the “pre-training-fine-tuning” method is used to learn new class knowledge, which on the one hand damages the original base class knowledge in the model when fine-tuning the model class-specific components, inevitably causing the performance of the model on the base classes to decline, and on the other hand cannot well integrate the base class knowledge in the model and the newly learned new class knowledge, and fully utilize the original base class knowledge in the model. SUMMARY

[0008] To solve the problems in the prior art, the purpose of the present application is to provide an incremental small sample instance segmentation method based on transfer learning, which adds a lightweight adapter and adopts a knowledge distillation strategy to overcome the problem that previous methods cannot fully utilize base class knowledge and the performance of old classes decreases when fine-tuning the model, and better integrate base class knowledge and new class knowledge, and better balance the stability-plasticity dilemma.

[0009] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is: an incremental small sample instance segmentation method based on transfer learning, comprising the following steps:

[0010] Step 1, constructing an incremental small sample instance segmentation model, the incremental small sample instance segmentation model comprising a feature extractor, a Transformer encoder (Transformer Encoder), a Transformer decoder (Transformer Decoder) and two prediction heads connected in turn;

[0011] Step 2, training the incremental small sample instance segmentation model;

[0012] Step 3, using the trained incremental small sample segmentation model to segment the foreground objects of the base class and the new class in the image to obtain the class and segmentation mask of each foreground object.

[0013] As a further improvement of the present application, in step 1, the feature extractor takes the data-augmented image as input and extracts multi-scale features of the image; the Transformer Encoder takes the multi-scale features output by the feature extractor as input, further enriches the image features through self-attention mechanism, and generates a pixel embedding map using the enriched image features, while providing topk tokens for the Transformer Decoder; the Transformer Decoder takes the topk tokens provided by the Transformer Encoder as the initialization of object queries, realizes the interaction between object queries through self-attention mechanism, realizes the interaction between object queries and image features output by the Transformer Encoder through cross-attention mechanism, and finally outputs the embedding of object queries as the input of two prediction heads; the two prediction heads are classification head and segmentation head respectively, which input the embedding of object queries into the classification head to get the class of each foreground object, and input the embedding of object queries into the segmentation head and dot product the output with the pixel embedding map output by the Transformer Encoder to get the segmentation mask of each foreground object.

[0014] As a further improvement of the present application, the multi-scale features include four scale feature maps with sizes of

[0015] As a further improvement of the present application, the Transformer Encoder first converts the multi-scale feature maps output by the feature extractor in dimension and size through a projection layer, converts the feature map with size to size , converts the feature map with size to size , converts the feature map with size to size and size respectively, then flattens and concatenates the four scale feature maps obtained by conversion as the token input of the Transformer Encoder into the first Transformer Encoder layer; the token output by the Transformer Encoder is used in three places: (1) the token is reshaped into a feature map with size , which is twice up-sampled to size The size of the pixel embedding map is added to the size of the feature map output by the feature extractor to obtain a pixel embedding map. The token is input into a classification head, and topk tokens are selected according to the category confidence to be used as the initialization of the object query in the Transformer Decoder.

[0016] As a further improvement of the present application, the Transformer Encoder is stacked by multiple Transformer Encoder layers, each of which includes a multi-head self-attention and a multi-layer perceptron (MLP).

[0017] As a further improvement of the present application, the projection layer includes four convolutional layers, three of which are used to convert the dimensions of the three feature maps with smaller scales, and one of which is used to convert the dimensions and sizes of the feature map with the smallest scale. The size of the feature map is converted to The size of the feature map is converted to The size of the feature map is converted to The size of the feature map is converted to The size of the feature map is converted to The size of the feature map is converted to The size of the feature map is converted to The size of the feature map is converted to

[0018] As a further improvement of the present application, the pixel embedding map is a tensor with a size of The pixel embedding map is used to perform dot product with the output of the segmentation head to obtain the segmentation result of the foreground object.

[0019] As a further improvement of the present application, the classification head is a cosine similarity-based classifier, and the output is the category of the foreground object. The segmentation head is a three-layer multi-layer perceptron including a hidden layer, and the output is a one-dimensional vector with a dimension of 256. The tensor is upsampled to HxWx1 size and passed through a sigmoid function to obtain a binary segmentation mask of the foreground object.

[0020] As a further improvement of the application, the Transformer Decoder is stacked by multiple Transformer Decoder layers, each of which comprises a multi-head self-attention, a multi-head cross-attention and a multi-layer perceptron (MLP).

[0021] As a further improvement of the application, the training phase in step 2 comprises a base training phase and a small sample incremental fine-tuning phase.

[0022] The base training phase is to train the incremental small sample instance segmentation model on the base class dataset, and the model after training is called the base model.

[0023] The small sample incremental fine-tuning phase specifically comprises the following steps:

[0024] The first step is to freeze all parameters of the base model.

[0025] The second step is to create a new incremental small sample instance segmentation model, which is called a new model, and initialize the new model using the parameters of the base model.

[0026] The third step is to insert a parallel adapter in the multi-layer perceptron of the first layer of the Transformer Encoder of the new model; the adapter is a bottleneck structure comprising a down-sampling layer, a nonlinear activation function ReLU and an up-sampling layer.

[0027] The fourth step is to freeze all parameters of the new model except the classifier and the adapter.

[0028] The fifth step is to train the new model on the new class dataset by the knowledge distillation strategy, taking the base model as the teacher model and the new model as the student model, and calculating the knowledge distillation loss on the token output by the first layer of the Transformer Encoder.

[0029] The sixth step is to add the classification weights of the base model to the classifier of the new model, and the new model after adding the classification weights of the base model is the final model of the small sample incremental fine-tuning phase, which has the ability to detect and segment all foreground objects of the base class and the new class in the image.

[0030] The application has the following beneficial effects:

[0031] 1. Efficient fine-tuning: the application learns new class knowledge by fine-tuning the adapter, and compared with the class-irrelevant components in the fine-tuned model, the application needs to fine-tune a small amount of parameters, which can reduce the time and resources required for fine-tuning on the new class dataset, and at the same time, alleviate the impact on the original model, further alleviate the catastrophic forgetting of the model on the old class.

[0032] 2. Fully Utilize Base Class Knowledge: This invention inserts the adapter into the multilayer perceptron of the first layer of the Transformer Encoder in parallel. Fine-tuning the adapter does not affect the base class knowledge stored in the original multilayer perceptron, and the original base class knowledge is integrated into the newly learned class knowledge, helping the model to better detect new categories. Compared to previous methods, this invention learns new knowledge without compromising base class knowledge and makes more full use of it.

[0033] 3. Overcoming forgetting: This invention reduces interference with old knowledge during the learning process of new categories by adding adapters and adopting knowledge distillation strategies, alleviates the catastrophic forgetting of the model in old categories, and better balances the stability-plasticity dilemma between base classes and new classes. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the incremental small sample instance segmentation model in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of each Transformer Encoder layer in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of each Transformer Decoder layer in the embodiments of the present invention;

[0037] Figure 4 This is a schematic diagram of the two-stage training of the incremental small sample instance segmentation model in the embodiments of the present invention;

[0038] Figure 5 This is a schematic diagram of the Transformer Encoder layer after adding an adapter in an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of the visualization results in the MS COCO2014 dataset with 10 samples in an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0041] like Figure 1 As shown, this invention proposes an incremental few-shot instance segmentation method based on transfer learning. The construction of the incremental few-shot instance segmentation model includes the following steps:

[0042] Step 1: Feature extraction step. Given an initial image, after data augmentation such as random cropping, flipping, and normalization, the image is input into the feature extractor to extract multi-scale features.

[0043] Step 2: Transformer Encoder. The multi-scale features obtained in step 1 are input into the Transformer Encoder. The Transformer Encoder first converts the multi-scale features in dimension and size through the projection layer, then flattens and concatenates the converted multi-scale features into the Transformer Encoder, and further enriches the features through multiple stacked Transformer Encoder layers. The output of the Transformer Encoder includes three parts: (1) pixel embedding map for obtaining the final segmentation result; (2) topk tokens for initializing the object query of the Transformer Decoder; (3) enriched image features.

[0044] Step 3: Transformer Decoder. The Transformer Decoder first initializes the object query using the topk tokens obtained in step 2, and then inputs it into multiple stacked Transformer Decoder layers. In each Transformer Decoder layer, first realize the interaction between object queries through self-attention mechanism, and then realize the interaction between object queries and enriched image features in step 2 through cross-attention mechanism. The embedding of the final output object query of the Transformer Decoder will be input into two prediction heads.

[0045] Step 4: Result prediction. Two prediction heads use the embedding of the object query obtained in step 3 as input to obtain the final segmentation result of the foreground object. Among them, the classification head will output the class of the foreground object; the segmentation head will output a one-dimensional vector of size 256, and then dot product with the pixel embedding map obtained in step 2, and apply the sigmoid function on the dot product result to obtain the binary segmentation mask of the foreground object.

[0046] As shown in Figure 2 each Transformer Encoder layer includes a multi-head self-attention and a multi-layer perceptron. In this embodiment, the Transformer Encoder is stacked by 6 Transformer Encoder layers.

[0047] As shown in Figure 3 Each Transformer Decoder layer includes a multi-head self-attention, a multi-head cross-attention and a multi-layer perceptron. In this embodiment, the Transformer Decoder is stacked by 9 Transformer Decoder layers.

[0048] As shown in Figure 4 The training of the entire model includes a base training phase and a small sample incremental fine-tuning phase. In this embodiment, the training process is as follows:

[0049] (1) Base training phase:

[0050] In the base training phase, the model will be trained on the base training set, and the images in the base training set only contain annotations of base class objects.

[0051] The loss function of the base training phase is:

[0052]

[0053] Among them, is the classification loss, using sigmoid focal loss as the loss function; is the segmentation loss, composed of cross-entropy loss function and dice loss function, that is is the cross-entropy loss function, is the dice loss function.

[0054] The model obtained in the base training phase is called the base model.

[0055] (2) Small sample incremental fine-tuning phase:

[0056] In the small sample incremental fine-tuning phase, the model will be trained on the new class dataset, and the images in the new class dataset only contain annotations of new class objects.

[0057] As shown in Figure 5 In the small sample incremental fine-tuning phase, this embodiment fine-tunes the model on the new class dataset by adding an adapter and adopting a knowledge distillation strategy, so that the model learns new class knowledge without damaging the original old class information in the model, alleviates the catastrophic forgetting of the model on the old class, and better balances the stability-plasticity dilemma.

[0058] First, this embodiment inserts an adapter parallel to the multi-layer perceptron in the Transformer Encoder layer. The Transformer Encoder layer after adding the adapter is as shown in Figure 3As shown, the adapter is a bottleneck structure consisting of a lower projection layer, a ReLU nonlinear activation function, and an upper projection layer. After adding the adapter, the output of this part is:

[0059] x o =σ(x i W down W up +LayerNorm (Multilayer Perceptron (x) i )+x o )

[0060] Among them, W down W represents the lower projection layer. up Let x represent the upper projection layer, σ represent the ReLU function, and x represent the upper projection layer. i x represents the features input into the multilayer perceptron. o This represents the final output feature, and LayerNorm represents layer normalization.

[0061] In this embodiment, an adapter is added only in the first Transformer Encoder layer, and the middle dimension of the adapter is set to 64.

[0062] In the small sample incremental fine-tuning stage, this embodiment uses the base model obtained in the basic training stage as the teacher model in the knowledge distillation strategy, and the new model after inserting the adapter as the student model in the knowledge distillation strategy. At the same time, the parameters of the student model other than the adapter and classifier are initialized with the parameters of the teacher model, and the parameters of the student model other than the adapter and classifier are frozen.

[0063] This embodiment performs knowledge distillation on the token output from the first layer of the Transformer Encoder. The specific steps are as follows:

[0064] Step 1: Reconstruct the token output from the first layer of the Transformer Encoder back into a multi-scale feature map, i.e., shape... The tensors are reshaped into those with sizes of 1 and 2. The four feature maps.

[0065] Step 2: Construct a label mask to identify the base class object region and the new class object region in the feature map. First, downsample the ground truth segmentation mask of the new class object with size H×W×1 in the image using bilinear interpolation to obtain... Four size binary segmentation masks. In this binary segmentation mask, if the value of a pixel point in the segmentation mask is 1, it proves that it is in the new class object region, then its corresponding identification mask is 0, if the value of a pixel point in the segmentation mask is 0, it proves that it is in the base class object region, then its corresponding identification mask is 1. In this way, the identification mask corresponding to the four feature maps is obtained.

[0066] Step 3: Apply the four identification masks obtained in step 2 to the calculation of the knowledge distillation loss, so that the calculation of the knowledge distillation loss is only performed on the base class object region, so as to make the student model base class object region features not deviate too much from the teacher model base class object region features while reducing the constraint on the model learning new class knowledge. The knowledge distillation loss is calculated on the four feature maps through L2 loss, that is:

[0067]

[0068] wherein, flag_mask represents the identification mask, represents the feature map of the new model, represents the feature map of the base model, w i and h i respectively represent the width and height of the feature map f i .

[0069] The total loss of the small sample incremental fine-tuning stage is:

[0070]

[0071] wherein, α is the weight of the knowledge distillation loss, which can be used as a trade-off parameter, and in this embodiment, it is set to 0.05.

[0072] After the model training is completed, the classification weights of the base class in the base model are merged into the classifier of the new model, and at this time the obtained model is the final model, which can segment the base class objects and new class objects appearing in the image.

[0073] In order to verify the performance of the above method, the following experiment is designed.

[0074] ResNet-50 as the backbone, and the experimental verification is carried out on two benchmark datasets MS COCO2014 and LVIS. MS COCO2014 contains 80 categories in total, among which 20 categories intersecting with the PSCAL VOC dataset are taken as new classes, and the remaining 60 categories are taken as base classes. LVIS contains 1230 categories in total, and according to the frequency of each category appearing in the image, it is divided into rare categories (less than 10 images), common categories (10-100 images), and frequent categories (more than 100 images). In the experiment, the rare categories are taken as new classes, and the common categories and frequent categories are taken as base classes.

[0075] On the MS COCO2014 dataset, the performance of the model on the base class, the new class and all classes is considered when the sample number is 1, 5 and 10. All experimental results are the average results of 10 random samplings. For the MS COCO2014 dataset, the average precision (AP) and the average precision (AP50) with a confidence threshold greater than 0.5 are taken as evaluation indexes; for the LVIS dataset, the average precision (AP) is taken as the evaluation index.

[0076] The experimental results on the MS COCO2014 dataset are shown in Table 1. Under the three sample number settings, the best effect is obtained, and the other methods are greatly exceeded.

[0077] Table 1 Performance comparison of incremental small sample instance segmentation method on MS COCO2014 dataset.

[0078]

[0079] The experimental results on the LVIS dataset are shown in Table 2. It can be seen that on the large dataset with a large number of categories, the method still has great advantages, and the various indexes are greatly exceeded by the other methods.

[0080] Table 2 Performance comparison of incremental small sample instance segmentation method on LVIS dataset

[0081]

[0082] In order to further verify the effectiveness of the proposed method, three groups of ablation experiments are designed.

[0083] To verify the effectiveness of the method component, ablation experiments were conducted on the important components of the method, and the results are shown in Table 3. In Table 3, in order to verify that the fine-tuning of the lightweight adapter inserted in parallel is a better choice than fine-tuning the remaining model components, the experimental results of fine-tuning the projection layer and the fine-tuning adapter are compared. It can be seen that fine-tuning the adapter can better alleviate the catastrophic forgetting of the model on the base class, while achieving competitive results on the new class as the fine-tuning projection layer. Therefore, fine-tuning the adapter can achieve a better balance between effectively preserving old knowledge and learning new knowledge, and is a better choice than fine-tuning the projection layer. By comparing the experimental results in the 2nd and 3rd rows of the table, it can be seen that knowledge distillation can further alleviate catastrophic forgetting with a slight decrease in performance on the new class.

[0084] Table 3 Ablation results of model components on MS COCO2014 dataset with sample size of 10

[0085]

[0086] To verify the influence of the intermediate dimension of the adapter on the performance of the model, the performance of the model on the base class and the new class under different intermediate dimensions of the adapter was tested, and the results are shown in Table 4. From Table 4, it can be seen that when the intermediate dimension decreases and the parameters that can be fine-tuned gradually decrease, the performance of the model on the base class increases, and the performance on the new class decreases. Therefore, the intermediate dimension of the adapter can be used as a trade-off parameter to be selected according to the actual situation.

[0087] Table 4 Ablation results of intermediate dimension of adapter on MS COCO2014 dataset with sample size of 10

[0088]

[0089] To verify the influence of the weight parameter a of the knowledge distillation loss on the performance of the model, the performance of the model under different weight parameter values was tested, and the results are shown in Table 5. The greater the weight parameter value, the more restrictive the knowledge distillation is on the model parameter update, which is more conducive to alleviating the performance decline of the model on the base class, but also limits the learning of new knowledge of the model on the new class. Therefore, the weight parameter value can also be used as a trade-off parameter to be selected according to the actual situation.

[0090] Table 5 Ablation results of weight parameter a of knowledge distillation loss on MS COCO2014 dataset with sample size of 10

[0091]

[0092]

[0093] In order to further show the effect of the embodiment, a visual experiment is performed on the MS COCO 2014 dataset under the setting of 10 samples, and results are shown in Figure 6 As can be seen from Figure 6 , the embodiment can well segment and classify the base class and new class objects in the image.

[0094] The above-described embodiments only express specific implementation manners of the present application, which are described in detail, but should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application.

Claims

1. An incremental few-shot instance segmentation method based on transfer learning, characterized in that, Includes the following steps: Step 1: Construct an incremental few-shot instance segmentation model, which includes a feature extractor, a Transformer encoder, a Transformer decoder, and two prediction heads connected in sequence. In step 1, the feature extractor takes the data-enhanced image as input and extracts multi-scale features from the image. The Transformer encoder takes the multi-scale features output by the feature extractor as input, further enriches the image features through a self-attention mechanism, and generates a pixel embedding map using the enriched image features, while providing the Transformer decoder with the selected first k tokens. The Transformer decoder uses the first k tokens provided by the Transformer encoder as initialization for its object queries, realizes the interaction between object queries through a self-attention mechanism, realizes the interaction between object queries and the image features output by the Transformer encoder through a cross-attention mechanism, and finally outputs the object query embedding as input to two prediction heads. The two prediction heads are a classification head and a segmentation head. The object query embedding is input into the classification head to obtain the category of each foreground object, and the object query embedding is input into the segmentation head. Its output is then multiplied by the pixel embedding map output by the Transformer encoder to obtain the segmentation mask of each foreground object. Step 2: Train the incremental small sample instance segmentation model; The training phase in step 2 includes a basic training phase and a small-sample incremental fine-tuning phase: The basic training phase involves training an incremental few-sample instance segmentation model on a base class dataset, and the model after training is referred to as the basic model. The small sample incremental fine-tuning stage specifically includes the following steps: Step 1: Freeze all parameters of the base model; The second step is to create a new incremental few-sample instance segmentation model, which is called the new model, and initialize the new model using the parameters of the base model. The third step is to insert a parallel adapter into the multilayer perceptron of the first layer of the Transformer encoder in the new model; the adapter is a bottleneck structure, including a downsampling layer, a nonlinear activation function ReLU, and an upsampling layer. Step 4: Freeze all parameters of the new model except for the classifier and adapter; Step 5: Train a new model on the new class dataset using the knowledge distillation strategy. Use the base model as the teacher model and the new model as the student model. Calculate the knowledge distillation loss on the tokens output by the first layer of the Transformer encoder. Step 6: Add the classification weights of the base model to the classifier of the new model. The new model after adding the classification weights of the base model is the final model in the small sample incremental fine-tuning stage, which has the ability to detect all base class and new class foreground objects in the segmented image. Step 3: Use the trained incremental few-shot segmentation model to segment the foreground objects of the base class and new class in the image to obtain the category and segmentation mask of each foreground object.

2. The incremental few-shot instance segmentation method based on transfer learning according to claim 1, characterized in that, The multi-scale features include feature maps at four scales, with sizes of [sizes to be filled in].

3. The incremental few-shot instance segmentation method based on transfer learning according to claim 2, characterized in that, The Transformer encoder first transforms the dimensions and size of the multi-scale feature map output by the feature extractor through a projection layer, reducing the size of the feature map to a specific value. Feature map conversion Size, make the size of Feature map conversion The size is The feature maps are converted into Size and The size is determined, and then the feature maps of the four scales obtained by the transformation are flattened and connected to serve as the token of the Transformer encoder, which is then input into the first Transformer encoder layer; the token finally output by the Transformer encoder is used in three places: (1) the token is reshaped into a feature map, and its value is used in the transformation. The feature map of the same size is upsampled by twice. Size, and compare it with the size of the output in the feature extractor. (1) Add the feature maps together to obtain the pixel embedding map; (2) Input these tokens into the classification head, select the first k tokens according to the category confidence, and use them as the initialization of object query in the Transformer decoder; (3) Input the tokens as enhanced image features into the Transformer decoder.

4. The incremental few-shot instance segmentation method based on transfer learning according to claim 3, characterized in that, The Transformer encoder is composed of multiple Transformer encoder layers stacked together, and each Transformer encoder includes a multi-head self-attention and a multilayer perceptron.

5. The incremental few-shot instance segmentation method based on transfer learning according to claim 3, characterized in that, The projection layer includes four convolutional layers, three of which perform dimensionality transformation on the three smaller feature maps, and one convolutional layer performs dimensionality and size transformation on the smallest feature map; the four convolutional layers are as follows: (1) The convolutional kernel size is 1×1, and the stride is 1, used to transform the feature map into the smallest feature map. Size of feature map converted to Size; (2) The kernel size is 1×1 with a stride of 1, used to convert the kernel size into a single kernel. Size of feature map converted to Size; (3) The kernel size is 1×1 with a stride of 1, used to convert the kernel size into a single kernel. Size of feature map converted to Size, (4) The kernel size is 3×3 with a stride of 2, used to convert the kernel size into a single kernel. Size of feature map converted to size.

6. The incremental few-shot instance segmentation method based on transfer learning according to claim 3, characterized in that, The pixel embedding image is of size . The tensor is used to perform a dot product with the output of the segmentation head to obtain the segmentation result of the foreground object.

7. The incremental few-shot instance segmentation method based on transfer learning according to claim 1, characterized in that, The classification head is a cosine similarity-based classifier, whose output is the category of the foreground object; the segmentation head is a three-layer multilayer perceptron, including one hidden layer, whose output is a 256-dimensional one-dimensional vector. This vector is then multiplied by the pixel embedding map generated by the Transformer encoder to obtain the segmentation head. The tensor is upsampled to size H×W×1 and then passed through the sigmoid function to obtain the binary segmentation mask of the foreground object.

8. The incremental few-shot instance segmentation method based on transfer learning according to claim 3, characterized in that, The Transformer decoder is composed of multiple stacked Transformer decoder layers. Each Transformer decoder layer includes a multi-head self-attention layer, a multi-head cross-attention layer, and a multilayer perceptron.

Images