Prototype enhanced small sample segmentation method based on category perception

Abstract

The application discloses a kind of based on class perception's prototype enhancement small sample segmentation method, data set is evenly divided into 4 subsets, each subset is taken turns as verification set, 1000 pairs of support-query image are randomly extracted from it for verification link, and other three subsets are used as training set to cross validation.Small sample segmentation network includes encoder, base learning ware, class perception module CAM, prototype enhancement module PEM, encoder and refining module.CAM is based on support feature and support mask, can perceive target class and non-target class simultaneously, and learn class weight from this, so that network pays more attention to target class area.PEM extracts support prototype simultaneously based on query image initial prediction extraction query prototype, generates mixed enhanced prototype by fusing two kinds of prototypes, simultaneously contains the information of both, improves the quality of final segmentation result.The application guides network to better avoid false activation when the number of support samples is limited, and more accurately separates out target class.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision and image processing, and particularly relates to a prototype enhancement small sample segmentation network based on category perception and a segmentation method thereof. BACKGROUND

[0002] With the continuous development of deep neural networks, especially the introduction of fully convolutional network (FCN) architecture, semantic segmentation has made significant progress in the field of image processing. Since the fully supervised semantic segmentation method [1,2] requires a large amount of labeled data, small sample segmentation technology has received widespread attention from researchers in recent years.

[0003] In the small sample segmentation task, the used dataset is divided into a support set and a query set, and the goal is to segment the query set image under the condition that one or more labeled images from the support set are known. In existing small sample semantic segmentation methods, the method based on prototype learning has good generalization performance and has become one of the mainstream methods in small sample segmentation tasks. Its advantage lies in that by learning the prototype of the category, the model can better adapt to new categories, so that the model can still achieve satisfactory segmentation effect under the condition of limited labeled data. However, the existing method [3-5] extracts target class information of support images to guide query images, and the target class information may be disturbed by non-target class information due to the limited number of support samples.

[0004] [Reference]

[0005] [1] Huang Z, Wang X, Huang L, et al. Ccnet: Criss-cross attention for semantic segmentation [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 603-612.

[0006] [2] Li X, Li X, Zhang L, et al. Improving semantic segmentation via decoupled body and edge supervision [C] / / Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16. Springer International Publishing, 2020: 435-452.

[0007] [3] Zhang B, Xiao J, Qin T. Self-guided and cross-guided learning for few-shot segmentation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 8312-8321.

[0008] [4] Liu W, Zhang C, Lin G, et al. Crnet: Cross-reference networks for few-shot segmentation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 4165-4173.

[0009] [5] Liu W, Zhang C, Lin G, et al. CRCNet: Few-shot segmentation with cross-reference and region-global conditional networks [J]. International Journal of Computer Vision, 2022, 130(12): 3140-3157.

[0010] [6] Shaban A, Bansal S, Liu Z, et al. One-shot learning for semantic segmentation [J]. arXiv preprint arXiv: 1709.03410, 2017.

[0011] [7] Nguyen K, Todorovic S. Feature weighting and boosting for few-shot segmentation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 622-631.

[0012] [8] Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects in context [C] / / Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer International Publishing, 2014: 740-755.

[0013] [9] Lang, Chunbo, et al. "Learning what not to segment: A new perspective on few-shot segmentation." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022. SUMMARY

[0014] In view of the defects that the existing prototype learning-based method is easy to be limited by the number of samples when extracting target class information, the application proposes a prototype enhancement small sample segmentation method based on class perception. The method can guide the network to better avoid false activation when the number of support samples is limited, so as to more accurately segment out the target class.

[0015] The present invention proposes a prototype-enhanced small sample segmentation method based on category perception. The prototype-enhanced small sample segmentation network based on category perception includes an encoder, a base learner, a category-aware module CAM, a prototype-enhanced module PEM, an encoder, and a refinement module R. Among them, the category-aware module is based on support features and support masks, and can simultaneously perceive the target class and non-target class, and thereby learn category weights to make the network pay more attention to the target class area. The prototype enhancement module not only extracts support prototypes, but also extracts query prototypes based on the initial prediction of the query image, and fuses the two prototypes to generate a hybrid enhanced prototype, which also contains information from both, thereby improving the quality of the final segmentation result. The method comprises the following steps:

[0016] Step 1) Obtain the PASCAL-5 used for network training and verification i and COCO-20 i The dataset is evenly divided into four subsets according to the amount of image data. For each subset, 1000 pairs of "support-query" images are randomly selected for verification. The support images refer to the labeled samples, and the query images refer to the unlabeled images to be segmented.

[0017] Step 2) Build a category-aware prototype-enhanced small sample segmentation network to segment the query image:

[0018] The network is trained and verified using the training set and verification set obtained in step S1. The network training process includes the following steps:

[0019] 2-1) Support image I for network input s and query image I q , firstly, the encoder Encoder(·) extracts the support feature F s and query feature F q , which is expressed as follows:

[0020] F s =Encoder(I s ) (1)

[0021] F q =Encoder(I q ) (2)

[0022] 2-2) Support feature F s and query feature F q Processing: including:

[0023] Using support feature F s and support mask M s Extract supporting image class information C s ;

[0024] Utilize query feature F q The query image initial prediction result output by the prototype enhancement module PEM Obtain query image class information C q ;

[0025]

[0026] Wherein, CAM(·) represents the category perception module CAM mapping;

[0027] Utilize support feature F s And support mask M s Extract foreground class prototype through mask average pooling MAP operation It is expressed as:

[0028]

[0029] Wherein, i is the image pixel position index, h and w are the feature map height and width respectively; [·] is Iverson bracket, which is equal to 1 if the internal condition is true, otherwise it is equal to 0; M s is a binary mask, M s (i) = 1 indicates that the ith pixel belongs to class c;

[0030] 2-3) The prototype enhancement module PEM utilizes the foreground class prototype And the query feature F q Generate mixed enhancement prototype The mixed enhancement prototype Is used for the activation of the query image target area, expressed as:

[0031]

[0032] Wherein, PEM(·) represents the prototype enhancement module PEM mapping;

[0033] The query feature F q , support image class information C s , query image class information C q And mixed enhancement prototype Carry out channel dimension connection to obtain enhanced query feature That is:

[0034]

[0035] 2-4) Based on the enhanced query feature Generate the prediction result p of the query image target class through the decoder tar It is:

[0036]

[0037] 2-5) using the refinement module RM to refine the prediction result p of the query image target class tar and the base learner processes the query feature F q obtained p base refinement is performed to obtain the final small sample segmentation result p out :

[0038] p out = RM (p tar , p base ) (8)

[0039] Further, the small sample segmentation method of the present application, wherein,

[0040] In step 1), for the four subsets, each subset is in turn taken as the validation set, while the other three subsets are taken as the training set for cross-validation.

[0041] In step 2), the design of the class-aware module CAM is that the class-aware module CAM uses the support feature F s , the support mask M s , the query feature F q and the initial prediction result of the query image obtained from the prototype enhancement module PEM to output the support image class information C s and the query image class information C q ; comprising:

[0042] 1) the support feature F s and the support mask M s are fused by Hadamard product to obtain the support feature after fusing the class information

[0043]

[0044] Wherein, the symbol ⊙ represents the Hadamard product operation, and the symbol ζ(·) represents the bilinear interpolation function; the support image class weight map W is obtained after pooling, convolution and activation operations are performed on the s as follows:

[0045]

[0046] Wherein, Pool(·) represents the pooling operation, Conv(·) represents the convolution network, and σ(·) represents the activation function;

[0047] The support image class information C s is generated using the support image class weight map W s :

[0048]

[0049] 2) For the query feature F q and the initial rough prediction result of the query image Perform Hadamard product operation to obtain query features after integrating category information

[0050]

[0051] Regarding the Perform pooling, convolution and activation operations to obtain the category weight map W of the query image q :

[0052]

[0053] Using the category weight map W of the query image q Generate query image class information C q :

[0054]

[0055] In step 2), the prototype enhancement module PEM includes an initial prediction submodule IPM, and the design of the prototype enhancement module PEM is as follows:

[0056] 1) The foreground prototype Extended to support feature F s Same size supported prototypes Utilize the supporting prototype For query feature F q Activate to get the initial prediction result of the query image The query feature F q and supporting prototypes Input together into the initial prediction submodule IPM to obtain the initial prediction result Expressed as:

[0057]

[0058] in, Denotes the decoder, and the FP(·) operation process is:

[0059]

[0060] 2) For query feature F q and the initial prediction results of the query image Perform masked average pooling MAP operation to extract the query prototype Expressed as:

[0061]

[0062] where i is the image pixel position index, h and w are the height and width of the feature map respectively; [·] is the Iverson bracket, which equals to 1 if the internal condition is true, otherwise equals to 0; represents the i-th pixel belongs to the c class;

[0063] 3) the foreground class prototype is fused with the query prototype in the channel dimension to obtain the mixed enhanced prototype that is:

[0064]

[0065] Compared with the prior art which is limited by the small number of support samples, especially the insufficient activation of the query image when the target class region is small, the prototype enhancement small sample segmentation network based on class perception proposed by the application comprehensively utilizes the target class information extracted from the support sample and the query sample to guide the network to focus on the target class region, and effectively reduces the information loss and error activation problem by integrating the query image information into the support prototype to effectively enhance the prototype quality, thereby improving the small sample segmentation performance. On the standard small sample dataset (PASCAL-5 i and COCO-20 i ), whether it is a 1-shot task or a 5-shot task, its performance is superior to other small sample segmentation methods. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is the overall architecture of the small sample segmentation network of the application;

[0067] Figure 2 is the class perception module in the application;

[0068] Figure 3 is the prototype enhancement module in the application;

[0069] Figure 4 is the subjective effect of the method of the application on the PASCAL-5 i dataset;

[0070] Figure 5 is the subjective effect of the method of the application on the COCO-20 i dataset. DETAILED DESCRIPTION

[0071] The application proposes a prototype enhancement small sample segmentation network based on class perception, which mainly consists of an encoding and decoding part, a class perception module and a prototype enhancement module. The process of obtaining the final small sample segmentation result along the image processing flow is as follows (as shown inFigure 1 as shown):

[0072] Base dataset preparation:

[0073] The performance evaluation of the present application is based on two widely used small sample segmentation datasets, PASCAL-5 i[6] and COCO-20 i[7] . PASCAL-5 i contains 20 classes in total, proposed by Sha-ban et al.; COCO-20 i is constructed by MSCOCO [8] contains 80 classes in total, COCO-20 i contains more small target and multi-target scenes, which is more consistent with the actual scene and more challenging.

[0074] The present application evenly divides the object classes of the two datasets into four subsets to perform experiments in a cross-validation manner.

[0075] For example, for the PASCAL-5 i dataset, the 4 subsets are respectively denoted as PASCAL-5 0 , PASCAL-5 1 , PASCAL-5 2 , PASCAL-5 3 Each subset is taken in turn as the validation set, while the other three subsets are taken as the training set for cross-validation. During the rotation process, there are one of the following 4 cases:

[0076] When PASCAL-5 0 is taken as the validation set, PASCAL-5 1 , PASCAL-5 2 , PASCAL-5 3 are the training dataset;

[0077] When PASCAL-5 1 is taken as the validation set, PASCAL-5 0 , PASCAL-5 2 , PASCAL-5 3 are the training dataset;

[0078] When PASCAL-5 2 is taken as the validation set, PASCAL-5 0 , PASCAL-5 1 , PASCAL-5 3 are the training dataset;

[0079] When PASCAL-5 3 is taken as the validation set, PASCAL-50 , PASCAL-5 1 , PASCAL-5 2 is a training data set.

[0080] For each subset as a validation set, 1000 pairs of "support-query" image pairs are randomly extracted from it for validation evaluation, the support image refers to a sample with a label, and the query image refers to an unlabeled image to be segmented.

[0081] The class-aware prototype enhancement small sample segmentation network:

[0082] The overall structure of the small sample segmentation network proposed by the application is shown in Figure 1 The small sample segmentation network comprises an encoder, a base learner, a class-aware module CAM, a prototype enhancement module PEM, an encoder and a refinement module RM; and the network is trained and verified by using the training set and the validation set obtained in step 1.

[0083] The application mainly embodies the class-aware module and the prototype enhancement module. After encoding and feature extraction, the class-aware module is used to fully obtain target class information by using support features and support masks, query features and query predictions, and effectively enhance the attention of the model to the target class; then the prototype enhancement module is used to extract query prototypes and support prototypes based on the initial prediction of the query image, and a hybrid enhanced prototype is generated by fusing the query prototypes and the support prototypes, so that the target region of the query image is activated, and information loss and error activation can be significantly reduced; after network processing, the final segmentation result is obtained by decoding. The overall analysis and module analysis of the network are as follows:

[0084] 1) Overall analysis of the network:

[0085] Referring to the overall structure of the network Figure 1 , for a given support image I s and a query image I q , first, the support features F s and the query features F q are extracted by the encoder Encoder(·), which can be represented as:

[0086] F s =Encoder(I s )

[0087] F q =Encoder(I q )

[0088] In order to improve the attention of the network to the target class while suppressing the interference of non-target class information, a class-aware module is designed, which on the one hand uses the support features F s and the support mask M sExtracting support image class information C s On the other hand, in the process of utilizing query feature F q and prototype enhancement module PEM to obtain query image initial prediction result , query image class information C q is also obtained from query samples, which can be expressed as:

[0089]

[0090] Wherein, CAM(·) represents the category-aware module mapping, which guides the network to focus on the target class region based on the two kinds of category information.

[0091] On the other hand, support feature F s and support image ground truth M s extract foreground class prototype through mask average pooling MAP operation , which can be expressed as:

[0092]

[0093] Wherein, i is the image pixel position index, h and w are the height and width of the feature map respectively. [·] is Iverson bracket, which is equal to 1 if the internal condition is true, otherwise it is equal to 0. M s is a binary mask, and M s (i) = 1 indicates that the ith pixel belongs to class c.

[0094] In order to overcome the inherent differences between support image and query image encoding features, a prototype enhancement module PEM is proposed in the application. Based on the foreground class prototype obtained, the query feature F q is used to obtain query image initial prediction, and the two prototypes are fused to generate a hybrid enhanced prototype (see the description in the prototype enhancement module design), which can be expressed as: Hybrid enhanced prototype contains both support image information and query image information, which can significantly reduce information loss and false activation when activating the target region of the query image, and can be expressed as:

[0095]

[0096] Wherein, PEM represents the prototype enhancement module mapping.

[0097] Then, the query feature F q , support image class information C s , query image class information C q and hybrid enhanced prototype The channel dimension is connected to obtain an enhanced query feature That is,

[0098]

[0099] Based on the enhanced query feature The prediction result p of the query image target class is generated through the decoder tar For:

[0100]

[0101] Next, the prediction result p of the query image target class is refined by using a refine module (RM) tar And the base learner processes the query feature F q The obtained p base Is refined to obtain the final small sample segmentation result p out :

[0102] p out = RM (p tar , p base )

[0103] 2) Category-aware module CAM:

[0104] The category-aware module CAM designed in the application has the structure as shown in Figure 2 The CAM uses the support feature F s , the support mask M s , the query feature F q And the initial prediction result of the query image obtained from the prototype enhancement module PEM Output to obtain the support image category information C s And the query image category information C q ; including:

[0105] In the CAM, the support feature F s And the support mask M s Through Hadamard product, the support feature after fusing the category information is obtained

[0106]

[0107] Wherein, the symbol ⊙ represents the Hadamard product operation, and the symbol ζ (·) represents the bilinear interpolation function. After the following pooling, convolution and activation operations are performed on The category weight map W of the support image is obtained s :

[0108]

[0109] Among them, Pool(·) represents the pooling operation, Conv(·) represents the convolutional network, and σ(·) represents the activation function.

[0110] Using the class weight map W of the support image s Generate support image class information C s :

[0111]

[0112] Similarly, for the query feature F q and the initial rough prediction result of the query image Perform Hadamard product operation to obtain query features after integrating category information

[0113]

[0114] Then, Perform the following pooling, convolution and activation operations to obtain the category weight map W of the query image: q :

[0115]

[0116] Finally, using the class weight map W q Generate query image class information C q :

[0117]

[0118] Thus, we get the supporting image information C s And query image information C q .

[0119] 3) Prototype Enhancement Module PEM:

[0120] The prototype enhancement module PEM includes an initial prediction submodule IPM. The structure of the prototype enhancement module PEM is as follows: Figure 3 As shown. In PEM, first the foreground class prototype Extended to support feature F s Same size supported prototypes Leverage support prototypes For query feature F q Activate to get the initial prediction result of the query image like Figure 3 As shown. The query feature F q and supporting prototypes Input together into the Initial Prediction Module (IPM) to obtain the initial prediction result It can be expressed as:

[0121]

[0122] in, Denotes the decoder, and the FP(·) operation process is:

[0123]

[0124] Get the initial prediction result of the query image After that, the query feature F q and initial prediction results Perform masked average pooling MAP operation to extract the query prototype It can be expressed as:

[0125]

[0126] Where i is the image pixel index, h and w are the height and width of the feature map, respectively. [·] is an Iverson bracket, which is equal to 1 if the inner condition is true, and 0 otherwise. Indicates that the i-th pixel belongs to class c.

[0127] Then, the foreground class prototype and query prototypes Perform channel-dimensional fusion to obtain a hybrid enhanced prototype Right now:

[0128]

[0129] 4) Loss function:

[0130] In this paper, the binary cross entropy loss (BCE) is used to train the model and calculate the predicted mask p of the query image. tar The corresponding true value mask M tar BCE loss between.

[0131]

[0132] n epi Indicates the number of training batches. In addition, the base learner loss and refinement module loss are also used. [9] Perform end-to-end training.

[0133] 5) Research Materials:

[0134] The method proposed in the present application is compared with eight small sample segmentation methods in the prior art, including RARE (2023), FECANet (2023), BAM (2022), DPCN (2022), APANet (2022), NTRENet (2022), HSNet (2021), HFA (2021) and the like.

[0135] Figure 4 and Figure 5 The segmentation effect of the proposed method on two standard small sample segmentation data sets (PASCAL-5i and COCO-20i) is directly shown. The first row shows the support image and its real mask, the second row shows the query image and its real mask, the third row shows the segmentation result of the baseline method, and the fourth row shows the subjective segmentation prediction of the method of the present application.

[0136] Table 1 and the continuation of Table 1 give the mIoU comparison of each method on PASCAL-5 i The bold and underlined results represent the best, and the italic and underlined results represent the second.

[0137] Table 2 and the continuation of Table 2 give the mIoU comparison of each method on COCO-20 i The bold and underlined results represent the best, and the italic and underlined results represent the second.

[0138] Table 1

[0139]

[0140]

[0141] Continuation of Table 1

[0142]

[0143]

[0144] Table 2

[0145]

[0146] Continuation of Table 2

[0147]

[0148]

[0149] It can be seen that the method of the present application achieves the most advanced performance on 1-shot and 5-shot tasks. When VGG16 is used as the backbone network, the method of the present application achieves the best performance on PASCAL-5 iThe mIoU index on the dataset is improved by 0.34% (1-shot) and 0.45% (5-shot) respectively; on the COCO-20 i The index improvement on the dataset is more obvious, and the mIoU index is improved by 1.30% (1-shot) and 2.07% (5-shot) than the second place; when ResNet50 is used as the backbone network, the method of the present application is superior to the second place by 0.67% (1-shot) and 1.31% (5-shot) on the PASCAL-5 i The mIoU index on the dataset is improved by 0.67% (1-shot), 1.31% (5-shot), and the method of the present application is superior to the second place by 2.25% (1-shot) and 2.32% (5-shot) on the COCO-20 i The mIoU index on the dataset is improved by 0.67% (1-shot), 1.31% (5-shot), and the method of the present application is superior to the second place by 2.25% (1-shot) and 2.32% (5-shot) on the COCO-20

[0150] Table 3 and Table 4 show the performance comparison of the method of the present application and other advanced methods in terms of FB-IoU.

[0151] Table 3 shows the FB-IoU comparison of each method on the PASCAL-5 i The bold and underlined results represent the best, and the italic and underlined results represent the second.

[0152] Table 3

[0153]

[0154]

[0155] Table 4 shows the FB-IoU comparison of each method on the COCO-20 i The bold and underlined results represent the best, and the italic and underlined results represent the second.

[0156] Table 4

[0157]

[0158] It can be seen that the method proposed in the present application achieves substantial improvement, especially on the COCO-20 i dataset, the advantage is more obvious.

[0159] Although the present application has been described in conjunction with the accompanying drawings, the present application is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative, not limiting, and those skilled in the art can make many improvements and changes under the inspiration of the present application without departing from the purpose of the present application, and these all belong to the protection of the present application.

Claims

1. A class-aware based prototype enhanced few-shot segmentation method, characterized in that, Comprising the following steps: Step 1) Obtain PASCAL-5 i and COCO-20 i Dataset used for network training and verification, divide the dataset into 4 subsets according to the image data volume, for each subset, randomly select 1000 pairs of "support-query" images for the verification link, the support image refers to the sample with label, and the query image refers to the unmarked image to be segmented; Step 2) constructing a class-aware prototype enhanced small sample segmentation network to segment the query image: The class-aware prototype enhanced small sample segmentation network comprises an encoder, a base learner, a class-aware module CAM, a prototype enhancement module PEM, an encoder and a refinement module RM; the network is trained and verified by using the training set and the verification set obtained in step S1; The network training process comprises the following steps: 2-1) Support image I for network input s and query image I q , firstly, the encoder Encoder(·) extracts the support feature F s and query feature F q , which is expressed as follows: F s = Encoder(I s ) (1) F q = Encoder(I q ) (2) 2-2) Processing the support features F s with the query features F q : including: Utilizing support features F s and support masks M s Extracting support image class information C s ; Utilizing query features F q With the prototype enhanced module PEM output query image initial prediction results Obtaining query image class information C q ; Wherein, CAM(·) represents the class-aware module CAM mapping; Utilizing support features F s and support masks M s foreground class prototypes are extracted by mask average pooling (MAP) operations is expressed as: where i is the image pixel position index, h and w are the feature map height and width, respectively; [•] is the Iverson bracket, which equals 1 if the internal condition is true, and 0 otherwise; M s is a binary mask, M s (i) = 1 indicates that the ith pixel belongs to class c. 2-3) the prototype enhancement module PEM utilizes the foreground class prototypes with the query feature F q generating a hybrid enhanced prototype the hybrid enhanced prototype for querying the activation of the image target region, denoted as: Wherein, PEM(·) represents the prototype enhancement module PEM mapping; query feature F q , support image class information C s , query image class information C q and hybrid enhanced prototype Channel dimension connection is performed to obtain enhanced query feature That is: 2-4) Based on enhancing query features Generating, by the decoder, a prediction of a target class of the query image tar For: 2-5) using the refinement module RM on the prediction result p of the query image target class tar and base learner processes the query feature F q resulting p base refinement is performed to obtain the final small sample segmentation result p out : p out = RM(p tar ,p base ) (8).

2. The small sample segmentation method of claim 1, wherein, In the step 1), for the four subsets, each subset is taken in turn as the verification set, and at the same time, the other three subsets are taken as the training set for cross-validation.

3. The small sample segmentation method of claim 1, wherein, The design of the category awareness module CAM is that the category awareness module CAM utilizes support features F s , support masks M s , query features F q and query image initial prediction results obtained from the prototype enhancement module PEM to output supported image category information C s and query image category information C q ; Comprising: 1) the support features F s and support masks M s support features with fused class information by Hadamard product Wherein, the symbol represents the Hadamard product operation, and the symbol represents a bilinear interpolation function; the After the pooling, convolution and activation operations, the class weight map W of the support image is obtained s As follows: Wherein, Pool(·) represents the pooling operation, Conv(·) represents the convolution network, and sigma(·) represents the activation function; Utilizing a class weight map W supporting the image s Generating class information C supporting the image s : 2) the query feature F q and the initial coarse prediction result of the query image performing Hadamard product operation to obtain the query feature after fusing the category information performing a pooling, a convolution, and an activation operation to obtain a class weight map W of the query image performing a pooling, a convolution, and an activation operation to obtain a class weight map W of the query image q : Utilizing a class weight map W of the query image q Generating query image class information C q :

4. The small sample segmentation method of claim 1, wherein, The prototype enhancement module PEM comprises an initial prediction submodule IPM, and the prototype enhancement module PEM is designed as: 1) extend the foreground class prototype to the same size as the support feature F s to obtain a support prototype use the support prototype to activate the query feature F q to obtain an initial prediction result of the query image input the query feature F q and the support prototype to the initial prediction sub-module IPM to obtain an initial prediction result is represented as: wherein denotes a decoder, and FP(·) is an operation process. 2) on query feature F q and query image initial prediction result mask average pooling (MAP) operation to extract query prototype is represented as: where i is the image pixel position index, h and w are the height and width of the feature map, respectively; [•] is the Iverson bracket, which equals 1 if the internal condition is true, and 0 otherwise; denotes that the i-th pixel belongs to class c. 3) merge the foreground class prototype with the query prototype i.e.:​​

Images