A single sample image segmentation method and device based on multiple prototypes and iterative enhancement

Abstract

The present invention provides a single-sample image segmentation method and device based on multiple prototypes and iterative enhancement. The method includes: obtaining a support image and a query image respectively, obtaining support features and query features based on the same feature extraction network; generating a number of support prototypes based on the foreground part of the support features; calculating the cosine similarity between any support prototype and the query feature to generate a corresponding similarity graph, placing each support prototype at a corresponding position according to a guide table, and generating a guide feature; at the same time, performing an accumulation operation on all similarity graphs to generate a probability graph guiding the query foreground position; connecting the query feature, the guide feature and the probability graph, performing multi-scale enhancement, and generating a query segmentation graph after decoding. The present invention uses output iteration to enhance the guidance information of the support feature, and uses the generated segmentation graph to correct the previously guided probability graph through addition and multiplication calculations, so that it can be re-guided more accurately and generate a more accurate segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a single sample image segmentation method and device based on multiple prototypes and iterative enhancement. Background Art

[0002] Image semantic segmentation is a classic and thorny problem in computer vision. Exploration and research in this field requires a considerable amount of densely annotated training datasets to support it. The collection of these datasets is costly and time-consuming, and current methods are difficult to generalize to unseen object categories in a data-efficient manner.

[0003] With the rapid development of deep learning technology, semantic segmentation algorithms based on deep convolutional neural networks have made significant progress, making deep convolutional neural networks the most mainstream processing method in the field of image segmentation. Deep convolutional neural networks have the following characteristics: (1) the use of a large amount of image data and powerful computing resources; (2) the convolution processing in the image has the characteristic of weight sharing for feature extraction; (3) the use of activation functions enhances the nonlinear expression ability of neural networks; (4) effective optimization algorithms accelerate the convergence speed of the network; (5) end-to-end training framework simplifies the segmentation process. Different from traditional image segmentation methods, semantic segmentation methods based on deep convolutional neural networks can learn rich high-level semantic features in images, thereby capturing deep abstract information of images, and can be automatically trained end-to-end, thus significantly improving the performance of image segmentation systems and making it the most mainstream method in scene parsing. Most existing semantic segmentation methods are based on fully convolutional networks (FCNs), which use fully convolutional layers instead of fully connected layers for pixel-level prediction. The latest research results in the field of semantic segmentation mainly come from multi-scale feature aggregation or attention mechanisms. These methods usually use dilated convolutional kernels and build an encoder-decoder structure to obtain a large receptive field while maintaining effective resolution.

[0004] Although these methods have achieved great success, they still require long training time and a large amount of pixel-level labeled ground truth to fully supervise the network. In addition, during the inference phase, the trained model cannot recognize new classes that are not present in the training set.

[0005] The few-shot segmentation method is to quickly learn new concepts from limited data, and only a few labeled samples are needed for the system to learn to perform semantic segmentation. The input and output of this method are both pixel-level, and it is able to propagate any set of pixel annotations from within an image or across image categories to unannotated pixels. Specifically, the purpose of few-shot semantic segmentation is to perform pixel-level classification of new classes without too many annotated support images. OSLSM first introduced this setting, proposed the few-shot segmentation method for the first time, and used parameter comparison to solve this problem. Subsequent methods pointed out the use of prototypes to represent the typical information of foreground objects present in support images, and predicted the pixel-level feature comparison between prototypes and query features through cosine similarity. With the in-depth research, some people proposed that convolution can be used to replace the cosine similarity of complex query images. In addition, many studies have also tried to fully exploit support features, and proposed a graph attention unit, which regards each position of the foreground feature in the support image as an individual and establishes a pixel-to-pixel correspondence between the query and support features. However, existing methods only focus on generating a single type of prototype features, ignoring the effects of scale and shape differences between support images and query images, resulting in loss of details. In addition, the multi-scale feature enhancement method of forward propagation adopted by the existing methods leads to incorrect guidance of the probability map as the semantic information continues to weaken during the forward propagation process.

[0006] Single-shot segmentation is a type of small-shot segmentation, that is, only one image is used in the support set, which requires us to fully exploit all the information in the support image. Current single-shot segmentation networks usually use the same network to extract features from the query image and the support image, and then use different feature matching methods and target mask transfer methods from the support image to the query image. Feature matching and mask transfer usually use prototype feature learning techniques. Prototype learning techniques compress the mask target object of the support image into one or several prototype feature vectors. Then, the pixel positions of similar features are found in the query image to segment the target. A major advantage of prototype learning is that prototype features are more robust to noise than pixel features. However, existing methods do not fully exploit the information in the support features and cannot fully cope with support and query images of different foreground scales. At the same time, as the semantic information of the probability map guiding the foreground position of the query image continues to weaken, it may be misguided. Based on previous experience, our goal is to design a module that can better extract the prototype of the support feature for feature matching, and at the same time design a module to supplement the lost semantic information in the probability map to complete the segmentation of the query image. Summary of the invention

[0007] According to the above-mentioned existing methods, the information in the supporting features is not fully exploited, and the technical problem that the semantic information of the probability map guiding the foreground position of the query image and the continuous weakening of the supporting and query images of different foreground scales leads to wrong guidance is not fully addressed. The present invention provides a single-sample image segmentation method and device based on multi-prototype and iterative enhancement, which adopts multi-prototype generation to fully obtain all the information of the supporting features, adopts output iteration to enhance the guiding information of the probability map, and uses the generated segmentation map to correct the previously guided probability map through addition and multiplication calculations, so that it can be re-guided more accurately and generate more accurate segmentation results.

[0008] The technical means adopted by the present invention are as follows:

[0009] A single sample image segmentation method based on multiple prototypes and iterative enhancement includes the following steps:

[0010] A support image and a query image are obtained respectively, wherein the support image and the query image contain objects of the same category, the support image is input into a feature extraction network, and intermediate layer features of the feature extraction are obtained as support features, and the query image is input into the same feature extraction network, and intermediate layer features of the feature extraction are obtained as query features;

[0011] Multiplying the support feature with the true value of the support image to obtain the foreground part of the support feature, and generating a plurality of support prototypes based on the foreground part of the support feature, wherein the support prototype is obtained by aggregating a cluster prototype and a superpixel prototype;

[0012] Calculate the cosine similarity between any supporting prototype and the query feature to generate the corresponding similarity graph. By comparing each similarity graph at the same position, retain the sequence number of the similarity graph with the highest similarity, and finally obtain a guidance table. Place each supporting prototype at the corresponding position according to the guidance table to generate a guidance feature that matches the query feature. At the same time, perform a cumulative operation on all similarity graphs to generate a probability map that guides the query foreground position.

[0013] After the query feature, the guide feature and the probability map are connected, a multi-scale enhancement process is performed to generate a multi-scale feature, the multi-scale feature is sent to a decoder for processing to generate a query segmentation map, and the probability map is corrected based on the query segmentation map.

[0014] Furthermore, a plurality of supporting prototypes are generated based on the foreground part of the supporting feature, including: processing the foreground part of the supporting feature based on the k-means clustering method, dividing the foreground part of the supporting feature into a plurality of data regions, wherein the number of data regions is preset; and performing a masked average pooling operation on each obtained data region to obtain a corresponding clustering prototype.

[0015] Furthermore, generating a plurality of supporting prototypes based on the foreground part of the supporting features also includes: processing the foreground part of the supporting features based on a superpixel guided clustering method, first initializing a random centroid, and determining the superpixel centroid as the superpixel prototype through continuous iteration.

[0016] Furthermore, after the query feature, the guide feature and the probability map are connected, enhancement processing based on a multi-scale approach is performed, including: providing multi-scale input information for feature enhancement through a dilated spatial convolution pooling pyramid or a pyramid pooling model.

[0017] Further, the probability map is modified based on the query segmentation map, including:

[0018] Performing a softMax operation on the query segmentation graph to obtain a channel graph of a channel;

[0019] First, a minimum-maximum normalization operation is performed on the channel map, and then an interval transformation is performed to transform it to the interval of -1 to 1 corresponding to the cosine calculation, and then it is accumulated on the probability map; at the same time, a minimum-maximum normalization operation is performed on the probability map to transform it to the interval of 0 to 1, and then it is multiplied with the initial channel map, and then an interval transformation from -1 to 1 is performed, and then it is accumulated on the original probability map.

[0020] The present invention also discloses a single sample image segmentation device based on multiple prototypes and iterative enhancement, comprising:

[0021] An image acquisition unit, which is used to respectively acquire a support image and a query image, wherein the support image and the query image contain objects of the same category, input the support image into a feature extraction network, and acquire intermediate layer features of the feature extraction as support features, and input the query image into the same feature extraction network, and acquire intermediate layer features of the feature extraction as query features;

[0022] A support prototype acquisition unit, which is used to multiply the support feature with the true value of the support image to obtain the foreground part of the support feature, and generate a plurality of support prototypes based on the foreground part of the support feature, wherein the support prototype is obtained by aggregating the cluster prototype and the superpixel prototype;

[0023] A guiding feature and probability map acquisition unit is used to calculate the cosine similarity between any supporting prototype and the query feature to generate a corresponding similarity map, and by comparing each similarity map at the same position, retaining the sequence number of the similarity map with the highest similarity, and finally obtaining a guiding table, placing each supporting prototype at the corresponding position according to the guiding table, and generating a guiding feature matching the query feature; at the same time, performing a cumulative operation on all similarity maps to generate a probability map guiding the query foreground position;

[0024] The query segmentation map acquisition unit is used to connect the query features, guide features and probability maps, perform enhancement processing based on a multi-scale method to generate multi-scale features, send the multi-scale features to a decoder for processing to generate a query segmentation map, and modify the probability map based on the query segmentation map.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] 1. The present invention adopts two different prototype generation methods, which fully exploit all the information contained in the features of a single supporting image. The obtained prototypes are more representative and can complement and cooperate with each other.

[0027] 2. The present invention adopts an iterative enhancement method to return the output query segmentation graph to the previous probability graph and correct it to make more accurate guidance, thereby generating a more accurate query result graph. Moreover, the iterative enhancement used does not contain any convolution operation and can be directly used in the inference stage without retraining to obtain accurate results.

[0028] Based on the above reasons, the present invention can be widely promoted in the fields of security, medical treatment, etc. where it is difficult to obtain training data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 This is a flow chart of the single sample image segmentation method based on multiple prototypes and iterative enhancement of the present invention.

[0031] Figure 2 This is a diagram of the feature extraction network architecture of the present invention.

[0032] Figure 3 This is a schematic diagram of the prototype generation part of the present invention.

[0033] Figure 4 A block diagram showing the principle of using ASPP for the feature enhancement portion of the present invention.

[0034] Figure 5 A block diagram showing the principle of using PPM for the feature enhancement portion of the present invention.

[0035] Figure 6 It is a principle block diagram of the iterative enhancement part of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] like Figure 1 As shown, the present invention provides a single sample image segmentation method based on multiple prototypes and iterative enhancement, comprising the following steps:

[0038] S1. Obtain a support image and a query image respectively, wherein the support image and the query image contain objects of the same category, input the support image into a feature extraction network, obtain intermediate layer features of the feature extraction as support features, input the query image into the same feature extraction network, obtain intermediate layer features of the feature extraction as query features.

[0039] Specifically, given two original images of the same target class, one as the support set image and the other as the query set image, the two images are input together into a shared ResNet-50 backbone network to obtain the intermediate layer features of the two images.

[0040] S2. Multiply the support feature with the true value of the support image to obtain the foreground part of the support feature, and generate a number of support prototypes based on the foreground part of the support feature, wherein the support prototype is obtained by aggregating cluster prototypes and superpixel prototypes.

[0041] Specifically, k-means clustering is applied to the supporting features. The supporting features are divided into data regions by setting the number of clusters. Then, the clustering prototype of the corresponding part is obtained by masked average pooling for each part of the area. The clustering prototype represents the feature information of this part.

[0042] After each feature is pooled into prototypes, a lot of information may be covered up and cannot be expressed. Therefore, it is necessary to use another method to re-divide the supporting features. Prototypes generated in different ways can have different expressions for the same detail features. This can minimize the loss of detail information of the supporting features.

[0043] The present invention preferably re-applies superpixel guided clustering to the supporting features. The superpixel guided clustering is based on the superpixel centroid as the prototype, and the superpixel guided clustering method is adaptive and can adaptively adjust the number of generated prototypes according to the size of the supporting image foreground, which is just different from the fixed number of prototypes generated regardless of the image size during k-means clustering, which results in the two methods generating different prototypes. First, a random centroid is initialized, and after several iterations, the final centroid is the prototype representing the surrounding features, that is, the superpixel prototype we want to obtain.

[0044] Aggregating the obtained cluster prototype and superpixel prototype together is the prototype of all the information containing the supporting features. Since this application is only for single sample segmentation and there is only one supporting image, it is particularly important to represent the supporting features as fully as possible.

[0045] S3. Calculate the cosine similarity between any supporting prototype and the query feature to generate a corresponding similarity graph. By comparing each similarity graph at the same position, retain the serial number of the similarity graph with the highest similarity, and finally obtain a guidance table. Place each supporting prototype at the corresponding position according to the guidance table to generate a guidance feature that matches the query feature. At the same time, perform a cumulative operation on all similarity graphs to generate a probability map that guides the query foreground position.

[0046] Specifically, all the supporting prototypes are taken out one by one, and each one is subjected to cosine similarity calculation with the query feature. Each one will generate a similarity graph, and each similarity graph is collected. By comparing each similarity graph at the same position, the serial number of the similarity graph with the highest similarity is retained, and finally a guidance table is obtained. This table indicates which supporting prototype is closest to each pixel of the query feature. Through the guidance table, the supporting prototype is placed in the corresponding position to obtain the guidance feature that matches the query feature. At the same time, all the similarity graphs containing similar information are accumulated to obtain a probability graph that guides the query foreground position.

[0047] S4. After connecting the query feature, the guide feature and the probability map, a multi-scale enhancement process is performed to generate a multi-scale feature, the multi-scale feature is sent to a decoder for processing to generate a query segmentation map, and the probability map is corrected based on the query segmentation map.

[0048] Specifically, query features, guide features and probability maps are connected to perform feature enhancement processing. The present invention preferably uses a multi-scale approach to perform feature enhancement. We use the widely used atrous spatial convolutional pooling pyramid (ASPP) or pyramid pooling model (PPM) in image segmentation to provide multi-scale input information. Finally, the multi-scale features are decoded to generate the initial query segmentation map.

[0049] The probability map indicates the location of the query image foreground, and the generated query segmentation map also indicates the location of the query image foreground. It should be noted that the probability map is calculated by the intermediate layer features. The semantic information of the intermediate layer features is not very rich, and it is easy to produce wrong guidance. Therefore, this application returns the generated query segmentation map to the probability map part, and corrects the probability map through mathematical operations, so that it can make more accurate guidance again and generate a more accurate query segmentation map. Moreover, our iterative operation can be applied only in the offline reasoning stage to obtain more accurate segmentation results. The specific operations are:

[0050] Perform a softmax operation on the binary image generated by the network, and take out the channel map of one channel. This map is the foreground probability of the corresponding position of the query image. Two operations are performed on this map. First, a minimum-maximum normalization operation is performed to make it more extreme only the probability of the corresponding part, and then an interval transformation is performed to transform it to the interval of -1 to 1 corresponding to the cosine calculation, and then accumulated to the probability map. For the second method, a minimum-maximum normalization operation is first performed on the probability map to transform it to the interval of 0 to 1, and then it is multiplied with the initial channel map, and then the interval transformation from -1 to 1 is performed, and then accumulated to the original probability map. The probability map is corrected with the generated segmentation map through the two methods of "addition" and "multiplication" to make a more accurate guidance again, so that a more accurate query segmentation map can be generated in the end.

[0051] The present invention is proposed based on the observation results of the support and query images. The scales of the foregrounds of different support and query images are very different. Some support images have very small foregrounds and the query images to be segmented have very large foregrounds, while some support images have very large foregrounds and the query images to be segmented have very small foregrounds. The method of the present invention is different from other methods in that, on the one hand, our method fully represents the characteristics of the support image and can adapt to support images and query images of any scale foreground; on the other hand, our method repairs the probability map indicating the query foreground position by iterative enhancement, so that it can produce more accurate guidance.

[0052] The scheme and effect of the present invention are further described below through specific application examples.

[0053] like Figure 1 The flowchart of the execution process of the method of the present invention is shown as follows. For a given support image and query image, they are first sent to the encoder. The encoder we choose is an extended version of the backbone network ResNet-50. Figure 2This is the structural block diagram of ResNet-50. After passing through the encoder, we take out the features of the third and fourth layers and aggregate them together. It should be noted that this embodiment uses the features of the middle layer. The high-level features contain too much semantic information. It is easy to remember the trained classes during the training phase, which is very unfavorable for the reasoning phase on new classes in the single sample segmentation task. In order to minimize the impact of overfitting, the third and fourth layers of features are used. The number of channels aggregated together is 1536. Then we perform dimensionality reduction through a 1×1 convolution to make the number of channels 256, that is, the number of channels of the obtained support features and query features are both 256.

[0054] The obtained support features and support truth values ​​are processed as follows Figure 3 The prototype generation operation shown in the figure uses two methods to generate prototypes. This embodiment preferably sets the number of clusters of k-means clustering to 5, that is, 5 prototypes can be generated using the k-means clustering method. The maximum number of superpixel centroids of superpixel clustering is set to 5, that is, 1 to 5 superpixel prototypes can be generated using the superpixel clustering method. Collect all the supporting prototypes and perform cosine similarity calculation with the query features to obtain 6 to 10 similarity graphs containing similar information in the range of -1 to 1. By comparing the similarity graphs, the most similar supporting prototype is matched with the query feature. At the same time, we add up all the similarity graphs to obtain a probability graph. The probability graph is the sum of all similar information and can indicate the location of the query prospect.

[0055] Then, feature enhancement operations are performed on the query features, matching features, and probability maps. In this embodiment, the Figure 4 The Atrous Spatial Convolutional Pooling Pyramid Architecture (ASPP) shown in Figure 5 The pyramid pooling architecture (PPM) shown in the figure. ASPP was proposed in deeplab, and an important method is to use dilated convolution. In the image segmentation task, we want to have a larger receptive field for the features extracted from the image, and we also want the resolution of the feature map not to drop too much (too much resolution loss will lose a lot of detailed information about the image boundary), but the key problem is that these two are contradictory. To obtain a larger receptive field, we need to use a larger convolution kernel or a larger step size when pooling. The former is too computationally intensive, and the latter will lose some resolution. The dilated convolution is used to solve this contradiction. It can obtain a larger receptive field without losing too much resolution. The advantage of dilated convolution is that it increases the receptive field without losing information through pooling, so that each convolution output contains a larger range of information.

[0056] When performing segmentation tasks, images have multi-scale problems, some are large and some are small. A common processing method is image pyramid, which is to reset the original image to different scales, input it into the same network, obtain different feature maps, and then fuse them. This method can indeed improve the accuracy, but another problem it brings is that the speed is too slow. In order to solve this problem, DeepLab v2 introduced the ASPP module, which uses parallel hole convolution layers with different expansion rates to capture multi-scale information of the feature map, and fuses the output results to obtain the image segmentation result.

[0057] The pyramid pooling model PPM is a module proposed in the semantic segmentation network PSPNet. Generally speaking, the deeper the network, the larger the receptive field, but there is still a gap between the theoretical receptive field and the receptive field in the actual network (the actual receptive field is smaller than the theoretical receptive field), which makes the network unable to effectively fuse global feature information. Global average pooling can effectively fuse global context information, but its ability to fuse and extract information is limited, and simply using global average pooling to compress information into one channel can easily lose a lot of useful information. Therefore, fusing features of different receptive fields and features of sub-regions can enhance feature representation capabilities. The PPM module can extract feature maps from the previous network and divide them into two branches here. One branch is divided into multiple sub-regions for global average pooling, and then the channel size is adjusted using 1×1 convolution, and then the size before pooling is obtained through bilinear interpolation, and finally the two branches are fused.

[0058] The generated multi-scale features are passed through a simple decoder to generate the initial query segmentation map of the last two channels. By observing that both the segmentation map and the probability map indicate the location of the foreground of the query image, Figure 6 The iterative enhancement process shown returns the segmentation map to the probability map, and through some calculation operations, the probability map is used to make more correct guidance. Since the scale of the segmentation map and the probability map is the same, we do not need to perform any scale transformation operations. Normalization uses the minimum-maximum normalization, and the normalized interval is between 0 and 1. In order to correspond to the similarity map between -1 and 1 obtained by the previous cosine similarity calculation, we need to perform an interval transformation to keep the interval consistent. The probability map can be corrected through simple addition and multiplication operations, and a new segmentation map can be generated by re-guiding. Since no convolution is used here, it can only be applied to the inference stage to produce good results without any speed loss. After several iterations, a segmentation map with high accuracy can be obtained.

[0059] The PASCAL-5i dataset is used during network training. PASCAL-5i consists of PASCAL VOC 2012 with extended annotations from the SDS dataset. The 20 classes are equally divided into 4 copies, each containing 5 classes. We train on PASCAL-5i for 200 epochs with a learning rate of 0.0025 and a batch size of 4. We use SGD as the optimizer. The minimum momentum and weight decay are set to 0.9 and 0.0001, respectively. The “poly” strategy is used to decay the learning rate. The backbone network ResNet-50 is initialized with ImageNet pre-trained weights. During training, samples are processed by mirroring and random rotation from -10 to 10 degrees. Finally, we randomly crop 473×473 pixels from the processed images as training samples. During inference, each input sample is resized to the training block size, but with respect to its original aspect ratio, padded with zeros, and then the predicted size is resized back to the original label size.

[0060] The present invention also discloses a single sample image segmentation device based on multiple prototypes and iterative enhancement, comprising:

[0061] An image acquisition unit, which is used to respectively acquire a support image and a query image, wherein the support image and the query image contain objects of the same category, input the support image into a feature extraction network, and acquire intermediate layer features of the feature extraction as support features, and input the query image into the same feature extraction network, and acquire intermediate layer features of the feature extraction as query features;

[0062] A support prototype acquisition unit, which is used to multiply the support feature with the true value of the support image to obtain the foreground part of the support feature, and generate a plurality of support prototypes based on the foreground part of the support feature, wherein the support prototype is obtained by aggregating the cluster prototype and the superpixel prototype;

[0063] A guiding feature and probability map acquisition unit is used to calculate the cosine similarity between any supporting prototype and the query feature to generate a corresponding similarity map, and by comparing each similarity map at the same position, retaining the sequence number of the similarity map with the highest similarity, and finally obtaining a guiding table, placing each supporting prototype at the corresponding position according to the guiding table, and generating a guiding feature matching the query feature; at the same time, performing a cumulative operation on all similarity maps to generate a probability map guiding the query foreground position;

[0064] The query segmentation map acquisition unit is used to connect the query features, guide features and probability maps, perform enhancement processing based on a multi-scale method to generate multi-scale features, send the multi-scale features to a decoder for processing to generate a query segmentation map, and modify the probability map based on the query segmentation map.

[0065] As for the embodiments of the present invention, since they correspond to the embodiments above, the description is relatively simple. For the relevant similarities, please refer to the description of the above embodiments, which will not be described in detail here.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single sample image segmentation method based on multiple prototypes and iterative enhancement, characterized in that: The following steps are involved: A support image and a query image are obtained respectively, wherein the support image and the query image contain objects of the same category, the support image is input into a feature extraction network, and intermediate layer features of the feature extraction are obtained as support features, and the query image is input into the same feature extraction network, and intermediate layer features of the feature extraction are obtained as query features; Multiplying the support feature with the true value of the support image to obtain the foreground part of the support feature, and generating a plurality of support prototypes based on the foreground part of the support feature, wherein the support prototype is obtained by aggregating a cluster prototype and a superpixel prototype; Calculate the cosine similarity between any supporting prototype and the query feature to generate the corresponding similarity graph. By comparing each similarity graph at the same position, retain the sequence number of the similarity graph with the highest similarity, and finally obtain a guidance table. Place each supporting prototype at the corresponding position according to the guidance table to generate a guidance feature that matches the query feature. At the same time, perform a cumulative operation on all similarity graphs to generate a probability map that guides the query foreground position. After the query feature, the guide feature and the probability map are connected, a multi-scale enhancement process is performed to generate a multi-scale feature, the multi-scale feature is sent to a decoder for processing to generate a query segmentation map, and the probability map is corrected based on the query segmentation map.

2. The single sample image segmentation method based on multiple prototypes and iterative enhancement according to claim 1, characterized in that: A plurality of supporting prototypes are generated based on the foreground part of the supporting feature, including: processing the foreground part of the supporting feature based on the k-means clustering method, dividing the foreground part of the supporting feature into a plurality of data regions, wherein the number of the data regions is preset; and performing a mask average pooling operation on each obtained data region to obtain a corresponding clustering prototype.

3. The single sample image segmentation method based on multiple prototypes and iterative enhancement according to claim 2, characterized in that: Generating a plurality of supporting prototypes based on the foreground part of the supporting feature also includes: processing the foreground part of the supporting feature based on a superpixel guided clustering method, first initializing a random centroid, and determining the superpixel centroid as the superpixel prototype through continuous iteration.

4. The single sample image segmentation method based on multiple prototypes and iterative enhancement according to claim 1, characterized in that: After the query feature, the guide feature and the probability map are connected, enhancement processing based on a multi-scale approach is performed, including: providing multi-scale input information for feature enhancement through a dilated spatial convolution pooling pyramid or a pyramid pooling model.

5. The single sample image segmentation method based on multiple prototypes and iterative enhancement according to claim 1, characterized in that: The probability map is modified based on the query segmentation map, including: Performing a softMax operation on the query segmentation graph to obtain a channel graph of a channel; First, a minimum-maximum normalization operation is performed on the channel map, and then an interval transformation is performed to transform it to the interval of -1 to 1 corresponding to the cosine calculation, and then it is accumulated on the probability map; at the same time, a minimum-maximum normalization operation is performed on the probability map to transform it to the interval of 0 to 1, and then it is multiplied with the initial channel map, and then an interval transformation from -1 to 1 is performed, and then it is accumulated on the original probability map.

6. A single sample image segmentation device based on multiple prototypes and iterative enhancement, characterized in that: include: An image acquisition unit, which is used to respectively acquire a support image and a query image, wherein the support image and the query image contain objects of the same category, input the support image into a feature extraction network, and acquire intermediate layer features of the feature extraction as support features, and input the query image into the same feature extraction network, and acquire intermediate layer features of the feature extraction as query features; A support prototype acquisition unit, which is used to multiply the support feature with the true value of the support image to obtain the foreground part of the support feature, and generate a plurality of support prototypes based on the foreground part of the support feature, wherein the support prototype is obtained by aggregating the cluster prototype and the superpixel prototype; A guiding feature and probability map acquisition unit is used to calculate the cosine similarity between any supporting prototype and the query feature to generate a corresponding similarity map, and by comparing each similarity map at the same position, retaining the sequence number of the similarity map with the highest similarity, and finally obtaining a guiding table, placing each supporting prototype at the corresponding position according to the guiding table, and generating a guiding feature matching the query feature; at the same time, performing a cumulative operation on all similarity maps to generate a probability map guiding the query foreground position; The query segmentation map acquisition unit is used to connect the query features, guide features and probability maps, perform enhancement processing based on a multi-scale method to generate multi-scale features, send the multi-scale features to a decoder for processing to generate a query segmentation map, and modify the probability map based on the query segmentation map.

Images