A small sample segmentation method and device based on fine-tuning and self-evaluation

Abstract

The application belongs to the technical field of deep learning and computer vision, and discloses a small sample segmentation method and device based on fine tuning and self-evaluation. The method firstly initializes system parameters, then predicts pseudo labels for query images to obtain query prototypes, updates the query pseudo labels after combining with support prototypes, fine tunes the system parameters by using the pseudo labels, then repeats the process to realize multiple fine tuning of the system, finally determines the best fine tuning round by using the designed self-evaluation method, and obtains the final segmentation result. The application alleviates the problem that the existing system uses categories or support images in biased training, and improves the segmentation accuracy. The designed self-evaluation method can evaluate the optimal fine tuning round, and prevent the error accumulation in the fine tuning process from causing the problem of low accuracy. Compared with the existing method, the fine tuning process only needs about 1% of the time cost and computing power cost, and can more quickly and efficiently adapt to new categories and obtain high-quality segmentation results.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision technology, and in particular to a few-sample segmentation method and apparatus based on fine-tuning and self-evaluation. Background Technology

[0002] With the development of deep learning, convolutional neural networks (CNNs) have made significant progress in semantic segmentation tasks. However, training such CNNs requires a large amount of densely labeled data, which is always difficult to obtain. Although semi-supervised learning can alleviate this problem, it still requires a large amount of weakly labeled or unlabeled data. In contrast, relying on strong background knowledge, humans can easily identify new categories by observing limited labeled data. Inspired by this, many researchers have focused on how to enable CNNs to identify new categories with only a small amount of labeled data; this field is called few-shot learning.

[0003] Few-shot segmentation, as an application of few-shot learning in segmentation tasks, focuses on segmenting target regions of new categories with only a limited amount of labeled data. Most existing few-shot segmentation research is based on a meta-learning paradigm. Existing methods obtain category prototypes from labeled support images, which are then used to activate unlabeled query features. A decoder then captures the segmentation result of the target query image. However, since the system parameters are obtained during training, the system tends to favor the categories used in training rather than the target new category, leading to lower segmentation accuracy for the new category. Fine-tuning the system using support images of the new category after training can partially alleviate this problem, but this will still bias the system towards the support images rather than the target query image, especially when there is a significant difference between the support and query images, making it difficult for the system to achieve high-quality segmentation of the target query image. This is the inherent problem with existing methods. Summary of the Invention

[0004] To alleviate the aforementioned problems, this invention proposes a few-shot segmentation method and apparatus based on fine-tuning and self-evaluation. After the few-shot segmentation system has been trained, this method further optimizes the system parameters on target query images of new categories, making the system more adaptable to the target query images and thus obtaining higher-quality segmentation results. This invention will ultimately output the probability... Figure Two The query image is valued as a pseudo-label, and the query loss is calculated using these pseudo-labels during fine-tuning to optimize system parameters, thus biasing the system parameters towards the target query image. However, due to the inherent errors in the pseudo-labels, using only this fine-tuning strategy may lead to error accumulation and worse segmentation results. Therefore, we creatively propose a self-evaluation mechanism to find the fine-tuning round that optimizes the system parameters, and use the query pseudo-label from that round as the final segmentation result, thereby mitigating error accumulation.

[0005] The technical scheme of the present application is as follows: a small sample segmentation method based on fine tuning and self-evaluation, realized based on a small sample segmentation system; the small sample segmentation system comprises an encoding module, a prototype generation module and a feature activation module, which are connected in sequence;

[0006] S1, divide existing data into a support set and a query set, train the small sample segmentation system in a traditional meta-training manner to obtain trained initial small sample segmentation system parameters;

[0007] S2, input the support image with support label and the target query image without label into the encoding module respectively to obtain support features and query features; input the support features and the support label into the prototype generation module to obtain a support prototype; activate the query features in the feature activation module by the support prototype to obtain a query probability map output by the small sample segmentation system, and obtain an initial query pseudo label after binarization;

[0008] S3, according to the initial query pseudo label and the query features output by the encoding module, obtain a query prototype through the prototype generation module, and obtain a total prototype by weighted averaging of the support prototype and the query prototype; update the query probability map by reactivating the query features in the feature activation module through the total prototype, and obtain an updated query pseudo label after binarization;

[0009] S4, calculate the loss according to the updated query pseudo label and the updated query probability map, and adjust the parameters of the encoding module and the feature activation module;

[0010] S5, repeat the processes of S2, S3 and S4 multiple times until a set number of adjustments is reached;

[0011] S7, after each round of adjustment, the small sample segmentation system performs self-evaluation, the query prototype activates the support features through the feature activation module to obtain a support probability map output by the small sample segmentation system, and obtains a support pseudo label after binarization, and compares it with the support label; according to the segmentation result of the small sample segmentation system on the support image, determine which round the small sample segmentation system reaches the optimal, and take the query pseudo label of the round as the final query image segmentation result.

[0012] The encoding module is used to extract image features, and the encoding module is a convolutional neural network pre-trained on an existing data set; during the training process of the small sample segmentation system in step S1, the parameters of the encoding module are fixed, and during the adjustment stage of the small sample segmentation system, the parameters of the encoding module are further optimized or adjusted.

[0013] The prototype generation module does not contain trainable parameters; the prototype generation module obtains a support prototype vector or a query prototype vector by performing a mask average pooling operation on the support feature or the query feature, and then expands the support prototype vector or the query prototype vector to obtain a support prototype or a query prototype;

[0014] p=Exp(Map(f,m)),

[0015] Wherein, f is the support feature or the query feature output by the encoding module, m is the support label or the query pseudo label, Map() is a mask average pooling operation, Exp() is an expansion operation, and p is the support prototype or the query prototype.

[0016] The feature activation module includes a trainable convolutional neural network, and the parameters of the convolutional neural network are optimized in the training and adjustment stages of the small sample segmentation system; the input of the feature activation module is the support prototype and the query feature, or the query prototype and the support feature, and the output is a query probability map or a support probability map.

[0017] The loss is a cross-entropy loss of the query pseudo label and the query probability map, and the parameters of the encoding module and the feature activation module are adjusted by an adaptive moment estimation optimization algorithm; the cross-entropy loss is as follows:

[0018]

[0019] Wherein, L is a loss value, BCE() is a binary cross-entropy loss, is a query probability map, is a query pseudo label.

[0020] The small sample segmentation system calculates the intersection over union of the support pseudo label and the support label as follows:

[0021]

[0022] Wherein, IoU is the intersection over union, TP represents the number of pixels that are both foreground in the support pseudo label and the support label, FN represents the number of pixels that are background in the support pseudo label and foreground in the support label, and FP represents the number of pixels that are foreground in the support pseudo label and background in the support label.

[0023] A small sample segmentation device based on fine-tuning and self-evaluation, comprising:

[0024] An image acquisition device for acquiring support images and query images;

[0025] A memory for storing support images, support labels, query images, query pseudo labels, segmentation results, and small sample segmentation system parameters;

[0026] A processor is configured to read support images, support labels, query images and query pseudo labels, and execute a corresponding computer program; when the computer program is executed, the processor is configured to:

[0027] Load initial small sample segmentation system parameters;

[0028] The labeled support images and unlabeled target query images are input into the encoding module to obtain support features and query features, the support prototype is obtained from the support features and the support labels through the prototype generation module, the support prototype is used to activate the query features in the feature activation module to obtain the query probability map output by the system, and the initial query pseudo label is obtained after binarization and is transmitted to the memory;

[0029] According to the initial query pseudo label and the query features, the query prototype is obtained through the prototype generation module, the support prototype and the query prototype are weighted and averaged to obtain the total prototype, and the total prototype is used to activate the query features in the feature activation module to update the query probability map, and the updated query pseudo label is obtained after binarization and is transmitted to the memory;

[0030] According to the updated query pseudo label and the query probability map, the loss is calculated, and the parameters of the encoding module and the feature activation module are adjusted;

[0031] The parameters of the encoding module and the feature activation module are repeatedly adjusted for multiple times until a set number of adjustments is reached;

[0032] The small sample segmentation system performs self-evaluation after each round of adjustment, the support features are activated through the feature activation module by using the query prototype to obtain the support probability map output by the small sample segmentation system, the support pseudo label is obtained after binarization, and is compared with the support label to obtain the segmentation result of the small sample segmentation system on the support image, which is stored in the memory, the optimal adjustment round of the small sample segmentation system is determined, the query pseudo label of the round is used as the final query image segmentation result, and is transmitted to the memory.

[0033] The beneficial effects of the present application are as follows:

[0034] 1. The present application uses query data itself to fine-tune the system, so that the system is more suitable for the target query image, and the problem of existing systems using classes or support images in biased training is alleviated, which can improve the segmentation accuracy of the system on the target query image.

[0035] 2. The self-evaluation method designed in the present application can evaluate which round of the fine-tuning process is optimal, which can prevent the problem of error accumulation in the fine-tuning process leading to reduced result accuracy.

[0036] 3. Compared with existing methods, the present invention is more efficient and can adapt to new categories more quickly. The fine-tuning process only requires about 1% of the time and computing power cost of existing methods.

[0037] 4. This invention uses initial query pseudo-labels and query features to obtain query prototypes and makes them participate in the activation of query features. This self-activation scheme can alleviate the problem that the support prototype cannot accurately activate query features when the support image and the query image are very different, and further improve the system's segmentation accuracy of the target query image. Attached Figure Description

[0038] Figure One This is a flowchart of the overall process of a small sample segmentation method based on fine-tuning and self-evaluation according to the present invention.

[0039] Figure Two This is a schematic diagram illustrating the process of querying pseudo-tags and supporting pseudo-tag generation in this invention;

[0040] Figure Three This is a schematic diagram of a small sample segmentation device based on fine-tuning and self-evaluation. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] like Figure One and Figure Two As shown, this invention discloses a small-sample segmentation system based on fine-tuning and self-evaluation, with the following specific steps:

[0043] S1, Constructing a small-sample segmentation system, mainly consists of three modules, as follows:

[0044] 1) Encoding module: Used to extract image features. This module is a convolutional neural network pre-trained on the ImageNet1K dataset and is not further optimized during system training. That is, the parameters remain unchanged during system training, but can be further optimized and fine-tuned during the system fine-tuning stage.

[0045] 2) Prototype Generation Module: This module does not contain trainable parameters. It obtains support or query prototype vectors by performing masked average pooling on the support or query features obtained from the encoding module. These vectors are then expanded to obtain a support or query prototype tensor, which is the final prototype. This process is expressed by the following formula:

[0046] p = Exp(Map(f,m)),

[0047] where f is the support or query feature output by the encoding module, m is the support label or query pseudo label, Map() is the mask average pooling operation, Exp() is the expansion operation, and p is the support or query prototype tensor.

[0048] 3) Feature activation module: This module contains a trainable convolutional neural network, and the network parameters can be optimized in the training and fine-tuning stages. The input of this module is the support prototype tensor and the query feature, or the query prototype tensor and the support feature, and the output is the corresponding query probability map or support probability map.

[0049] S2, divide the existing data into support set and query set, and train the system in a traditional meta-training manner to obtain the trained initial system parameters.

[0050] S3, input the labeled support image and unlabeled target query image into the encoding module respectively to obtain the support feature and query feature; then obtain the support prototype through the prototype generation module according to the support feature and support label; activate the query feature in the feature activation module using the support prototype to obtain the query probability map output by the system, and obtain the initial query pseudo label after binarization.

[0051] S4, obtain the query prototype through the prototype generation module according to the initial query pseudo label and query feature; then obtain the total prototype by weighted average of the support prototype and the query prototype, and activate the query feature in the feature activation module using the total prototype to update the query probability map, and obtain the updated query pseudo label after binarization.

[0052] S5, according to the updated query pseudo label and query probability map, calculate the cross-entropy loss of the query pseudo label and query probability map, and use the adaptive moment estimation optimization algorithm to fine-tune the parameters of the encoding module and the feature activation module. The loss is represented by the following formula:

[0053]

[0054] where L is the loss value, BCE() is the binary cross-entropy loss, is the query probability map, is the query pseudo label.

[0055] S6, for the fine-tuned parameters of the encoding module and the feature activation module, repeat the processes of S3, S4 and S5 multiple times to realize multiple fine-tuning of the system until the set fine-tuning times are reached.

[0056] S7. After each round of fine-tuning, the system performs a self-evaluation. Using the query prototype, it activates support features through the feature activation module, obtaining the system's output support probability map. After binarization, it obtains support pseudo-labels, which are compared with the support labels to evaluate the system's segmentation results on the support image. This determines in which fine-tuning round the system reaches its optimal state, and the query pseudo-label from that round is taken as the final query image segmentation result. The specific evaluation method is to calculate the intersection-union ratio (IUU) of the support pseudo-labels and the support labels. The calculation method is expressed by the formula:

[0057]

[0058] Where IoU is the intersection-union ratio, TP represents the number of pixels where both the pseudo-label and the supporting label are in the foreground, FN represents the number of pixels where the pseudo-label is in the background and the supporting label is in the foreground, and FP represents the number of pixels where the pseudo-label is in the foreground and the supporting label is in the background.

[0059] like Figure Three As shown, a small sample segmentation device based on fine-tuning and self-evaluation includes:

[0060] Image acquisition equipment used to acquire supporting images and query images;

[0061] The memory is used to store supporting images, supporting labels, query images, query pseudo-labels, segmentation results, and few-sample segmentation system parameters.

[0062] The processor is configured to read supporting images, supporting tags, query images, and query pseudo-tags, and execute corresponding computer programs; when the computer programs are executed, the processor is configured to:

[0063] Load the initial system parameters into the system;

[0064] Labeled support images and unlabeled target query images are input into the encoding module to obtain support features and query features. Then, based on the support features and support labels, a support prototype is obtained through the prototype generation module. The query features are activated in the feature activation module using the support prototype to obtain the query probability map output by the system. After binarization, the initial query pseudo-label is obtained and transmitted to the memory.

[0065] Based on the initial query pseudo-label and query features, the query prototype is obtained through the prototype generation module; then the supporting prototype and the query prototype are weighted and averaged to obtain the total prototype. The total prototype is used to reactivate the query features in the feature activation module, thereby updating the query probability map. After binarization, the updated query pseudo-label is obtained and transmitted to the memory.

[0066] Based on the updated query pseudo-labels and query probability map, the loss is calculated, thereby fine-tuning the parameters of the encoding module and the feature activation module;

[0067] The above process is repeated multiple times for the parameters of the fine-tuned encoding module and feature activation module to achieve multiple fine-tuning of the system until the set number of fine-tuning times is reached.

[0068] After each round of fine-tuning, the system performs a self-evaluation. By using the query prototype to activate the supporting features through the feature activation module, the system outputs a support probability map. After binarization, a support pseudo-label is obtained and compared with the support label to evaluate the system's segmentation result on the support image. This determines in which round of fine-tuning the system reaches its optimal state. The query pseudo-label of that round is then used as the final query image segmentation result and transmitted to the memory.

[0069] Analysis of Experimental Results

[0070] This embodiment is based on PASCAL-5. i and COCO-20 i Experiments were conducted on two datasets: PASCAL-5. i It contains 20 categories, divided into four subsets, each containing five categories. (COCO-20) i The total of 80 categories were divided into four subsets, each containing 20 categories. This embodiment also conducted experiments with one and five support images. The encoding module used a ResNet101 network, and cross-validation was performed on the four subsets of the two datasets. The mean intersection-over-union ratio (mIoU) was used as the evaluation metric; a higher mIoU value indicates better segmentation performance. The results were also compared with five other recent schemes, as shown in Tables 1 and 2.

[0071] As can be seen from the table, for the case of only one supporting image, PASCAL-5 i The average mIoU metric of the dataset reached 70.6%, COCO-20 i The average mIoU metric for the dataset reaches 52.2%. For the case with five supporting images, PASCAL-5... i The average mIoU metric of the dataset reached 74.9%, COCO-20 iThe average mIoU metric of the dataset reaches 57.8%. Compared to the millions of support images required by traditional deep learning methods, the few-shot segmentation method can achieve relatively accurate image segmentation using only five or even one support image. Furthermore, the mIoU metric obtained in this invention is significantly higher than existing methods, meaning that compared to other few-shot segmentation schemes, this invention achieves more accurate segmentation. These results demonstrate that the few-shot segmentation method based on fine-tuning and self-evaluation proposed in this invention is effective in improving the segmentation performance of the system.

[0072] The five options used for comparison are from the following sources:

[0073] Option 1: Wuhan University of Technology. A small-sample semantic segmentation method and device: CN202210436471.1 [P]. 2022-05-24.

[0074] Option 2: Beijing University of Technology. A cross-domain few-sample image semantic segmentation method based on memory mechanism: CN202210707799.2[P]. 2022-11-18.

[0075] Scheme 3: Z.Tian, ​​H.Zhao, M.Shu, Z.Yang, R.Li, and J.Jia, "Prior Guided FeatureEnrichment Network for Few-Shot Segmentation", IEEE Trans.PatternAnal.Mach.Intell., vol.44, no.2, pp.1050--1065, Feb.2022.

[0076] Scheme 4: J.Min, D.Kang, and M.Cho, "Hypercorrelation squeeze for few-shotsegmentation," Proc.Int.Conf.Comput.Vis.(ICCV), 2021, pp.6941--6952.

[0077] Solution 5: F.Qi, P.Wenjie, T.Yu-Wing, and T.Chi-Keung, "Self-Support Few-ShotSemantic Segmentation," Proc.Eur.Conf.Comput.Vis.(ECCV), 2022.

[0078] Table 1 PASCAL-5 i Dataset Experiment Results (%)

[0079]

[0080] Table 2 COCO-20 i Dataset Experiment Results (%)

[0081]

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A small-sample segmentation method based on fine-tuning and self-evaluation, characterized in that, This is implemented based on a few-sample segmentation system; the few-sample segmentation system includes an encoding module, a prototype generation module, and a feature activation module, which are connected in sequence. S1. Divide the existing data into a support set and a query set, and train the small sample segmentation system using the traditional meta-training method to obtain the initial parameters of the trained small sample segmentation system. S2. Input the supporting image with support labels and the target query image without labels into the encoding module respectively to obtain the supporting features and query features respectively; Supporting features and supporting labels are input into the prototype generation module to obtain a supporting prototype; the supporting prototype activates the query features in the feature activation module to obtain the query probability map output by the small sample segmentation system, which is then binarized to obtain the initial query pseudo-label. S3. Based on the initial query pseudo-label and the query features output by the encoding module, the query prototype is obtained through the prototype generation module. The supporting prototype and the query prototype are weighted and averaged to obtain the total prototype. The query features are reactivated in the feature activation module through the total prototype to update the query probability map. After binarization, the updated query pseudo-label is obtained. S4. Based on the updated query pseudo-labels and the updated query probability map, calculate the loss and fine-tune the parameters of the encoding module and the feature activation module. S5. Repeat steps S2, S3 and S4 multiple times until the set number of fine adjustments is reached; S6. After each round of fine-tuning, the few-shot segmentation system performs self-evaluation. The query prototype activates the supporting features through the feature activation module to obtain the support probability map output by the few-shot segmentation system. After binarization, the support pseudo-label is obtained and compared with the support label. Based on the segmentation results of the few-shot segmentation system on the support image, it is determined in which round the few-shot segmentation system reaches its optimal state. The query pseudo-label of that round is used as the final query image segmentation result.

2. The small sample segmentation method based on fine-tuning and self-evaluation according to claim 1, characterized in that, The encoding module is used to extract image features. The encoding module is a convolutional neural network pre-trained on an existing dataset. In the few-shot segmentation system training process in step S1, the parameters of the encoding module are fixed. In the few-shot segmentation system fine-tuning stage, the parameters of the encoding module are further optimized or fine-tuned.

3. The small sample segmentation method based on fine-tuning and self-evaluation according to claim 1 or 2, characterized in that, The prototype generation module does not contain trainable parameters; the prototype generation module obtains the supporting prototype vector or query prototype vector by performing masked average pooling operation on the supporting features or query features, and then expands the supporting prototype vector or query prototype vector to obtain the supporting prototype or query prototype. ， Where f represents the supported features or query features output by the encoding module, m represents the supported labels or query pseudo-labels, and Map() represents the masked average pooling operation. () represents the expansion operation, and p represents the supporting prototype or query prototype.

4. The small sample segmentation method based on fine-tuning and self-evaluation according to claim 3, characterized in that, The feature activation module includes a trainable convolutional neural network, whose parameters are optimized during the training and fine-tuning phases of the few-sample segmentation system. The input to the feature activation module is either a supporting prototype and a query feature, or a query prototype and supporting features, and the output is either a query probability map or a supporting probability map.

5. The small sample segmentation method based on fine-tuning and self-evaluation according to claim 1, 2, or 4, characterized in that, The loss is the cross-entropy loss between the query pseudo-label and the query probability graph, and the parameters of the encoding module and the feature activation module are fine-tuned using an adaptive moment estimation optimization algorithm; the cross-entropy loss is as follows: ， Where L is the loss value, and BCE() is the binary cross-entropy loss. To query the probability graph, To query for pseudo tags.

6. The small sample segmentation method based on fine-tuning and self-evaluation according to claim 5, characterized in that, The specific segmentation results of the few-sample segmentation system for the supporting images are as follows: calculate the intersection-union ratio of the supporting pseudo-labels and the supporting labels; ， in For the intersection-union comparison, TP represents the number of pixels where both the pseudo-label and the supporting label are in the foreground, FN represents the number of pixels where the pseudo-label is in the background and the supporting label is in the foreground, and FP represents the number of pixels where the pseudo-label is in the foreground and the supporting label is in the background.

7. A small sample segmentation device based on fine-tuning and self-evaluation, characterized in that, include: Image acquisition equipment used to acquire supporting images and query images; The memory is used to store supporting images, supporting labels, query images, query pseudo-labels, segmentation results, and few-sample segmentation system parameters. The processor is configured to read supporting images, supporting tags, query images, and query pseudo-tags, and execute corresponding computer programs; when the computer programs are executed, the processor is configured to: Load the initial small sample segmentation system parameters; Labeled support images and unlabeled target query images are input into the encoding module to obtain support features and query features. Based on the support features and support labels, a support prototype is obtained through the prototype generation module. The query features are activated in the feature activation module using the support prototype to obtain the query probability map output by the system. After binarization, the initial query pseudo-label is obtained and transmitted to the memory. Based on the initial query pseudo-label and query features, the query prototype is obtained through the prototype generation module. Then, the supporting prototype and the query prototype are weighted and averaged to obtain the total prototype. The query features are reactivated in the feature activation module using the total prototype, thereby updating the query probability map. After binarization, the updated query pseudo-label is obtained and transmitted to the memory. Based on the updated query pseudo-labels and query probability map, calculate the loss and fine-tune the parameters of the encoding module and the feature activation module; Repeatedly fine-tune the parameters of the encoding module and the feature activation module until the set number of fine-tuning steps is reached; The few-shot segmentation system performs self-evaluation after each round of fine-tuning. It activates support features through the feature activation module using the query prototype to obtain the support probability map output by the few-shot segmentation system. After binarization, it obtains the support pseudo-labels and compares them with the support labels to obtain the segmentation results of the few-shot segmentation system on the support image. These results are stored in memory. The optimal fine-tuning round is determined, and the query pseudo-label of this round is used as the final query image segmentation result and transmitted to memory.

Images