A Few-Shot Semantic Segmentation Method Based on Learner Integration Strategy

Abstract

The present invention relates to a few-shot semantic segmentation method based on a learner integration strategy. Aiming at the problem of base class bias existing in previous related work, that is, the base class target regions are easily misactivated, the present invention proposes to introduce an additional auxiliary branch (base learner) on the basis of a traditional model (meta-learner) to explicitly identify the base class targets, and then adaptively integrate the rough prediction results of the two learners to generate accurate segmentation masks. Considering the sensitivity of the meta-learner, the present invention further introduces an adjustment factor to estimate the differences between input images to promote model integration inference. Generally speaking, the present invention provides a novel and general solution for improving the problem of misactivation of base class targets by existing few-shot segmentation models, which is not limited to a certain base learner and meta-learner, significantly improves the performance of existing segmentation networks and can be extended to more practical and challenging generalized settings, showing good application prospects and potential.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a few-shot semantic segmentation method based on a learner integration strategy, which can effectively alleviate the bias problem of traditional models towards base-class objects. Background Art

[0002] Thanks to the well-established large-scale datasets, many computer vision technologies have developed rapidly in the past few years. As a basic task in computer vision, semantic segmentation has also made remarkable progress and has broad application prospects in the fields of military, medicine, transportation, crops, etc.

[0003] However, collecting sufficient labeled data for neural network model training requires a large amount of time, manpower, and material resources. Especially for dense prediction tasks such as semantic segmentation and instance segmentation, the labeling cost is even higher. Contrary to the machine learning paradigm, people can easily identify new concepts or patterns from a few examples, which has greatly stimulated the research interest of relevant personnel. In this context, few-shot learning-based methods have emerged, successfully establishing network models that can be generalized to unknown domains with scarce labeled samples.

[0004] The present invention mainly focuses on the application of few-shot learning technology in the field of semantic segmentation, that is, few-shot semantic segmentation, aiming to segment objects of specific semantic categories from the original image using only a small amount of labeled training data. Driven by the progress made in few-shot classification tasks, most existing few-shot segmentation methods attempt to achieve generalization through a meta-learning framework. Specifically, a series of learning tasks are randomly sampled from the base-class dataset to simulate the few-shot scenarios of new classes, with the aim of matching the training and testing conditions. However, the effects achieved by such a training paradigm are not ideal. Meta-training on the base-class dataset with a large number of labeled samples inevitably introduces bias towards the seen classes and cannot achieve ideal class-agnosticism, which to a certain extent hinders the recognition of new concepts. Furthermore, when segmenting difficult query samples with similar class compositions to the base-class samples, the generalization performance of the model may be on the verge of collapse. In addition to designing more powerful feature extraction networks, adjusting the usage method of the base-class dataset with a large number of labeled samples is also another way to alleviate the above bias problem, but it has been ignored by previous research work. Therefore, how to more efficiently use the labeled base-class samples to help the network identify those areas that are prone to misclassification has become the key to breaking through the performance bottleneck of few-shot semantic segmentation models. Summary of the Invention

[0005] Technical Problems to be Solved

[0006] To avoid the deficiencies of the prior art, the present invention proposes a few-shot semantic segmentation method based on a learner integration strategy. In previous related work, there was a problem of base class bias, that is, the problem of easily misactivating the target regions of the base class. On the basis of a traditional model (meta-learner), an additional auxiliary branch (base-learner) is introduced to explicitly identify the base class targets, and then the rough prediction results of the two learners are adaptively integrated to generate accurate segmentation masks. Considering the sensitivity of the meta-learner, the present invention further introduces a modulation factor to estimate the differences between input images to promote model integration reasoning. Generally speaking, the present invention provides a novel and general solution to improve the problem of misactivating base class targets in existing few-shot segmentation models, which is not limited to a certain base-learner and meta-learner. While significantly improving the performance of existing segmentation networks, it can be extended to more practical and challenging generalized settings, showing good application prospects and potential.

[0007] Technical solution

[0008] A few-shot semantic segmentation method based on a learner integration strategy, characterized by the following steps:

[0009] Step 1, construct a network: The first network is a feature extraction network, one output end is connected to the base-learner network and another meta-learner network, and another output end of the feature extraction network is connected to the meta-learner network; the output connections of the base-learner network and the meta-learner network are connected to the integration module;

[0010] Step 2, network training:

[0011] Step a: Given a query image It first passes through the feature extraction network E(·) shared with the meta-learner and an additional convolutional block to obtain an intermediate feature map This process is:

[0012]

[0013] where, F conv (·) represents consecutive convolutional operations, which represent the last convolutional block; c, h, w respectively represent the dimensions of channels, height, and width, and h×w represents the smallest spatial resolution among all the extracted features.

[0014] Step b: Subsequently, the decoder network D b (·) gradually increases the spatial scale of the intermediate feature map to enrich the segmentation clues related to details and finally generate a prediction result. The above process can be defined as:

[0015]

[0016] Among them, the soft max(·) operation is performed along the channel dimension to generate the prediction probability map p b , N b represents the number of base classes in the current data partition; the decoder network D b (·) includes a pyramid pooling module for increasing the receptive field, a convolutional layer for integrating features at various scales, and an output layer composed of 1×1 convolutions;

[0017] Step c: Given a support set S = {x s , m s} and a query image the goal of the meta-learner is to segment the objects in the query image that belong to the same category as the annotation mask m s under the guidance of the support set S; after using the backbone network to extract the features of the support and query images, a 1×1 convolutional layer is immediately added to reduce the channel dimension and generate an intermediate feature map. The process is as follows:

[0018]

[0019]

[0020] Among them, E(·) represents the feature extraction network shared with the base class learner, and F 1×1 (·) is a 1×1 convolution that encodes the input feature dimension to a lower dimension; and respectively represent the intermediate feature maps of the support image and the query image extracted after passing through the backbone network and the 1×1 convolution;

[0021] Step d: Using the mask average pooling method, category-related important segmentation clues, called prototypes v are calculated based on the support feature map s and its corresponding mask m s . The process is as follows:

[0022]

[0023] Among them, F pool (·) represents the average pooling operation, ⊙ represents the Hadamard product, and I(·) is a function that can transform the support image mask into the same shape and size as the support image feature through interpolation and dilation techniques, denoted as I:

[0024] Step e: The target regions in the query image feature are effectively activated under the guidance of the support prototype v s , and the segmentation prediction is also obtained after passing through the decoder of the meta-learner. The above process is:

[0025]

[0026] Among them, D m (·) represents the decoder network of the meta-learner, which is composed of an atrous spatial pyramid pooling module and two convolutional layers with residual connections; F guidance (·) is a module in the few-shot semantic segmentation task, which plays the role of transmitting annotation information from the support branch to the query branch to provide specialized segmentation clues;

[0027] Step f: Use the evaluation result of the scene difference between the support-query image pair to adjust the rough prediction generated by the meta-learner; specifically, first integrate the foreground probability maps generated by the base learner to obtain a prediction for the background region with respect to the few-shot semantic segmentation task. The process is as follows:

[0028]

[0029] Among them, The superscript "f" represents the foreground, and the subscript "b" represents the base learner;

[0030] Step g: Calculate the Gram-style matrix using the low-level features extracted from the fixed backbone network :

[0031]

[0032]

[0033] Among them, N = H1×W1; F reshape (·) converts the size of the input tensor to C1×N, and after calculating the Gram matrices G s , G q corresponding to the two features respectively, evaluate the Frobenius norm value of their difference matrix G s -G q to obtain a comprehensive index ψ for guiding the adjustment process. The above process is as follows:

[0034] ψ = ||G s -G q || F , (10)

[0035] Among them, ||·|| F represents the Frobenius norm value of the input matrix;

[0036] Step h: The rough results generated by the base learner and the meta-learner are integrated under the guidance of the adjustment factor ψ to generate a fine segmentation prediction p f , and the process is as follows:

[0037]

[0038]

[0039] Among them, p m , p b represent the prediction results of the meta - learner and the base - learner respectively. The superscripts "0" and "1" represent the background and the foreground respectively. F ensemble (·) and F ψ (·) are both 1×1 convolution operations with specific initialization parameters. represents the concatenation operation along the channel dimension;

[0040] Step i: The cross - entropy loss is used to evaluate the difference between the prediction results of the model at all spatial positions h×w and the ground truth. The process is as follows:

[0041]

[0042] Among them, n bs is the number of samples in each batch of training data.

[0043] Similar to the training method of the base - learner, the loss function of the meta - learner also uses the cross - entropy loss. The multi - class cross - entropy loss is adjusted to the binary cross - entropy loss to evaluate the difference between the model prediction p m and the ground truth m q . The process is as follows:

[0044]

[0045] Among them, n e is the number of training scenarios in each batch of data;

[0046] The total loss in the meta - training stage includes the prediction loss of the meta - learner and the final loss L final of the prediction result and the ground truth after passing through the integration module. It is:

[0047] L = L final + λL meta , (15)

[0048]

[0049] Among them, λ is the loss function adjustment factor, which is set to 1.0. Its magnitude can be used to control the relative learning speed between the meta - learner and the integration module;

[0050] Repeat Step a to Step i;

[0051] Step 3: Network inference: The inference process of the network only includes Step a to Step h, without the process of calculating the loss and backpropagation in Step i.

[0052] The few-shot semantic segmentation method is not limited to a specific meta-learner and base-learner, and is extended to few-shot 3D point cloud segmentation, remote sensing image segmentation, general and cross-domain settings.

[0053] Beneficial effects

[0054] A few-shot semantic segmentation method based on a learner integration strategy proposed by the present invention aims at the problem of base class bias existing in previous related work, that is, it is easy to wrongly activate the target area of the base class. The present invention proposes to introduce an additional auxiliary branch (base-learner) on the basis of a traditional model (meta-learner) to explicitly identify the base class target, and then adaptively integrate the rough prediction results of the two learners to generate an accurate segmentation mask. Considering the sensitivity of the meta-learner, the present invention further introduces an adjustment factor to estimate the difference between input images to promote model integration reasoning. Generally speaking, the present invention provides a novel and general solution to improve the problem of wrongly activating the base class target by the existing few-shot segmentation model, which is not limited to a specific base-learner and meta-learner. While significantly improving the performance of the existing segmentation network, it can be extended to more practical and challenging general settings, showing good application prospects and potential.

[0055] The beneficial effects of the present invention are as follows: Introducing an additional auxiliary branch (base-learner) on the basis of a traditional model (meta-learner) can better help the network suppress irrelevant areas that are prone to misclassification. Since the base-learner is trained on a dataset with a large number of labeled samples, its prediction results are very reliable. In addition, the difference between input images is considered during learner integration, further promoting the complementary effect of the two branches. The solution provided by the present invention is general, not limited to a specific meta-learner and base-learner, and can be naturally extended to few-shot 3D point cloud segmentation, remote sensing image segmentation, general and cross-domain settings, etc., showing its good application prospects and development potential. Description of the drawings

[0056] Figure 1 is a schematic diagram of the few-shot semantic segmentation network structure based on the learner integration strategy in the present invention;

[0057] Figure 2 is a flowchart for calculating the adjustment factor ψ in the present invention;

[0058] Figure 3 is the result image of standard few-shot semantic segmentation using the method of the present invention;

[0059] Figure 4 is the result image of general few-shot semantic segmentation using the method of the present invention. Detailed implementation manners

[0060] The present invention will be further described in conjunction with embodiments and drawings as follows:

[0061] For the few-shot semantic segmentation method based on the learner integration strategy of the present invention, it will be elaborated in detail from three aspects: network construction, network training, and network inference:

[0062] Network construction: First, build the few-shot semantic segmentation method based on the learner integration strategy of the present invention, which mainly includes three parts: the base learner, the meta-learner, and the integration module. Among them, the base learner and the meta-learner share the same feature extraction (backbone) network, such as the Residual Network ResNet50, ResNet101, or the VGG16 recognition network. Generally speaking, the parameters of this part need to be fixed in the few-shot segmentation task. It should be noted that the learner integration strategy proposed by the present invention is not limited to a specific base learner and meta-learner. Here, the PSPNet and PFENet networks are taken as examples for introduction to facilitate analysis and understanding. The base learner is sent to the pyramid pooling module after passing through the feature extraction network to help the model capture features of different scales, which plays a very good role in promoting the segmentation of targets of various sizes. Finally, it is sent to the decoder network composed of convolutional layers to output the multi-class segmentation prediction results. Different from this, the meta-learner accepts two inputs: the support image and the query image. After passing through the feature extraction network respectively, dimensionality reduction is performed to improve the calculation efficiency. The masked average pooling method is then adopted, which condenses the target region of the support image into an abstract feature vector with strong representational ability, also called a prototype. The prototype obtained from the support image features is fused with the query features to activate the relevant target regions. Finally, the convolutional layer with a residual structure is passed through to generate the few-shot segmentation prediction results. The integration module is mainly used to integrate the rough prediction results output by the two learners. Its core idea is to use the prediction of the base learner for the foreground region (base class) to help the meta-learner suppress the prediction for the background region (base class). In addition, considering the sensitivity of the meta-learner to the differences between the input support samples and query samples, the present invention extracts the low-level features of the two branches and performs differential evaluation by calculating the norm of the difference of the Gram matrix (also called the style matrix) to obtain the adjustment factor. Finally, the prediction results of the two and the adjustment factor are fused in the form of feature map concatenation to further optimize the segmentation mask of the rough prediction. The above process can be described more specifically as follows:

[0063] Step 1: Given a query image It first passes through the feature extraction network E(·) shared with the meta-learner and an additional convolutional block to obtain the intermediate feature map This process can be expressed as:

[0064]

[0065] Among them, F conv (·) represents consecutive convolution operations. Taking the ResNet series models as an example, it represents the last convolutional block, that is, Block4. c, h, and w represent the dimensions of channels, height, and width respectively, and h×w represents the smallest spatial resolution among all the extracted features.

[0066] Step 2: Subsequently, the decoder network D b (·) gradually increases the spatial scale of the intermediate feature map to enrich the segmentation clues related to details and finally generate the prediction result. The above process can be defined as:

[0067]

[0068] Among them, the softmax(·) operation is performed along the channel dimension to generate the prediction probability map p b , and N b represents the number of base classes in the current data partition. Taking PSPNet as an example, the decoder network D b (·) includes a pyramid pooling module for increasing the receptive field, convolutional layers for integrating features at various scales, and an output layer composed of 1×1 convolutions.

[0069] Step 3: Given a support set S = {x s , m s} and a query image , the goal of the meta-learner is to segment the objects in the query image that are of the same category as the annotation mask m s under the guidance of the support set S. After using the backbone network to extract the features of the support and query images, a 1×1 convolutional layer is immediately added to reduce the channel dimension and generate the intermediate feature map. This process can be expressed as:

[0070]

[0071]

[0072] Among them, E(·) represents the feature extraction network shared with the base class learner, and F 1×1 (·) is a 1×1 convolution that can encode the input feature dimension to a lower dimension. and respectively represent the intermediate feature maps of the support image and the query image extracted after passing through the backbone network and the 1×1 convolution.

[0073] Step 4: Further, the present invention utilizes the masked average pooling method. According to the support feature map and its corresponding mask m s Calculate the important segmentation clues related to the category, also known as the prototype v s , and this process can be summarized as:

[0074]

[0075] where F pool (·) represents the average pooling operation, and ⊙ represents the Hadamard product. I(·) is a function that can transform the support image mask into the same shape and size as the support image features through interpolation and dilation techniques, and can be simply expressed as I:

[0076] Step 5: Subsequently, the target region in the query image features is effectively activated under the guidance of the support prototype v s , and after passing through the decoder of the meta-learner, the segmentation prediction is also obtained. The above process can be defined as:

[0077]

[0078] where D m (·) represents the decoder network of the meta-learner, which is generally composed of an atrous spatial pyramid pooling module and two convolutional layers with residual connections. F guidance (·) is a very important module in the few-shot semantic segmentation task, which plays the role of transmitting annotation information from the support branch to the query branch to provide specialized segmentation clues. In the present invention, the specific implementation is to dilate the support prototype v s to the same spatial size as the query image features , and then concatenate it with the query features, and further transmit and integrate relevant information through the decoder network D m (·).

[0079] Step 6: Considering that the meta-learner is often very sensitive to the quality of the support image, the present invention further proposes to use the evaluation result of the scene difference between the support-query image pair to adjust the rough prediction generated by the meta-learner. Specifically, first integrate the foreground probability map generated by the base learner to obtain the prediction of the background region relative to the few-shot semantic segmentation task. This process can be expressed as:

[0080]

[0081] where the superscript "f" of represents the foreground, and the subscript "b" represents the base learner.

[0082] Step 7: Subsequently, low-level features extracted from the fixed backbone network are used to calculate the Gram-style matrix. Since the calculation methods of the two features are similar, here, taking the support image features as an example, the related process can be summarized as follows:

[0083]

[0084]

[0085] where N = H1 × W1; F reshape (·) converts the size of the input tensor to C1 × N. After separately calculating the Gram matrices G s , G q corresponding to the two features, the difference matrix G s - G q is evaluated for the Frobenius norm value to obtain a comprehensive metric ψ for guiding the adjustment process. The above process can be expressed as:

[0086] ψ = ||G s - G q || F , (10)

[0087] where ||·|| F represents the Frobenius norm value of the input matrix.

[0088] Step 8: Finally, the rough results generated by the base learner and the meta-learner are integrated under the guidance of the adjustment factor ψ, further generating a refined segmentation prediction p f , and this process can be defined as:

[0089]

[0090]

[0091] where p m , p b represent the prediction results of the meta-learner and the base learner respectively. The superscripts "0" and "1" represent the background and foreground respectively. F ensemble (·) and F ψ (·) are both 1×1 convolution operations with specific initialization parameters. The former aims to adjust the rough result of the meta-learner, while the latter aims to integrate the two learners. It should be noted that the specific initialization parameters are crucial for the fusion process of the two learners. It is necessary to ensure that the meta-learner has obtained a certain segmentation ability before introducing the base learner to help suppress the confusing base class regions. represents the concatenation operation along the channel dimension.

[0092] Network training: Generally speaking, the training process of the few-shot semantic segmentation method based on the learner integration strategy proposed by the present invention can be divided into two parts: base learner training, and meta-learner and integration module training. First, the training process of the base learner is introduced:

[0093] Different from the episodic learning paradigm widely adopted in the few-shot scenario, the present invention follows the paradigm of standard supervised learning to train the base learner. The cross entropy (CE) loss is used to evaluate the difference between the prediction results of the model at all spatial positions h×w and the ground truth, and this process can be expressed as:

[0094]

[0095] where n bs is the number of samples in each batch of training data. It is necessary to explain the reason for not jointly training the two learners here. The most direct way to predict the base class region in the query image is to follow the standard semantic segmentation network, such as PSPNet, DeepLab, etc. However, it is not practical to build such a large network on the basis of the original few-shot segmentation model, which will undoubtedly introduce a lot of additional parameters and slow down the inference speed. The present invention further attempts to design a unified framework in which the two learners share the backbone network, but advanced few-shot semantic segmentation methods often freeze the parameters of the backbone network during the training phase to enhance generalization. Such an operation is inconsistent with the training scheme of the standard segmentation model, which will undoubtedly affect the performance of the base learner. More importantly, it is unknown whether the base learner can adapt to the episodic learning paradigm commonly used in the few-shot domain. Therefore, a two-stage training strategy is adopted in the present invention.

[0096] Secondly, the training process of the meta-learner and the integration module is introduced:

[0097] Similar to the training method of the base learner, the loss function of the meta-learner also uses the cross entropy loss. The difference is that the training categories of the few-shot segmentation model are two categories (foreground class and background class), so the multi-class cross entropy loss is adjusted to the binary cross entropy (BCE) loss to evaluate the prediction p m of the model and the difference between the ground truth m q , and this process can be summarized as:

[0098]

[0099] where n eis the number of training scenarios in each batch of data. The total loss in the meta-training stage includes the prediction loss of the meta-learner and the final loss L between the prediction result and the ground truth after passing through the integration module final , which can be defined as:

[0100] L = L final + λL meta , (15)

[0101]

[0102] where λ is the loss function adjustment factor, which is set to 1.0, and its magnitude can be used to control the ratio of the learning speeds between the meta-learner and the integration module.

[0103] Network inference: The model after two-stage training has two different inference methods when dealing with the segmentation task on new classes: direct inference and fine-tuning inference. For the case where the number of support samples is small, the direct inference method often has better performance than the fine-tuning inference method because fewer training samples are prone to overfitting, which in turn hinders the generalization ability for unknown query samples of new classes. According to experience, when the number of support samples reaches a certain number, such as 30, the fine-tuning inference method can obtain a model with better segmentation performance. It should be noted that the present invention adopts a different training / inference strategy from the previous ones in the case of having multiple available labeled samples K-shot. Currently, most few-shot segmentation methods tend to calculate the mean of the prototypes corresponding to each sample extracted from the support branch, and then use it to guide the subsequent segmentation process. In such an inference method, the contribution of each sample is considered the same. When there are significant differences between the support samples and the query samples, equal contributions often result in suboptimal results because poor-quality support samples cannot provide more relevant guidance for the current task. Therefore, the present invention further proposes to adaptively estimate the fusion weight of each support sample based on the adjustment factor ψ, where a smaller ψ (smaller difference between samples) corresponds to a greater contribution, and vice versa.

[0104] Specifically, given the adjustment factor ψ corresponding to each support sample i , first integrate it into a unified vector through a concatenation operation Subsequently, design two fully connected layers to generate the fusion weight η of the support samples. The above process can be defined as:

[0105]

[0106] where represents the weight of the fully connected layer, and r represents the dimensionality reduction factor. Finally, obtain the final adjustment factor ψ for integration by means of weighted summation.

[0107] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0108] Operating environment of this embodiment: 10 Casteng HeLi GPU servers (CPU is Intel Xeon E5-2698, GPU is RTX 2080Ti with 12G), the operating system of the server is Ubuntu 16.04.5 LTS, and the relevant dependencies and corresponding versions are Python 3.8, PyTorch 1.7.0, cuda11.0, torchvision 0.8.1, tensorboardX 2.14. The experiment uses the publicly available few-shot semantic segmentation datasets PASCAL-5 i (including 20 semantic categories, 5953 training images, 1449 test images, 4 data partitioning methods), COCO-20 i (including 80 semantic categories, 82081 training images, 40137 test images, 4 data partitioning methods), and the annotation format is segmentation mask (.png). The training adopts the mini-batch method, and the optimizer adopts SGD. In the base learner training stage, the number of samples per batch is 12, and the initial learning rate is 0.0025; in the meta-learner training stage, the number of samples per batch is 8, and the initial learning rate is set to 0.05.

[0109] The specific implementation process of the present invention is as follows:

[0110] 1. Network construction

[0111] The overall framework of the few-shot semantic segmentation network based on the learner integration strategy is as Figure 1 shown, mainly including three parts: the base learner, the meta-learner, and the integration module. Among them, the backbone network shared by the two learners is the VGG16 classification network without the fully connected layer; the base learner is the PSPNet standard segmentation network, and its decoder network includes a pyramid pooling module, a dimensionality reduction convolutional layer, and an output convolutional layer; the meta-learner is the PFENet few-shot segmentation network, and its decoder includes an atrous spatial pyramid pooling and 2 convolutional layers with residual connections; the integration module is composed of 1×1 convolutional layers. When conducting experiments on PASCAL-5 i and COCO-20 i N b takes 15 and 60 respectively. The low-level features used to calculate the style Gram matrix are the output feature maps of Block2. Taking the PASCAL-5 i dataset as an example, its feature size is

[0112] The specific processing procedure of the entire network is as follows:

[0113] (1) Input query - support image pair and support mask ε = {x q , x s , m s} where the query image x q obtains high-level features related to the base learner after passing through the backbone network (Block1-4) obtains intermediate-level features related to the meta-learner after passing through part of the backbone network (Block1-3) and downsampling convolution Similarly, the support image x s also obtains intermediate-level features related to the meta-learner

[0114] (2) Capture the support image features in the target region through the masked average pooling operation and compress it to obtain the abstract class prototype v s .

[0115] (3) Expand the class prototype v s to the same spatial size as the query image features , and then concatenate the two features. This process can be defined as:

[0116]

[0117] where Expand(·) represents the expansion operation, which can stack the input vector to a given size. represents the concatenation operation.

[0118] (4) The query image features guided by the prototype and the high-level query image features passing through the backbone network are respectively fed into the meta-decoder D m (·) and the base decoder D b (·) for segmentation prediction to obtain the prediction probability maps and

[0119] (5) According to the foreground / background concept, the prediction probability map p m of the meta-learner can be divided into a foreground prediction map and a background prediction map ; the probability prediction map p m of the base-learner can be divided into a foreground prediction map and a background prediction map Two parts, where the former is summed to obtain an overall prediction regarding the foreground The latter is not used in subsequent operations due to its complex composition.

[0120] (6) For the integration module, first calculate the corresponding style Gram matrix based on the low-level features of the query-support image pair As shown, subsequently calculate the difference G such as Figure 2 shown, and then obtain the adjustment factor ψ by calculating the F-norm of G s -G q

[0121] (7) Expand the adjustment factor to and then concatenate it with the foreground and background prediction maps provided by the meta-learner respectively. Then, reduce the dimension from 2 to 1 through a 1×1 convolution to obtain the adjusted meta-learner prediction result

[0122] (8) Similarly, concatenate the foreground prediction of the base-learner with the background prediction of the adjusted meta-learner and then use a 1×1 convolution to reduce the dimension, finally obtaining the prediction map regarding the background

[0123] (9) Concatenate the foreground and background prediction probability maps to obtain Finally obtain the segmentation mask by determining the index of the maximum value (argmax)

[0124] 2. Network Training

[0125] The training process of the few-shot semantic segmentation network based on the learner integration strategy is divided into two stages: base-learner training, and co-training of the meta-learner and the integration module. After the first stage of training, the network parameters of the base-learner are frozen. During the second stage of training, the meta-learner shares the backbone network (frozen) of the base-learner and trains the remaining part of the parameters and the parameters of the integration module in an end-to-end manner.

[0126] (1) The loss function for the first stage of training is as follows:

[0127]

[0128] where i is the index of each batch of training samples, and CE represents the cross-entropy loss. p b；i , represent the prediction probability map of the base-learner and the ground truth mask of the base class respectively.

[0129] (2) The loss function for the second stage of training is as follows:

[0130] L​stage2 = L final + λL meta , (20)

[0131] where L final is the final loss of the integrated model, and L meta is the prediction loss of the meta-learner. λ = 1 is the weight factor. Specifically, these two parts of the loss are respectively defined as:

[0132]

[0133]

[0134] where i is the index of each batch of training episodes, and BCE represents the binary cross-entropy loss. represents the ground truth mask of the new class, p m；i represent the final predicted probability map and the meta-learner predicted probability map respectively.

[0135] 3. Performance Evaluation

[0136] The mean intersection over union (mIoU) and foreground-background IoU (FB-IoU) are selected to evaluate the effectiveness of the method of the present invention, and their definitions are as follows:

[0137]

[0138]

[0139] where C represents the total number of image classes included in the test set under the current data partition, that is, the number of new classes. The mIoU metric calculates the average of the IoU results for each class in the new classes, where IoU c represents the IoU metric corresponding to class c; while FB-IoU ignores the concept of the target class and regards the current task as a foreground / background segmentation task independent of classes, and only calculates the average of the foreground and background evaluation metrics IoU F and IoU B In general, mIoU can better reflect the generalization ability of the model and the quality of the prediction results compared to FB-IoU, so it is selected as the main evaluation metric in the few-shot semantic segmentation field. Table 1 shows the method of the present invention (introducing the base learner and the integration module) and the baseline scheme on the standard few-shot semantic segmentation datasets PASCAL-5 i and COCO-20 iFrom the comparison results above, it can be seen that the proposed scheme significantly improves the segmentation performance, indicating the effectiveness of suppressing irrelevant base class regions. The qualitative results of standard few-shot semantic segmentation and generalized few-shot semantic segmentation using the method of the present invention are as Figure 3 and Figure 4 shown.

[0140] Comparison of Quantitative Experimental Results in Table 1

[0141]

Claims

1. A few-shot semantic segmentation method based on a learner integration strategy, characterized in that The steps are as follows: Step 1, construct the network: The first network is a feature extraction network. One output end is connected to the base learner network and another meta-learner network. The other output end of the feature extraction network is connected to the meta-learner network. The output of the base learner network and the meta-learner network are connected to the integration module; Step 2, network training: Step a: Given a query image It first passes through the feature extraction network E(·) shared with the meta-learner and an additional convolutional block to obtain an intermediate feature map This process is as follows: Among them, F conv (·) represents consecutive convolution operations, which represent the last convolution block; c, h, and w represent the dimensions of channels, height, and width respectively, and h×w represents the smallest spatial resolution among all the extracted features; Step b: Subsequently, the decoder network D b (·) gradually increases the spatial scale of the intermediate feature map to enrich the segmentation clues related to details and finally generate the prediction result. The above process can be defined as: Among them, the softmax(·) operation is performed along the channel dimension to generate the prediction probability map p b , N b represents the number of base classes in the current data partition; the decoder network D b (·) includes a pyramid pooling module for increasing the receptive field, a convolutional layer for integrating features at various scales, and an output layer composed of 1×1 convolutions; Step c: Given a support set S = {x s ,m s } and a query image The goal of the meta-learner is to segment the query image with the annotation mask m under the guidance of the support set S. s Objects of the same category; after using the backbone network to extract support and query image features, a 1×1 convolutional layer is added to reduce the channel dimension and produce an intermediate feature map. The process is: Among them, E(·) represents the feature extraction network shared with the base learner, and F 1×1 (·) is a 1×1 convolution that encodes the input feature dimension to a lower dimension; and respectively represent the intermediate feature maps of the support image and the query image extracted after passing through the backbone network and the 1×1 convolution; Step d: The masked average pooling method is utilized to calculate the class-related important segmentation clues, called prototypes v, according to the support feature map and its corresponding mask m s The process is as follows: s ​ Among them, F pool (·) represents the average pooling operation, ⊙ represents the Hadamard product, and I(·) is a function that can transform the support image mask into the same shape and size as the support image features through interpolation and dilation techniques, denoted as I: Step e: Query image features The target region in s is effectively activated under the guidance of the support prototype v, and segmentation predictions are also obtained after passing through the decoder of the meta-learner. The above process is as follows: Among them, D m (·) represents the decoder network of the meta-learner, which is composed of an atrous spatial pyramid pooling module and two convolutional layers with residual connections; F guidance (·) is a module in the few-shot semantic segmentation task, which plays the role of transmitting annotation information from the support branch to the query branch to provide specialized segmentation clues; Step f: Use the evaluation result of the scene difference between the support-query image pairs to adjust the rough prediction generated by the meta-learner. Specifically, first integrate the foreground probability maps generated by the base learner to obtain the prediction of the background region for the few-shot semantic segmentation task. The process is as follows: Among them, the superscript "f" represents the foreground, and the subscript "b" represents the base learner; Step g: Compute the Gram style matrix using the low-level features extracted from a fixed backbone network Calculate the Gram style matrix: where N = H1×W1; F reshape (·) Convert the size of the input tensor to C1×N, and calculate the Gram matrices G corresponding to the two features respectively s , G q After that, for the difference matrix G s - G q Evaluate the F-norm value to obtain a comprehensive index ψ for guiding the adjustment process. The above process is as follows: ψ = ||G s -G q || F , (10) where ||·|| F represents the Frobenius norm value of the input matrix; Step h: The rough results generated by the base learner and the meta-learner are integrated under the guidance of the adjustment factor ψ to generate a refined segmentation prediction p f , and the process is as follows: Among them, p m , p b represent the prediction results of the meta-learner and the base-learner respectively. The superscripts "0" and "1" represent the background and foreground respectively. F ensemble (·) and F ψ (·) are both 1×1 convolution operations with specific initialization parameters. represents the concatenation operation along the channel dimension; Step i: Use the cross-entropy loss to evaluate the difference between the prediction result of the model at all spatial positions h×w and the ground truth. The process is as follows: where n bs is the number of samples in each batch of training data, Similar to the training method of the base learner, the loss function of the meta-learner also uses the cross-entropy loss. The multi-class cross-entropy loss is adjusted to the binary cross-entropy loss to evaluate the difference between the model prediction p m and the true value m q . The process is as follows: where n e is the number of training scenarios in each batch of data; The total loss in the meta-training stage includes the prediction loss of the meta-learner and the final loss \(L\) between the prediction result and the ground truth after passing through the integration module final , which is L = L final + λL meta , (15) Among them, λ is the loss function adjustment factor, which is set to 1.0, and its magnitude can be used to control the relative learning speed between the meta-learner and the integration module; Repeat Step a to Step i; Step 3, network inference: The inference process of the network only includes Step a to Step h, without the process of calculating the loss and backpropagation in Step i.

2. The few-shot semantic segmentation method based on the learner integration strategy according to claim 1, characterized in that: The few-shot semantic segmentation method is not limited to a certain meta-learner and base learner, and is extended to few-shot 3D point cloud segmentation, remote sensing image segmentation, general and cross-domain settings.

Images