A weakly supervised few-shot semantic segmentation method based on meta-learning framework

Abstract

The present invention discloses a weakly supervised small-sample semantic segmentation method based on a meta-learning framework. To address the problem that the weakly supervised semantic segmentation method based on CAM only focuses on the most discriminative area and cannot cover the entire target, the present invention adopts a weighted repeated class activation mapping method to generate better seed areas to cover more targets and improve the quality of pseudo-masks. To address the problem that traditional networks need to be retrained to learn new tasks, resulting in inefficiency and lack of intelligence, the present invention adopts a small-sample semantic segmentation method based on a meta-learning algorithm to complete new segmentation tasks by learning a small number of samples, thereby improving the generalization performance of neural networks.

Description

Technical Field

[0001] The present invention relates to a weakly supervised small sample semantic segmentation method based on a meta-learning framework, belonging to image recognition technology. Background Art

[0002] Existing statistics show that the segmentation and annotation time for a single 1280×720 pixel image is approximately 1.5 hours, making pixel-by-pixel labeling of large image datasets prohibitively expensive. In contrast, weakly supervised labels, such as bounding boxes, points, scribbles, and image-level labels, are more readily available. Image-level labels, which simply mark the object categories contained in an image, offer the lowest level of supervision and are the easiest to obtain. Class activation mapping (CAM) is a commonly used weakly supervised semantic segmentation method because it can visualize salient regions in an image.

[0003] like Figure 1 The figure shows a traditional weakly supervised semantic segmentation network, which uses CAM to generate seed regions to provide cues for semantic segmentation. CAM can visualize salient areas in an image and has therefore become a commonly used weakly supervised semantic segmentation method. However, CAM's direct pseudo-label generation has significant drawbacks. It only focuses on the most discriminative regions and fails to cover the entire object. In particular, the model needs to be retrained to learn new tasks, which is both inefficient and lacks intelligence.

[0004] Early on, Brendan Lake et al. proposed human-level concept learning through probabilistic programming reasoning, laying the foundation for the development of few-shot learning and meta-learning. Few-shot learning rapidly deploys models on new tasks using a small number of samples. Meta-learning can be viewed as a method for achieving few-shot semantic segmentation, and its learning strategy effectively addresses the small-shot problem. Meta-learning algorithms are used to update initialization parameters to adapt to new tasks, leveraging prior experience to guide new tasks.

[0005] Recently, several methods for weakly supervised semantic segmentation using meta-learning have been proposed. Based on the MAML method, Gama et al. implemented weakly supervised semantic segmentation on medical and agricultural images using sparse annotations. Lee et al. established an encoder meta-learner to achieve semantic segmentation based on image-level labels. Specifically, they calculated a prototype loss during training and used k-nearest neighbor classification to obtain segmentation results during inference. However, the main network used, DeepLabV3+, is fixed after pre-training, and all parameters are no longer updated. Summary of the Invention

[0006] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a weakly supervised small-sample semantic segmentation method based on a meta-learning framework; to address the problem that the weakly supervised semantic segmentation method based on CAM only focuses on the most discriminative area and cannot cover the entire target, the present invention adopts a weighted repeated class activation mapping method to generate better seed areas to cover more targets and improve the quality of pseudo-masks; to address the problem that traditional networks need to be retrained to learn new tasks, resulting in inefficiency and lack of intelligence, the present invention adopts a small-sample semantic segmentation method based on meta-learning, uses existing experience to guide new tasks, and enables the network to learn.

[0007] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A weakly supervised small-sample semantic segmentation method based on a meta-learning framework includes a pseudo-mask generation module and a semantic segmentation network. The pseudo-mask generation module is used to learn a training set, and a pseudo-mask is generated based on sample images in the training set and their corresponding weak labels. The training set is divided into a support set and a query set. The meta-learning method and a Transformer module are combined to optimize the parameters of the semantic segmentation network and the classifier, and the pseudo-mask is used as an image label to participate in the training of the semantic segmentation network (the existing technology uses the real label as the image label to train the semantic segmentation network. In this case, a pseudo-mask is used to replace the real label, which is also called the ground truth). The support set is used to train the semantic segmentation network and predict the segmentation result. The trained semantic segmentation network is used to predict the segmentation result of the query set, and the Transformer module is used to update the weight of the classifier.

[0009] Specifically, the pseudo mask generation module adopts the weighted repeated class activation mapping method (WRCAM method) to obtain the pseudo mask W_CAM:

[0010] W_CAM=α×ReCAM1+(1-α)×ReCAM2

[0011] Where: α represents the adjustment weight, ReCAM1 is the pseudo mask obtained based on the repeated class activation mapping module I, and ReCAM2 is the pseudo mask obtained based on the repeated class activation mapping module II and the dual attention module; the structures of the repeated class activation mapping module I and the repeated class activation mapping module II are the same, and the dual attention module consists of a channel attention mechanism and a spatial attention mechanism.

[0012] The repeated class activation mapping module I is used to generate the initial pseudo mask, namely ReCAM1; the repeated class activation mapping module II plays a fine-tuning role, providing compensation information for ReCAM1, namely ReCAM2. The dual attention module plays a major role. As in the prior art, the channel attention mechanism and the spatial attention mechanism are respectively expressed as:

[0013] A c =Q(C(P c (F c (I))))

[0014] A s =Q(C(P s (F s (I))))

[0015] Among them: A c represents the output of the channel attention mechanism, A s represents the output of the spatial attention mechanism; F c (I) represents the input of the channel attention mechanism, F s (I) represents the input of the spatial attention mechanism; P c (·) represents the channel average pooling function, P s (·) represents the spatial average pooling function; C(·) represents the convolution operation, and Q(·) represents the adjustment function.

[0016] Specifically, the repeated class activation mapping module includes two class activation mapping modules, respectively referred to as class activation mapping module I and class activation mapping module II;

[0017] The input image is passed through the Class Activation Mapping module I to obtain the image label score for each pixel position; this score is superimposed on the input image to form an initial mask, and the input image and the initial mask are passed through the Class Activation Mapping module II to obtain a new image label score for each pixel position; the new image label for each pixel position is weighted summed to obtain a repeated pseudo mask for the repeated Class Activation Mapping module:

[0018] CAM 1_k (x,y)=F(G(f(x,y)))

[0019] CAM=∑ω k CAM 1_k (x,y)

[0020] CAM 2_k (x,y)=F(G(f(x,y)+CAM 1_k (x,y)))

[0021] ReCAM=∑ω k CAM 2_k (x,y)

[0022] Where: f(x,y) represents the feature value of the (x,y) position of the input image, G(·) represents the global average pooling function, and F(·) represents the fully connected function; CAM 1_k(x,y) represents the score of the (x,y) position of the input image obtained by the class activation mapping module I belonging to the image label k, CAM 2_k (x,y) represents the score of the (x,y) position of the input image obtained by the class activation mapping module II belonging to the image label k; ω k The weight that the (x,y) position of the input image belongs to image label k.

[0023] Specifically, when training the repetitive class activation mapping module, class activation mapping module I is trained first. After the CAM of class activation mapping module I converges, class activation mapping module II is trained again until ReCAM converges. This case targets the classification problem of multiple class labels. A sample image has multiple class labels, so a multi-label soft margin loss supervision function is used to supervise the training of class activation mapping module I and class activation mapping module II:

[0024]

[0025] Where: p[k]∈{0,1} represents the image label prediction of the input image by the class activation mapping module, p[k]=0 means that the predicted input image does not contain image label k, and p[k]=1 means that the predicted input image contains image label k; q[k]∈{0,1} represents the true image label of the input image, q[k]=0 means that the input image does not contain image label k, and q[k]=1 means that the input image contains image label k; K represents the total number of image labels.

[0026] Specifically, the total loss L of the pseudo mask generation module is:

[0027]

[0028] Wherein: Loss1 represents the loss of class activation mapping module I in repeated class activation mapping module I, and Loss2 represents the loss of class activation mapping module II in repeated class activation mapping module II.

[0029] Specifically, the present invention uses a meta-learning method to realize weakly supervised semantic segmentation in a small sample scenario. The meta-learning method adopts an inner and outer double-loop mechanism. The inner loop uses the support set to train the semantic segmentation network and classifier, and obtains the optimized parameters of the semantic segmentation network and the initial weights of the classifier through the gradient descent method; the outer loop uses the Transformer module to update the weights of the classifier.

[0030] Specifically, the Transformer module is a transformer composed of a multi-head attention mechanism, that is, n heads are used to perform n linear mappings to map the subspaces represented by different sub-regions, so that the classifier pays attention to the information from the subspaces of different regions; since the transformer module has a strong global information perception ability, the multi-head attention mechanism can be used to process intra-class differences and inter-class similarities.

[0031] Specifically, the Transformer module updates the weights of the classifier according to the following function:

[0032]

[0033] Where: * represents the weight of the updated classifier, and ω represents the weight of the classifier before the update; represents the first linear layer of the classifier, Multi_Att(·) represents the multi-head attention mechanism, and F represents the feature map of the input image.

[0034] Beneficial effects: The weakly supervised small-sample semantic segmentation method based on the meta-learning framework provided by the present invention has the following advantages over the existing technology: 1. For small-sample scenarios, a weakly supervised semantic segmentation method based on a dual-loop optimization meta-learning algorithm is proposed; 2. No pixel-by-pixel annotation is required during network training, and only simple category annotation (image-level label) is used to achieve weakly supervised semantic segmentation of images; 3. In the mask generation stage, the WRCAM method is used to expand the target coverage and improve the quality of the generated pseudo-mask; 4. In view of the problem that traditional networks are only trained and tested for specific tasks and have poor generalization, a small-sample semantic segmentation method based on meta-learning is proposed, which utilizes prior knowledge and self-adjusts according to new tasks. During training, better performance can be achieved on new tasks through a few steps of gradient updates; 5. In the segmentation stage, the meta-learning method is combined with the transformer module to obtain an efficient segmentation network with better generalization ability; 6. In the experimental stage, sufficient experiments were carried out, and the comparison results with SOTA proved that the method of the present invention has achieved more advanced performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the traditional weakly supervised semantic segmentation network;

[0036] Figure 2 Schematic diagram of the weakly supervised semantic segmentation network of the present invention;

[0037] Figure 3 Schematic diagram of the structure of the activation mapping module for repeated classes;

[0038] Figure 4This is the semantic segmentation result of the dataset PACAL-5i. DETAILED DESCRIPTION

[0039] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] In the description of the present invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate description and simplify the description of the present invention. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation, and therefore should not be construed as limiting the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] Existing research work has mainly focused on supervised tasks, with less research on weakly supervised tasks. Since weakly supervised labels can effectively reduce the annotation cost of visual tasks, weakly supervised semantic segmentation has become one of the hot research topics. This case proposes a weakly supervised small-sample semantic segmentation method based on a meta-learning framework. This method uses prior knowledge and self-adjusts according to new tasks, making the network efficient and capable of generalizing well to new tasks. In the pseudo-mask generation stage, this case designs a WRCAM method based on a channel-space attention mechanism to enhance the coverage area of ​​the target in the pseudo-mask. In the small-sample semantic segmentation stage, through an optimization-based meta-learning method, the optimized pseudo-mask is used to achieve small-sample semantic segmentation. Experimental results show that this method is not only significantly better than the weakly supervised SOTA methods in the same field, but can also reach or even exceed some supervised SOTA methods.

[0042] like Figure 2 The figure shows the weakly supervised small-sample semantic segmentation method proposed in this case based on the meta-learning framework. First, the pseudo-mask generation module is used to learn the training set. Pseudo-masks are generated based on the sample images in the training set and their corresponding weak labels, and the training set is divided into two parts: the support set and the query set. Then, the parameters of the semantic segmentation network and the classifier are optimized by combining the meta-learning method and the Transformer module. The pseudo-mask is used as the image label in the training of the semantic segmentation network. The support set is used to train the semantic segmentation network and predict the segmentation results. Then, the trained semantic segmentation network is used to predict the segmentation results of the query set, and the Transformer module is used to update the classifier weights.

[0043] Part 1: Generate a pseudo mask

[0044] like Figure 2As shown, the pseudo mask generation module adopts the weighted repeated class activation mapping method (WRCAM method) to obtain the pseudo mask W_CAM:

[0045] W_CAM=α×ReCAM1+(1-α)×ReCAM2 (1)

[0046] Where: α represents the adjustment weight, ReCAM1 is the pseudo mask obtained based on the repeated class activation mapping module I, and ReCAM2 is the pseudo mask obtained based on the repeated class activation mapping module II and the dual attention module; the structures of the repeated class activation mapping module I and the repeated class activation mapping module II are the same, and the dual attention module consists of a channel attention mechanism and a spatial attention mechanism.

[0047] The WRCAM method can generate better seed areas to cover more targets. This method can effectively reduce the loss value and thus generate high-quality masks. The repeated class activation mapping module I is used to generate the initial pseudo mask, namely ReCAM1. The repeated class activation mapping module II plays a fine-tuning role, providing compensation information for ReCAM1, namely ReCAM2, to enhance the effective coverage area of ​​the pseudo mask. The dual attention module is used to extract important information. For example, in the prior art, the channel attention mechanism and the spatial attention mechanism are respectively expressed as:

[0048] A c =Q(C(P c (F c (I)))) (2)

[0049] A s =Q(C(P s (F s (I)))) (3)

[0050] Among them: A c represents the output of the channel attention mechanism, A s represents the output of the spatial attention mechanism; F c (I) represents the input of the channel attention mechanism, F s (I) represents the input of the spatial attention mechanism; P c (·) represents the channel average pooling function, P s (·) represents the spatial average pooling function; C(·) represents the convolution operation, and Q(·) represents the adjustment function.

[0051] like Figure 3As shown in the figure, the repeated class activation mapping module includes two class activation mapping modules, namely class activation mapping module I and class activation mapping module II. The input image is processed by class activation mapping module I to obtain an image label score for each pixel position. The score is superimposed on the input image to form an initial mask. The input image and the initial mask are processed by class activation mapping module II to obtain a new image label score for each pixel position. The new image label for each pixel position is weighted and summed to obtain a repeated pseudo mask for the repeated class activation mapping module:

[0052] CAM 1_k (x,y)=F(G(f(x,y))) (4)

[0053] CAM=∑ω k CAM 1_k (x,y) (5)

[0054] CAM 2_k (x,y)=F(G(f(x,y)+CAM 1_k (x,y))) (6)

[0055] ReCAM=∑ω k CAM 2_k (x,y) (7)

[0056] Where: f(x,y) represents the feature value of the (x,y) position of the input image, G(·) represents the global average pooling function, and F(·) represents the fully connected function; CAM1_ k (x,y) represents the score of the (x,y) position of the input image obtained by the class activation mapping module I belonging to the image label k, CAM 2_k (x,y) represents the score of the (x,y) position of the input image obtained by the class activation mapping module II belonging to the image label k; ω k The weight that the (x,y) position of the input image belongs to image label k.

[0057] When training the repeated class activation mapping module, first train the class activation mapping module I. After the CAM of the class activation mapping module I converges, train the class activation mapping module II until ReCAM converges. Use the multi-label soft margin loss supervision function to supervise the training of the class activation mapping module I and the class activation mapping module II:

[0058]

[0059] Where: p[k]∈{0,1} represents the image label prediction of the input image by the class activation mapping module, p[k]=0 means that the predicted input image does not contain image label k, and p[k]=1 means that the predicted input image contains image label k; q[k]∈{0,1} represents the true image label of the input image, q[k]=0 means that the input image does not contain image label k, and q[k]=1 means that the input image contains image label k; K represents the total number of image labels.

[0060] The total loss L of the pseudo mask generation module is:

[0061]

[0062] Wherein: Loss1 represents the loss of class activation mapping module I in repeated class activation mapping module I, and Loss2 represents the loss of class activation mapping module II in repeated class activation mapping module II.

[0063] Part II: Training the Semantic Segmentation Network and Classifier

[0064] Meta-learning enables machines to learn to learn, leveraging existing experience to guide new tasks. Meta-learning methods can be categorized into three approaches: 1. Metric learning methods for learning similar spaces, using multi-model meta-learning to quantify knowledge transfer between different models and tasks at a microscopic level; 2. Memory network methods that store learning experience and rapidly encode new tasks based on external memory; and 3. Gradient descent-based methods for fine-tuning optimization algorithms.

[0065] The meta-learning method in this case adopts an inner and outer double loop mechanism. The inner loop uses the support set to train the semantic segmentation network and classifier, and obtains the optimization parameters of the semantic segmentation network and the initial weights of the classifier through the gradient descent method; the outer loop uses the Transformer module to update the weights of the classifier. The Transformer module is a transformer composed of a multi-head attention mechanism, that is, it uses n heads to perform n linear mappings to map the subspaces represented by different sub-regions, so that the classifier pays attention to the information from the subspaces of different regions, thereby extracting more feature information, and using residual connections to reduce information loss. The Transformer module updates the weights of the classifier according to the following function:

[0066]

[0067] Where: * represents the weight of the updated classifier, and ω represents the weight of the classifier before the update; represents the first linear layer of the classifier, Multi_Att(·) represents the multi-head attention mechanism, and F represents the feature map of the input image.

[0068] Part III: Experiment

[0069] 4.1 Experimental Details

[0070] The datasets used for the experiments are Pascal-5i and the original Pascal VOC 2012. Pascal-5i is a combination of the widely adopted Pascal VOC 2012 and SDS. Within each classification, there are 20 categories, 15 of which are used for training and 5 for testing. It is divided into four parts: ① split = 0, [airplane, bicycle, bird, boat, bottle]; ② split = 1, [bus, car, cat, chair, cow]; ③ split = 2, [dining table, dog, horse, motorcycle, person]; ④ split = 3, [potted plant, sheep, sofa, train, TV / monitor]. Pascal VOC 2012 and Pascal-5i have the same classes, except that the former dataset has more training images than the latter.

[0071] During WRCAM's pseudo-mask generation, ResNet was used as the backbone network. CAM's learning rate was 0.01, weight decay was 1e-4, and the adjustment weight α in formula (1) was set to 0.5. During the small sample segmentation phase, ResNet was still used as the backbone network. During training, the learning rate was set to 0.0025, momentum was 0.9, and model dropout was 0.1. The following experiments were conducted using an RTX2080Ti.

[0072] The mean Intersection over Union (mIoU) is a metric for measuring semantic segmentation results. Specifically, the Intersection over Union (IoU) of each category is calculated based on the confusion matrix, and the mIoU is obtained by averaging the IoU scores of each category.

[0073] 4.2 Comparison of experimental results

[0074] In this section, we conducted several experiments to demonstrate the effectiveness of our proposed method. By comparing the experimental results with several recently published few-shot semantic segmentation methods, we concluded that our model achieves improved segmentation performance. Table 1 shows the mIoU values ​​of various supervised methods on PASCAL-5i for different backbones and parts. Given that single-shot learning can better illustrate the role of machine learning in solving few-shot learning problems, we focused on single-shot learning and compared our method with other few-shot semantic segmentation methods. Although several methods, such as PGNet and RPMMs, achieved higher average mIoU than ours, their per-part mIoU values ​​varied significantly, indicating that their models performed poorly on specific classes. In contrast, our model showed a relatively mild trend, with smaller differences. This demonstrates the good stability and generalization of our model. Under the same backbone network conditions, our results outperformed PPNet and CWT overall, and partially outperformed RPMMs and DAN. Therefore, our weakly supervised semantic segmentation performance is comparable to that of supervised state-of-the-art few-shot methods.

[0075] Table 1 Comparison of MIoU values ​​between this application and supervised small sample methods on PASCAL-5i

[0076]

[0077]

[0078] Given the limited research on weakly supervised few-shot semantic segmentation, this paper also compares this method with weakly supervised semantic segmentation methods published in recent years. PLML and TOSFL are the latest weakly supervised few-shot semantic segmentation methods, and our method significantly outperforms these methods. This shows that our weakly supervised few-shot semantic segmentation model has better generalization and robustness. Results compared with other weakly supervised semantic segmentation methods show that these methods perform better on specific datasets because they do not consider the network's generalization performance for new tasks. For example, our mIoU value is slightly lower than DSNA, WSIAL, SEAM, and GWSM. However, it is encouraging that under VGG16, the performance of this model is still higher than SGDN and DSNA, and close to SEAM-base and DSNA.

[0079] Table 2 Comparison of MIoU values ​​between this application and the weakly supervised small sample method on PASCAL-5i

[0080]

[0081]

[0082] Detailed experimental results on the larger Pascal VOC2012 dataset are listed in Table III, where Q is the predicted result for the query set. The results show that the trends in the four components are consistent with previous experiments. Even on larger training sets, our approach maintains its performance, further demonstrating the robustness of our model.

[0083] Table 3 Experimental results of this application on Pascal VOC2012

[0084]

[0085] Figure 4 Figure 6 shows the segmentation performance of our method during the inference phase. The first column shows the ground truth for the support set, the second column shows the ground truth for the query set, and the third column shows the predictions for the query set. This section lists different scenarios that affect segmentation performance. The results show that the model achieves the best segmentation results when the support and query sets are simple and the target is well-defined. In the first and second rows, the learned objects of car and cat are clear and complete, with a concise background, allowing the model to accurately segment the query target. In the third row, the support objects are partially occluded, but the background is relatively simple. For the query data, the model achieves relatively complete segmentation. In the next row, the target is incomplete in the support set, but most of the target objects are correctly segmented in the query. Surrounding objects with similar appearance to the target can confuse the model and lead to segmentation errors, as shown in row 5. As shown in row 6, although the model learns features well, it mistakenly identifies a portion of a car reflected in the water as part of the target. The segmentation results shown in the last row show that the model fails to accurately segment objects in the query set when multiple small objects are present and heavily occluded. Specifically, a chair and part of a table are both identified as chairs in the query set.

[0086] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0087] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0088] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the scope of protection of the present invention.

Claims

1. A weakly supervised small-sample semantic segmentation method based on a meta-learning framework, characterized by: The method includes a pseudo-mask generation module and a semantic segmentation network. The pseudo-mask generation module is used to learn the training set, and pseudo-masks are generated according to sample images in the training set and their corresponding weak labels. The training set is divided into a support set and a query set. The parameters of the semantic segmentation network and the classifier are optimized by combining the meta-learning method and the Transformer module. The pseudo-mask is used as the image label to participate in the training of the semantic segmentation network. The support set is used to train the semantic segmentation network and predict the segmentation results. The trained semantic segmentation network is used to predict the segmentation results of the query set, and the Transformer module is used to update the weights of the classifier. The pseudo-mask generation module adopts the weighted repeated class activation mapping method to obtain the pseudo-mask W_CAM: W_CAM=α×ReCAM1+(1-α)×ReCAM2 Where: α represents the adjustment weight, ReCAM1 is the pseudo mask obtained based on the repeated class activation mapping module I, and ReCAM2 is the pseudo mask obtained based on the repeated class activation mapping module II and the dual attention module; the structures of the repeated class activation mapping module I and the repeated class activation mapping module II are the same, and the dual attention module consists of a channel attention mechanism and a spatial attention mechanism.

2. The weakly supervised small-sample semantic segmentation method based on a meta-learning framework according to claim 1 is characterized by: The repeated class activation mapping module includes two class activation mapping modules, namely class activation mapping module I and class activation mapping module II; The input image is passed through the Class Activation Mapping module I to obtain the image label score for each pixel position; this score is superimposed on the input image to form an initial mask, and the input image and the initial mask are passed through the Class Activation Mapping module II to obtain a new image label score for each pixel position; the new image label for each pixel position is weighted summed to obtain a repeated pseudo mask for the repeated Class Activation Mapping module: CAM 1_k (x,y)=F(G(f(x,y))) CAM=∑ω k CAM 1_k (x,y) CAM 2_k (x,y)=F(G(f(x,y)+CAM 1_k (x,y))) ReCAM=∑ω k CAM 2_k (x,y) Where: f(x,y) represents the feature value of the (x,y) position of the input image, G(·) represents the global average pooling function, and F(·) represents the fully connected function; CAM 1_k (x,y) represents the score of the (x,y) position of the input image obtained by the class activation mapping module I belonging to the image label k, CAM 2_k (x,y) represents the score of the (x,y) position of the input image obtained by the class activation mapping module II belonging to the image label k; ω k The weight that the (x,y) position of the input image belongs to image label k.

3. The weakly supervised small-sample semantic segmentation method based on a meta-learning framework according to claim 1, characterized in that: When training the repeated class activation mapping module, first train the class activation mapping module I. After the CAM of the class activation mapping module I converges, train the class activation mapping module II until ReCAM converges. Use the multi-label soft margin loss supervision function to supervise the training of the class activation mapping module I and the class activation mapping module II: Where: p[k]∈{0,1} represents the image label prediction of the input image by the class activation mapping module, p[k]=0 means that the predicted input image does not contain image label k, and p[k]=1 means that the predicted input image contains image label k; q[k]∈{0,1} represents the true image label of the input image, q[k]=0 means that the input image does not contain image label k, and q[k]=1 means that the input image contains image label k; K represents the total number of image labels.

4. The weakly supervised small-sample semantic segmentation method based on a meta-learning framework according to claim 1, characterized in that: The total loss L of the pseudo mask generation module is: Wherein: Loss1 represents the loss of class activation mapping module I in repeated class activation mapping module I, and Loss2 represents the loss of class activation mapping module II in repeated class activation mapping module II.

5. The weakly supervised small-sample semantic segmentation method based on a meta-learning framework according to claim 1, characterized in that: The meta-learning method adopts an inner and outer double loop mechanism. The inner loop uses the support set to train the semantic segmentation network and classifier, and obtains the optimized parameters of the semantic segmentation network and the initial weights of the classifier through the gradient descent method; the outer loop uses the Transformer module to update the weights of the classifier.

6. The weakly supervised small-sample semantic segmentation method based on a meta-learning framework according to claim 1, characterized in that: The Transformer module is a transformer composed of a multi-head attention mechanism, that is, n heads are used to perform n linear mappings to map the subspaces represented by different subregions, so that the classifier pays attention to the information from the subspaces of different regions.

7. The weakly supervised small-sample semantic segmentation method based on a meta-learning framework according to claim 6, characterized in that: The Transformer module updates the weights of the classifier according to the following function: Where: * represents the weight of the updated classifier, and ω represents the weight of the classifier before the update; represents the first linear layer of the classifier, Multi_Att(·) represents the multi-head attention mechanism, and F represents the feature map of the input image.

Images