Single-stage weakly supervised semantic segmentation method and device based on affinity and media

Abstract

This invention relates to a single-stage weakly supervised semantic segmentation method, device, and medium based on class affinity, comprising the following steps: acquiring an image to be segmented, inputting it into a single-stage weakly supervised semantic segmentation network based on class affinity, and obtaining a semantic segmentation result of the image to be segmented; wherein, the single-stage weakly supervised semantic segmentation network based on class affinity includes a segmentation head and a class affinity mining module; the segmentation head is jointly trained through two sets of pseudo-labels and is used to obtain the segmentation result based on the image to be segmented; one set of pseudo-labels is generated based on a global class activation map, which is generated based on a pre-trained classification head and the image to be segmented; the other set of pseudo-labels is generated based on a global class affinity region map, which is generated by the class affinity mining module constructed based on the global class activation map. Compared with the prior art, this invention can further improve the accuracy of semantic segmentation results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a single-stage weakly supervised semantic segmentation method, device and medium based on class affinity. Background Technology

[0002] Weakly supervised semantic segmentation is a challenging task that typically relies on image-level labels. Class activation maps (NAMRs) are a common technique used to extract category-specific regions from deep neural networks. After generating pseudo-labels using NMRs, a post-processing network combined with dense conditional random fields (CRFs) is used to optimize the pseudo-labels, which are then used as a supervisory signal to oversee the semantic segmentation model. However, this multi-stage paradigm often suffers from significant efficiency limitations due to the need to train multiple models. Currently, researchers have proposed efficient single-stage solutions for weakly supervised semantic segmentation, where pseudo-label generation and segmentation head training are performed simultaneously. However, their performance is often susceptible to the inherent fuzziness of NMRs. NMRs often struggle to accurately determine the boundaries and details of target objects, and directly generating pseudo-labels from NMRs results in poor quality supervisory signals for the segmentation head. Therefore, improving the use of NMRs to enhance the performance of weakly supervised semantic segmentation algorithms has become a key research challenge.

[0003] To address the issue of insufficient global information in weakly supervised semantic segmentation, the paper "Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers" (Ru L, Zhan Y, Yu B, et al. [C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022:16846-16855) proposes an end-to-end solution based on Vision Transformer. By leveraging the consistency between the Transformer's self-attention mechanism and semantic relevance, it proposes a method to extract semantic relevance from attention to generate more complete initial pseudo-labels. This method can ensure the local consistency of pseudo-labels, but it fails to effectively handle the consistency problem between the global and local views of the class activation graph, lacking dense local-to-global correspondences. This problem may lead to suboptimal performance of the class activation graph in cases lacking sufficient contextual information or with local targets, affecting the quality of pseudo-labels. Therefore, a semantic segmentation method needs to be designed to further improve semantic segmentation performance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art by providing a single-stage weakly supervised semantic segmentation method, device, and medium based on class affinity, thereby further improving the accuracy of semantic segmentation results.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A single-stage weakly supervised semantic segmentation method based on class affinity includes the following steps:

[0007] The image to be segmented is obtained and input into a single-stage weakly supervised semantic segmentation network based on class affinity to obtain the semantic segmentation result of the image to be segmented.

[0008] The single-stage weakly supervised semantic segmentation network based on class affinity includes a segmentation head and a class affinity mining module. The segmentation head is jointly trained using two sets of pseudo-labels to obtain segmentation results from the image to be segmented. One set of pseudo-labels is generated based on a global class activation map, which is generated based on a pre-trained classification head and the image to be segmented. The other set of pseudo-labels is generated based on a global class affinity region map, which is generated by the class affinity mining module based on the global class activation map. The class affinity mining module is constructed based on the self-supervised visual transformer DINO.

[0009] Furthermore, the specific process of generating pseudo-labels based on the global class activation graph is as follows:

[0010] The image to be segmented is divided into multiple image patches and abstracted, which are then fed into a pre-trained classification head to generate a global class activation map. The global class activation map is then input into a post-processing network to generate pseudo-labels.

[0011] Furthermore, the image to be segmented is divided into multiple image patches and abstracted using the Mix Transformer.

[0012] Furthermore, the process of generating pseudo-labels based on a global class affinity region graph is as follows:

[0013] A category-agnostic affinity graph is constructed based on the global class activation graph using the self-supervised visual transformer DINO.

[0014] The highly activated regions of the global class activation graph are selected as initial seeds and propagated on the class-agnostic affinity graph to obtain a global class affinity region graph.

[0015] The post-processing network generates pseudo-labels based on the global affinity region graph.

[0016] Furthermore, the accuracy of the global affinity region graph is improved through prototype learning methods.

[0017] Furthermore, in the prototype learning method, the prototype of each category is updated with momentum through the representation of the high-activation region of the corresponding category, as shown in the following expression:

[0018]

[0019] Among them, P c Let Q be the momentum prototype of the c-th category, where τ is the momentum and Q is the momentum prototype. c Let t be the set of highly activated regions, and t be the representation of the current highly activated region.

[0020] Furthermore, the single-stage weakly supervised semantic segmentation network based on class affinity also includes a class activation graph enhancement module, which generates local class activation graphs and performs dense mapping between the local class activation graphs and the global class activation graph.

[0021] Furthermore, the specific steps corresponding to the density are as follows:

[0022] The local class activation graph and the global class activation graph are aligned to the same spatial coordinates. The correspondence between the two is determined by the pseudo-labels generated by the global class activation graph, so that the parts with the same pseudo-labels have similar activation values.

[0023] The present invention also provides an electronic device, including a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the above-described method.

[0024] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. In the single-stage weakly supervised semantic segmentation network based on class affinity constructed in this invention, the segmentation head is jointly trained by two sets of pseudo-labels. One set of pseudo-labels is generated based on the global class activation map, and the other set of pseudo-labels is generated based on the global class affinity region map. The global class affinity region map is generated based on the global class activation map by the class affinity mining module constructed based on the self-supervised visual transformer DINO. Under the self-attention mechanism of DINO, a class-independent affinity map can be established, which can more accurately capture the correlation between different regions in the image to be segmented, and provide supplementary supervision signals for the segmentation head, thereby improving the semantic segmentation performance.

[0027] 2. This invention constructs a category-agnostic affinity graph based on a global class activation graph using a self-supervised visual transformer DINO. It selects highly activated regions of the global class activation graph as initial seeds and propagates them on the category-agnostic affinity graph to obtain a global class affinity region graph. A post-processing network is then used to generate pseudo-labels based on the global class affinity region graph, which helps to mitigate the influence of the inherent ambiguity of the global class activation graph and provides clear semantic segmentation auxiliary information.

[0028] 3. This invention introduces a class activation graph alignment enhancement module into a single-stage weakly supervised semantic segmentation network based on class affinity, in order to solve the problem that existing technologies fail to handle the global and local views of class activation graphs. Figure 1 To address the consistency issue, after generating the local class activation graph, this module performs a dense mapping between it and the global class activation graph. This ensures that the parts of the global and local class activation graphs with the same pseudo-labels have similar activation values, encouraging dense local-to-global class activation graph mappings. This improves the quality of the class activation graph, enhances the accuracy of pseudo-labels, and further improves semantic segmentation performance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of a single-stage weakly supervised semantic segmentation network based on class affinity. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0031] Example:

[0032] This embodiment provides a single-stage weakly supervised semantic segmentation method based on class affinity, including the following steps:

[0033] The image to be segmented is obtained and input into a single-stage weakly supervised semantic segmentation network based on class affinity to obtain the semantic segmentation result of the image to be segmented.

[0034] Single-stage weakly supervised semantic segmentation networks based on affinity, such as Figure 1 As shown, the system includes a segmentation head, a backbone network, a class activation map enhancement module, and a class affinity mining module. The segmentation head is jointly trained using two sets of pseudo-labels to obtain segmentation results from the image to be segmented. One set of pseudo-labels is generated based on a global class activation map, which is generated based on a pre-trained classification head and the image to be segmented. The other set of pseudo-labels is generated based on a global class affinity region map, which is generated by the class affinity mining module based on the global class activation map.

[0035] Specifically, for each batch of images, they are first fed into a Mix Transformer (a variant of the Vision Transformer, see the paper Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929,2020) to obtain a token for each image patch. The image patch tokens are then fed into the classification head and the segmentation head to obtain classification scores and segmentation outputs (the classification head is pre-trained using existing image-level annotations). Then, a global class activation map is generated based on the weights of the classification head. The global class activation map is fed into a post-processing network (see the paper Ahn J, Kwak S. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2018:4981-4990) to generate a set of pseudo-labels for supervising the segmentation head.

[0036] Building upon this, this invention selects highly activated regions from the global class activation map as seeds, which are then fed into a class affinity mining module to mine a class affinity map (with the same size dimension as the class activation map). Similarly, a post-processing network generates another set of pseudo-labels. This module includes two key steps. The first is the initial seed selection stage, which involves choosing the top k% of highly activated regions on the global class activation map as initial seed points. The second is the class affinity map generation stage, where a class-agnostic affinity map is constructed from the global class activation map using a self-supervised visual transformer (DINO). While affinity maps effectively model the affinity between image patches, they do not reveal the class. This invention propagates the initial seeds across the class-agnostic affinity map to obtain a global class affinity map (with known class) for use in supervised weakly supervised semantic segmentation. This helps mitigate the inherent ambiguity of the class activation map and provides clear semantic segmentation auxiliary information. Specifically, for N image patches (after flattening), the category-agnostic affinity graph is an undirected graph of size N*N. For example, M(i,j) is the affinity between i and j. Then, with the initial seed i, the affinity between i and all other image patches is searched. If M(i,j) is greater than the threshold, it is considered to be in the same allocation category as the initial seed.

[0037] This invention further improves the accuracy of class affinity region maps through a prototype learning method. The prototype of each class is updated with momentum using the token of the high-activation region of the corresponding class. By comparing the similarity with image patch tokens, some erroneous affinity regions obtained due to the selection of incorrect seeds can be corrected. The specific expression is as follows:

[0038]

[0039] Among them, P c Let τ be the momentum prototype for the c-th class (one is set for each class, initially initialized randomly and uniformly), where τ is the momentum, with a default value of 0.7, and Q is the momentum prototype. c Let t be the set of highly activated regions, and t be the representation of the current highly activated region.

[0040] From a formulaic perspective, the momentum prototype is to establish a representation for each category, which is updated slowly (controlled by τ).

[0041]

[0042]

[0043] In the above formula, S c Let F represent the affinity region map, and M represent the representation of the current image patch. c This represents the class activation graph for the c-th category.

[0044] To select better initial seeds, this invention also introduces a class activation graph enhancement module for generating local class activation graphs and performing dense mapping between the local and global class activation graphs. This encourages local-to-global mapping of class activation graphs, thus addressing the failure of existing technologies to handle both global and local views of class activation graphs. Figure 1 To address the issue of consistency and improve the quality of the global activation map, this invention specifically crops and enhances the original image, inputs it into the backbone network to obtain a local activation map, and then aligns the local and global activation maps to the same spatial coordinates. The correspondence between the two is determined using pseudo-labels; activation values ​​of parts with the same pseudo-label are encouraged to be as similar as possible, while activation values ​​of different parts are encouraged to be dissimilar.

[0045]

[0046] Among them, R + R represents the part that is the same as the pseudo-tag. - L represents the different parts of the pseudo-tag. align The loss function can be considered as a constraint during the model training process.

[0047] The class activation graph alignment enhancement module encourages dense local-to-global class activation graph correspondences, thereby improving the quality of class activation graphs, enhancing the accuracy of pseudo-labels, and further improving semantic segmentation performance. By obtaining class activation graphs from local perspectives and constraining consistency with global perspectives (complete images), the class activation graphs can cover more complete regions, improving their robustness and providing better initial seeds for the class affinity mining module.

[0048] To verify the effectiveness of the method of this invention, this embodiment compares the method of this invention with existing methods on three datasets, with the metric being mIoU, i.e., the intersection-union ratio of the segmentation result and the ground-truth. The experimental results are shown in Table 1, which shows that the method of this invention achieves better segmentation results compared with existing methods.

[0049] Table 1 Comparative Experiment Results of Examples

[0050]

[0051] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] The above description of the embodiments is provided to enable those skilled in the art to understand and use the invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the invention should be within the protection scope of the present invention.

Claims

1. A single-stage weakly supervised semantic segmentation method based on affinity, characterized in that, Includes the following steps: The image to be segmented is obtained and input into a single-stage weakly supervised semantic segmentation network based on class affinity to obtain the semantic segmentation result of the image to be segmented. The single-stage weakly supervised semantic segmentation network based on class affinity includes a segmentation head and a class affinity mining module. The segmentation head is jointly trained using two sets of pseudo-labels to obtain segmentation results from the image to be segmented. One set of pseudo-labels is generated based on a global class activation map, which is generated based on a pre-trained classification head and the image to be segmented. The other set of pseudo-labels is generated based on a global class affinity region map, which is generated by the class affinity mining module based on the global class activation map. The class affinity mining module is constructed based on the self-supervised visual transformer DINO.

2. The single-stage weakly supervised semantic segmentation method based on class affinity according to claim 1, characterized in that, The specific process for generating pseudo-labels based on the global class activation graph is as follows: The image to be segmented is divided into multiple image patches and abstracted, which are then fed into a pre-trained classification head to generate a global class activation map. The global class activation map is then input into a post-processing network to generate pseudo-labels.

3. The single-stage weakly supervised semantic segmentation method based on affinity according to claim 2, characterized in that, The Mix Transformer is used to segment the image into multiple image patches and perform abstract representation.

4. The single-stage weakly supervised semantic segmentation method based on affinity according to claim 1, characterized in that, The process of generating pseudo-labels based on a global affinity region graph is as follows: A category-agnostic affinity graph is constructed based on the global class activation graph using the self-supervised visual transformer DINO. The highly activated regions of the global class activation graph are selected as initial seeds and propagated on the class-agnostic affinity graph to obtain a global class affinity region graph. The post-processing network generates pseudo-labels based on the global affinity region graph.

5. The single-stage weakly supervised semantic segmentation method based on affinity according to claim 4, characterized in that, The accuracy of the global affinity region graph is improved by using a prototype learning method.

6. The single-stage weakly supervised semantic segmentation method based on affinity according to claim 5, characterized in that, In prototype learning, the prototype of each category is updated with momentum through the representation of the high-activation region of the corresponding category, as shown in the following expression: Among them, P c Let Q be the momentum prototype of the c-th category, where τ is the momentum and Q is the momentum prototype. c Let t be the set of highly activated regions, and t be the representation of the current highly activated region.

7. The single-stage weakly supervised semantic segmentation method based on affinity according to claim 1, characterized in that, The single-stage weakly supervised semantic segmentation network based on class affinity also includes a class activation graph enhancement module, which generates local class activation graphs and performs dense mapping between the local class activation graphs and the global class activation graphs.

8. A single-stage weakly supervised semantic segmentation method based on class affinity according to claim 7, characterized in that, The specific steps corresponding to the density are as follows: The local class activation graph and the global class activation graph are aligned to the same spatial coordinates. The correspondence between the two is determined by the pseudo-labels generated by the global class activation graph, so that the parts with the same pseudo-labels have similar activation values.

9. An electronic device comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images