An image-level weakly supervised semantic segmentation method based on attention and local splicing

Abstract

This invention provides an image-level weakly supervised semantic segmentation method based on attention and local stitching, comprising: constructing a semantic segmentation network model, including a feature extraction backbone network, an image-level weakly supervised module, and a segmentation head; the image-level weakly supervised module includes a recurrent multi-level cross-attention module and a pseudo-label generation module; extracting deep image features and semantic information through the feature extraction backbone network to obtain feature maps; introducing long-distance dependencies between pixels in different levels of feature maps through the recurrent multi-level cross-attention module; and using pseudo-labels generated by the pseudo-label generation module as real labels; this invention designs a complete end-to-end image-level weakly supervised semantic segmentation network, which can solve the two major defects of weakly supervised semantic segmentation from pseudo-label generation to segmentation network training in only one step, greatly simplifying the network structure of weakly supervised semantic segmentation and avoiding the complexity and excessive size of the algorithm program.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an image-level weakly supervised semantic segmentation method based on attention and local stitching. Background Technology

[0002] Semantic segmentation is a crucial technical field in computer vision, primarily involving pixel-by-pixel image recognition based on the semantic meaning of input images. Its applications are extensive, including: 1. Autonomous driving systems: After images are detected by onboard cameras or LiDAR, they are input into a neural network. The computer can automatically segment and classify the images, avoiding obstacles such as pedestrians and vehicles, achieving environmental perception. 2. Smart healthcare: Primarily used in medical image analysis, such as tumor image segmentation and dental caries diagnosis, effectively reducing the workload of doctors and assisting in diagnosis. 3. Geographic Information Systems (GIS): By training neural networks to recognize satellite remote sensing images, it automatically identifies roads, rivers, crops, buildings, etc., and classifies and labels each pixel in the image.

[0003] Fully Convolutional Networks (FCNs) are a framework for image semantic segmentation proposed in 2015. FCNs primarily replace the fully connected layers of traditional CNNs with convolutional blocks. The network output is a heatmap rather than a recognized category, and there is no limitation on the number of neurons in the input layer of fully connected layers, so the input to the convolutional blocks can accept images of different sizes. Furthermore, to address the image size reduction caused by convolution and pooling, upsampling is used to restore the image size.

[0004] The UNET network, published in the same year, was an improvement on FCN. It's a typical encoder-decoder structure consisting of three parts: encoder upsampling, decoder downsampling, and skip-layer connections. The continuous downsampling in the UNET network led to a significant decrease in image resolution, resulting in the loss of a large amount of detail that was difficult to recover even through upsampling. Based on this, the Deeplab series introduced dilated convolutional blocks to replace downsampling layers for the first time. This expanded the receptive field and captured more contextual information without changing the image dimensions or increasing parameters. Simultaneously, the ASPP module was introduced to achieve multi-scale feature map fusion, greatly improving the semantic segmentation effect.

[0005] Building upon the above research, the latest research proposes a combination of the transformer algorithm and semantic segmentation, which utilizes an attention mechanism to better capture global contextual information and improve algorithm accuracy.

[0006] Despite significant progress in semantic segmentation in recent years, mainstream semantic segmentation algorithms still require large amounts of labeled data. Manually labeling this data pixel by pixel is time-consuming, labor-intensive, and extremely expensive. To alleviate this burden, researchers have increasingly focused on building label-efficient deep learning algorithms in recent years, with weakly supervised semantic segmentation being one such example. Weakly supervised semantic segmentation no longer uses traditionally fully manually labeled data for training; instead, it uses coarser but simpler labeled data. Based on the form of the weak supervision signal, common weakly supervised semantic segmentation algorithms can be divided into the following four categories:

[0007] Image-level annotation: only annotates the category to which relevant objects in an image belong; it is the simplest type of annotation.

[0008] Object point annotation: Annotate a specific point on each object and its corresponding category;

[0009] Object bounding box annotation: Annotate the bounding box containing each object, as well as its corresponding category;

[0010] Object graffiti annotation: Draw a line on each object and add its corresponding category.

[0011] Among the most researched methods, image-level labeled weakly supervised semantic segmentation remains the most prevalent. However, the challenge of image-level weakly supervised semantic segmentation lies in the significant gap between image-level and pixel-level supervision. To bridge this gap in supervision information, the following two steps are typically followed: First, using image-level labeled image category tags, a classification model is trained through single-label or multi-label classification. This model generates seed regions by calculating the class feature response map (hereinafter referred to as class activation map CAM) of the corresponding category in the image, and generates pseudo-labels by propagating semantic information from the seed regions to the entire image. Then, the pseudo-labels are used as "real labels" to train a traditional segmentation network, thereby achieving pixel-level, dense image prediction.

[0012] Image-level weakly supervised semantic segmentation has the following two shortcomings:

[0013] One drawback is that to implement a two-step process for image-level weakly supervised semantic segmentation, multiple branch neural networks typically need to be designed separately, such as pseudo-label generation networks, class activation map (CAM) optimization networks, and semantic segmentation networks. While the two-step method can effectively improve segmentation performance, it also leads to problems such as a large and complex algorithm and a significant increase in computational cost.

[0014] Second, Class Activation Maps (CAMs) essentially utilize the prior knowledge of cross-label constraints to locate seed regions in an image based on information provided by the classification model. However, CAMs have two limitations: (1) Incompleteness: The seed regions generated by CAMs are usually coarse, ignoring many edge details of objects and often failing to cover the complete semantic region of the target class; (2) Redundancy: A CAM may overlap with regions of other classes, leading to misclassification. These two points result in inaccurate generated pseudo-labels, thus affecting subsequent segmentation performance. Summary of the Invention

[0015] To address the aforementioned shortcomings of existing technologies, this invention provides an image-level weakly supervised semantic segmentation method based on attention and local stitching to solve the aforementioned technical problems.

[0016] This invention provides an image-level weakly supervised semantic segmentation method based on attention and local stitching, comprising:

[0017] Obtain the training sample set and preprocess the labeled images in the training sample set;

[0018] A semantic segmentation network model is constructed, including a feature extraction backbone network, an image-level weak supervision module, and a segmentation head. The image-level weak supervision module includes a recurrent multi-level cross-attention module and a pseudo-label generation module. The feature extraction backbone network extracts deep image features and semantic information to obtain feature maps. The recurrent multi-level cross-attention module introduces long-distance dependencies between pixels in different levels of feature maps. Pseudo-labels generated by the pseudo-label generation module serve as real labels. One branch of the pseudo-label generation module non-overlaps the feature maps into four small blocks, generates four small block class activation maps, and stitches them together. The other branch of the pseudo-label generation module non-overlaps the original input image into four blocks, uses positional weights instead of global weights to generate four small block class activation maps, and stitches them together. The class activation maps from the two branches are then concatenated to obtain the final class activation map (CAM). The class activation map (CAM) and pseudo-labels are input into the segmentation head to obtain the segmentation result image.

[0019] The semantic segmentation network model is trained by selecting a loss function, calculating the loss between the trained segmentation image and the labeled image, backpropagating it to the semantic segmentation network model to update the network weights, gradually reaching the optimal mode of the model, and then saving the semantic segmentation model.

[0020] Furthermore, the feature extraction backbone network is a residual network ResNet101, which takes the image x as input and obtains the feature map f(x) after passing through the feature extraction backbone network.

[0021] Furthermore, the cyclic multi-level cross-attention module is used for:

[0022] The feature map f(x) output by the feature extraction backbone network is used as the input feature of the recurrent multi-level cross-attention module;

[0023] The feature map f(x) is first passed through a 3D convolutional block, and then through a Batch Normalization (BN) layer to generate a shallow feature map f rich in local information. 11 (x);

[0024] shallow feature map f 11 (x) is divided into two parts, and each part is processed by a one-dimensional convolutional block to obtain feature maps K and Q;

[0025] shallow feature map f 11 (x) is then passed through a one-dimensional convolutional block and a BN layer to generate a deep feature map f rich in semantic information. 12 (x);

[0026] deep features f 12 (x) is processed by a one-dimensional convolutional block to obtain the feature map V;

[0027] In this context, feature maps K, Q, and V are all four-dimensional matrices B×C×H×W, where B represents the number of data points passed to the program for training in a single pass, C represents the number of channels in the feature map, H represents the length of the feature map, and W represents the width of the feature map.

[0028] Using feature maps K and Q as input, perform an Affinity operation. For each position in feature map K, sum the pixel values ​​of all channels with the corresponding pixel values ​​of all channels in Q at that position, and then perform Softmax normalization to obtain the shallow feature map f. 11 (x) The weight matrix of the relationships between each pixel;

[0029] The weight matrix is ​​then aggregated with the feature map V. Specifically, the pixel values ​​of all channels at each position in the weight matrix are weighted and summed with the corresponding pixel values ​​at each layer of the feature map V, along with the pixel values ​​in their respective rows and columns, to obtain the aggregated matrix.

[0030] Combine the aggregation matrix and deep feature map f 12 (x) are added together to obtain the feature map f. 21 (x);

[0031] Feature map f 21 (x) generates deep features f after passing through a one-dimensional convolutional block and a BN layer. 22 (x);

[0032] With feature map f 21 (x), f21 (x) is used as the input feature and then input again into the recurrent multi-level cross-attention module to generate the final feature map f. 31 (x).

[0033] Furthermore, the pseudo-tag generation module adopts a two-branch structure;

[0034] First, the original input image x is divided into four non-overlapping smaller images x'. Then, x and x' are processed through a feature extraction backbone network, and feature maps f(x) and f(x') are obtained in the recurrent multi-level cross-attention module, respectively. 11 , f(x') 12 , f(x') 21 , f(x') 22 ;

[0035] One of the branches is used to: divide each layer of the feature map f(x) into four non-overlapping feature regions, perform global average pooling and expand them in the four feature regions respectively, generate the weight matrix W corresponding to the class through the fully connected layer, multiply the four feature regions with the corresponding weight matrix W and sum them to obtain the spliced ​​class activation map CAM0.

[0036] Another branch is used to: combine four small feature maps f(x') 11 , f(x') 12 , f(x') 21 , f(x') 22 Global average pooling is performed on the entire region, followed by a fully connected layer to generate a weight matrix W corresponding to each class. This W is then multiplied by the corresponding small feature map of each layer and summed to obtain four class activation maps.

[0037] CAM 11 CAM 12 CAM 21 CAM 22 And seamlessly stitch them together, classifying the activation graph CAM1;

[0038] Finally, the class activation graphs CAM0 and CAM1 obtained from the two branches are added together to obtain the final class activation graph CAM. The corresponding pseudo-labels are then generated from the class activation graph CAM.

[0039] Furthermore, the generated pseudo-labels are fed into the segmentation head as supervision information for model training. During the training process, the weights of the network are updated through gradient descent and backpropagation. A loss function and the optimal evaluation metric are selected. The loss is calculated between the trained images and the labeled images, and backpropagated to the network to update the network weights, gradually reaching the optimal model mode.

[0040] Furthermore, in the training of the semantic segmentation network model, the loss function selected for calculating the loss of the segmentation image and the label image is the cross-entropy loss function. This cross-entropy loss function is based on the pseudo-labels provided by the pseudo-label generation module, and the formula is as follows:

[0041]

[0042] Where L is the loss between the segmentation result image and the label image, N is the number of samples, c is the true sample class, C is the number of sample classes, and y ic This is the sample label value. If the label of pixel k is c, then it equals 1; otherwise, it is 0. ic These are the output probabilities of class c, which are obtained from a trained segmentation network and normalized using Softmax.

[0043] Furthermore, the loss functions L1 and L2 of the two branches in the pseudo-label generation module are Cross Entropy Loss functions, with the following formula:

[0044]

[0045] Where c represents the number of sample categories, C is the number of sample categories, and y c These are sample label values, belonging to class c (y). c It equals 1 if it is equal to 1, otherwise it is 0. The output probability of belonging to class C is predicted by the pseudo-label generation module.

[0046] Furthermore, the pseudo-label generation module also introduces a 1-norm between the class activation maps CAM0 and CAM1, and generates a loss function L3 for the activation map CAM. This function minimizes the distance between the two class activation maps before and after the stitching process, improving the consistency of network predictions. The formula is as follows:

[0047] L3 = ||CAM0-CAM1||1

[0048] Furthermore, it also includes: storing the semantic segmentation network model after weight training, and the feature map f obtained after passing through the recurrent multi-level cross-attention module. 31 (x) is used as the input to the segmentation head. It first passes through a 3D convolutional block and a ReLU layer, and then through another 3D convolutional block to finally obtain the segmentation map.

[0049] The beneficial effects of this invention are as follows:

[0050] (1) This invention designs a complete end-to-end image-level weakly supervised semantic segmentation network. Through the pseudo-label generation module and the cyclic multi-level cross-attention module, the two modules work together to solve the two major defects of weakly supervised semantic segmentation, namely, generating pseudo-labels and training the segmentation network, in just one step. This greatly simplifies the network structure of weakly supervised semantic segmentation, avoids the complexity and excessive size of the algorithm program, and significantly improves the computational cost of the semantic segmentation network compared with the traditional two-step image-level weakly supervised semantic segmentation method.

[0051] (2) This invention employs a two-branch structure in the pseudo-label generation module. One branch cuts the input image into four non-overlapping blocks, generates four small class activation maps (CAMs) through a classification network, and then stitches them together. The stitched large class activation map CAM focuses on a larger object region than the original class activation map CAM. The other branch uses the original-sized input image as input. Instead of using the complete feature map for full average pooling to generate global weights, the feature map is cut into four non-overlapping blocks, and global average pooling is performed on each block. Each layer of the feature map thus obtains four positional weights. Replacing the global weights with positional weights makes the class activation map CAM focus more on the local features of the object, especially the extraction of shallow positional information, which is more detailed, thereby improving the quality of the class activation map CAM.

[0052] (3) This invention introduces a cyclic multi-level cross-attention module. For each pixel in the image, a cross-attention module acquires the contextual information of all pixels on its cross path. Through further cyclic operations, each pixel can eventually capture complete long-distance image correlations. At the same time, the attention module fuses shallow and deep features to explore long-distance pixel correlations between feature maps of different levels. This solves the problems of large computational load and inability to take into account long-distance relationships between feature maps of different levels in traditional non-local attention modules, greatly improving the semantic segmentation effect. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is the backbone network structure of a semantic segmentation network model constructed according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of a cyclic multi-level cross-attention module according to an embodiment of the present invention;

[0056] Figure 3This is a schematic diagram of the structure of a pseudo-label generation module according to an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of a segmentation head according to an embodiment of the present invention. Detailed Implementation

[0058] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0059] Addressing the first deficiency of the existing technical solution:

[0060] This invention designs a complete end-to-end weakly supervised semantic segmentation network. Specifically, it comprises two modules: a pseudo-label generation module and a recurrent multi-level cross-attention module. These two modules work together to solve the two major problems of weakly supervised semantic segmentation—from generating pseudo-labels to training the segmentation network—in a single step, greatly simplifying the network structure of weakly supervised semantic segmentation.

[0061] Regarding the second defect of the existing technical solution:

[0062] To generate more detailed class activation maps (CAMs), the pseudo-label generation module employs a two-branch structure. Research found that stitching together CAMs generated from non-overlapping image segments results in a larger object region being identified by the CAMs generated from the original segments. Therefore, one branch follows this approach, segmenting the input image into four non-overlapping blocks, generating four smaller CAMs via a classification network, and then stitching them together. The resulting larger CAM focuses on a larger and more complete object region.

[0063] Another branch uses the original-sized input image as input. Since global weights cannot fully represent the importance of different locations in the feature map for the target class, instead of using the complete feature map for full average pooling (GAP) to generate global weights, the feature map is divided into four non-overlapping blocks, and global average pooling is performed on each block. Each feature map layer thus obtains four positional weights. Replacing global weights with positional weights makes the Class Activation Map (CAM) focus more on the local features of the object, improving the quality of the CAM.

[0064] Furthermore, to improve semantic segmentation performance, this invention introduces a cyclic multi-level cross-attention module. For each pixel, a cross-attention module acquires the contextual information of all pixels along its cross path. Through further cyclic operations, each pixel can ultimately capture complete long-distance image correlations. Simultaneously, the attention module fuses shallow and deep features to explore long-distance image correlations between features at different levels. This significantly improves semantic segmentation performance while addressing the issues of high computational cost of the attention module and long-distance relationships between features at different levels.

[0065] The following section provides a detailed description of the image-level weakly supervised semantic segmentation method based on attention and local stitching provided in the embodiments of the present invention.

[0066] The method includes: acquiring a training sample set and a test sample set, and preprocessing the labeled images of the training samples;

[0067] A semantic segmentation network model is constructed, including a feature extraction backbone network, an image-level weak supervision module, and a segmentation head. The image-level weak supervision module includes a recurrent multi-level cross-attention module and a pseudo-label generation module. The feature extraction backbone network extracts deep features and semantic information from the image. The recurrent multi-level cross-attention module introduces long-distance dependencies between pixels in feature maps at different levels. The pseudo-labels generated by the pseudo-label generation module are used as real labels. The CAM-like activation map and the pseudo-labels are input into the segmentation head to obtain the segmentation result map.

[0068] The semantic segmentation network model is trained by selecting a loss function, calculating the loss between the trained image and the labeled image, and backpropagating it to update the network weights, gradually reaching the optimal model mode and preserving the semantic segmentation model structure.

[0069] In this embodiment, the backbone network structure built in step two is as follows: Figure 1 As shown, it is mainly divided into three parts: the first part is a feature extraction backbone network used to extract deep features and semantic information of the image; the second part is an image-level weak supervision module that introduces long-distance dependencies between pixels in feature maps at different levels and generates CAM activation maps and pseudo-labels; the second part is further divided into two branches: a cyclic multi-level cross-attention module and a pseudo-label generation module; the third part is a segmentation head, which uses the feature maps and pseudo-labels output by the image-level weak supervision module as input and supervision information, respectively, and finally obtains the segmentation result map.

[0070] Optionally, as an embodiment of the present invention, the datasets used are PASCOL VOC 2012 and COCO, respectively. The PASCOL VOC 2012 and COCO datasets are divided into training sample sets and test sample sets, respectively. After all data is preprocessed, the cropped image with a size of 448×448 (H×W) is used as the input of the entire semantic segmentation network, and the images labeled with categories in the training sample set and the test sample set are used as label images.

[0071] Optionally, as an embodiment of the present invention, the feature extraction backbone network is a residual network ResNet101, which takes the image x (H×W is 448×448) as input and obtains the feature map f(x) after passing through the feature extraction backbone network.

[0072] Specifically, the ResNet101 residual network can extract deeper features from images while addressing issues such as degraded neural network performance, gradient vanishing, or gradient explosion when the number of network layers is too deep, thus improving the accuracy and efficiency of feature extraction.

[0073] Optionally, as an embodiment of the present invention, the second part of the pseudo-label generation module is further divided into two branches: a cyclic multi-level cross-attention module and a pseudo-label generation module. The cyclic multi-level cross-attention module is as follows: Figure 2 As shown, this is used to introduce long-distance dependencies between pixels in feature maps at different levels.

[0074] The feature map f(x) output by the feature extraction backbone network is used as the input feature of the recurrent multi-level cross-attention module;

[0075] The feature map f(x) is first passed through a 3D convolutional block, and then through a Batch Normalization (BN) layer to generate a shallow feature map f rich in local information. 11 (x);

[0076] shallow feature map f 11 (x) is divided into two parts, and each part is processed by a one-dimensional convolutional block to obtain feature maps K and Q;

[0077] shallow feature map f 11 (x) is then passed through a one-dimensional convolutional block and a BN layer to generate a deep feature map f rich in semantic information. 12 (x);

[0078] deep features f 12 (x) is processed by a one-dimensional convolutional block to obtain the feature map V;

[0079] In this embodiment, the three-dimensional convolutional block has a kernel size of 3×3, a stride of 1, and a padding number p of 1; the one-dimensional convolutional block has a kernel size of 1×1, a stride of 1, and a padding number p of 0.

[0080] In this embodiment, feature maps K, Q, and V are all four-dimensional matrices B×C×H×W, where B represents the number of data points passed to the program for training in a single pass, C represents the number of channels in the feature map, H represents the length of the feature map, and W represents the width of the feature map. In this embodiment, B is set to 4, the number of channels C for feature maps K and Q is 128, and the number of channels C for feature map V is 1024. The length and width H×W of the three feature maps are all 56×56.

[0081] Using feature maps K and Q as input, perform an Affinity operation. For each position in feature map K, sum the pixel values ​​of all channels with the corresponding pixel values ​​of all channels in Q at that position, and then perform Softmax normalization to obtain the shallow feature map f. 11 (x) The weight matrix of the relationship between each pixel; in this embodiment, the size of the obtained weight matrix is ​​56×56×112.

[0082] The weight matrix is ​​aggregated with the feature map V, that is, the pixel values ​​of all channels at each position of the weight matrix are weighted and summed with the pixel values ​​of the corresponding positions, rows and columns of each layer of the feature map V. In this embodiment, after aggregation, an aggregated matrix with 1024 channels C and a length and width H×W of 56×56 is generated.

[0083] Combine the aggregation matrix and deep feature map f 12 (x) are added together to obtain the feature map f. 21 (x);

[0084] f 21 (x) generates deep features f after passing through a one-dimensional convolutional block and a BN layer. 22 (x); in this embodiment f 21 (x) passes through a one-dimensional convolutional block with kernel k of size 1×1, stride s of 1, and padding p of 0;

[0085] f 21 (x), f 21 (x) is used as the input feature and then input again into the recurrent multi-level cross-attention module to generate the final feature map f. 31 (x).

[0086] like Figure 3As shown, this invention employs a two-branch structure in the pseudo-label generation module. One branch cuts the input feature map into four non-overlapping blocks, generates four small class activation maps (CAMs) through a classification network, and then stitches them together. The resulting large class activation map CAM focuses on a larger object region than the original class activation map CAM. The other branch takes the original image as input and, instead of using the complete feature map for full average pooling (GAP) to generate global weights, cuts the feature map into four non-overlapping blocks and performs global average pooling on each feature region. Each feature map layer thus obtains four positional weights. These positional weights replace the global weights to generate class activation maps CAMs for each region. This makes the class activation map CAMs focus more on the local features of the object, especially the extraction of shallow positional information, which is more detailed, thereby improving the quality of the class activation map CAMs.

[0087] Specifically, one branch is used to: input the image x into the feature extraction backbone network to obtain a feature map f(x); divide each layer of the feature map f(x) into four non-overlapping feature regions, and perform global average pooling (GAP) and expand them in the four feature regions respectively, generate a weight matrix W corresponding to the class through a fully connected layer, and multiply the four feature regions with the corresponding weight matrix W and sum them to obtain the class activation map CAM0.

[0088] Specifically, another branch: First, the original input image x is divided into four non-overlapping small images x'; in this embodiment, the size of the four cropped images H×W is 112×112; then, after the four small images x' pass through the feature extraction backbone network, feature maps f(x) and f(x') are obtained respectively in the recurrent multi-level cross-attention module. 11 , f(x') 12 , f(x') 21 , f(x') 22 ; Four small feature maps f(x') 11 , f(x') 12 , f(x') 21 , f(x') 22 Global average pooling is performed on the entire region, followed by a fully connected layer to generate a weight matrix W corresponding to each class. This W is then multiplied by the corresponding small feature map of each layer and summed to obtain four class activation maps CAM. 11 CAM 12 CAM 21 CAM 22 In this embodiment, four class activation graphs

[0089] CAM 11 CAM 12 CAM 21 CAM 22The length and width H×W are both 112×112; and they are seamlessly spliced ​​together to form an activation map CAM1; in this embodiment, the length and width H×W of the activation map CAM1 are 448×448.

[0090] Merging the two branches: Finally, the class activation graph CAM0 and class activation graph CAM1 are added together to obtain the final class activation graph CAM. In this embodiment, the length and width H×W of the class activation graph CAM are 448×448.

[0091] Finally, the corresponding pseudo-labels are generated through the Class Activation Map (CAM). One common method for generating pseudo-labels is thresholding, which generally consists of the following steps:

[0092] Set a threshold: Select an appropriate threshold to filter regions with high activation values ​​in the Class Activation Map (CAM). The threshold can be set manually or selected based on statistical analysis or cross-validation methods.

[0093] Region segmentation: The Class Activation Map (CAM) is segmented into regions based on a set threshold. Pixels above the threshold are retained, while pixels below the threshold are marked as 0. A normalization operation is then performed, resulting in a binary image where regions with high activation values ​​are highlighted. The specific formula for region segmentation is as follows:

[0094]

[0095] Among them, M i,j It is a class activation map CAM, where i and j represent the pixel index positions of the class activation map CAM, and α is the set threshold.

[0096] Morphological operations, such as dilation and erosion, can be used to further expand or filter candidate regions; dilation can expand regions with high activation values, while erosion can remove isolated small regions.

[0097] Final candidate regions: Based on the results of thresholding and morphological operations, the final candidate regions are obtained, which are the regions related to the pseudo-labels.

[0098] Optionally, as an embodiment of the present invention, the generated pseudo-labels are fed into the segmentation head as supervision information for model training. During training, the network weights are updated using gradient descent and backpropagation. A loss function and optimal evaluation metric are selected. The loss is calculated between the trained network image and the labeled image, and backpropagated to update the network weights, gradually reaching the optimal model mode. The present invention sets the number of training rounds to 200.

[0099] Optionally, as an embodiment of the present invention, after training the model through the network, the output segmentation map and the labeled images in the training sample set are used to calculate the loss function. The loss function is selected as the cross-entropy loss function, which is based on the pseudo-labels provided by the pseudo-label generation module, and the formula is:

[0100]

[0101] Where L is the loss, N is the number of samples, c is the ground truth, C is the number of sample classes, and y ic This is the sample label value. If the label of pixel k is c, then it equals 1; otherwise, it is 0. ic These are the output probabilities of class c, which are obtained from a trained segmentation network and normalized using Softmax.

[0102] Optionally, as an embodiment of the present invention, the loss functions L1 and L2 of the two branches in the pseudo-label generation module are Cross Entropy Loss functions, with the following formula:

[0103]

[0104] Where c represents the number of sample categories, C is the number of sample categories, and y c These are sample label values, belonging to class c (y). c It equals 1 if it is equal to 1, otherwise it is 0. The output probability of belonging to class C is predicted by the pseudo-label generation module.

[0105] Optionally, as an embodiment of the present invention, the pseudo-label generation module further includes a L1-norm introduced between the class activation graph CAM0 and the class activation graph CAM1, and a loss function L3 for generating the activation graph CAM, used to minimize the distance between the two class activation graphs before and after splicing, thereby improving the consistency of network predictions. The formula is as follows:

[0106] L3 = ||CAM0-CAM1||1

[0107] Optionally, as one embodiment of the present invention, such as Figure 4 As shown, the feature map f obtained after the cyclic multi-level cross-attention module 31 (x) is used as the input to the segmentation head. It first passes through a convolutional block with a kernel size of 3×3, stride s of 1, padding p of 12, and dilation d of 12, and a ReLU layer. Then it passes through another convolutional block with a kernel size of 3×3, stride s of 1, padding p of 12, and dilation d of 12 to finally obtain the segmentation image.

[0108] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention. Such modifications or substitutions should all fall within the scope of the invention, or any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. An image-level weakly supervised semantic segmentation method based on attention and local stitching, characterized in that, include: Obtain the training sample set and preprocess the labeled images in the training sample set; A semantic segmentation network model is constructed, including a feature extraction backbone network, an image-level weak supervision module, and a segmentation head. The image-level weak supervision module includes a recurrent multi-level cross-attention module and a pseudo-label generation module. The feature extraction backbone network extracts deep image features and semantic information to obtain feature maps. The recurrent multi-level cross-attention module introduces long-distance dependencies between pixels in different levels of feature maps. Pseudo-labels generated by the pseudo-label generation module serve as real labels. One branch of the pseudo-label generation module non-overlaps the feature maps into four small blocks, generates four small block class activation maps, and stitches them together. The other branch of the pseudo-label generation module non-overlaps the original input image into four blocks, uses positional weights instead of global weights to generate four small block class activation maps, and stitches them together. The class activation maps from the two branches are then concatenated to obtain the final class activation map (CAM). The class activation map (CAM) and pseudo-labels are input into the segmentation head to obtain the segmentation result image. The semantic segmentation network model is trained by selecting a loss function, calculating the loss between the trained segmentation image and the labeled image, backpropagating it to the semantic segmentation network model to update the network weights, gradually reaching the optimal mode of the model, and saving the semantic segmentation model. The cyclic multi-level cross-attention module is used for: The feature map f(x) output by the feature extraction backbone network is used as the input feature of the recurrent multi-level cross-attention module; The feature map f(x) is first passed through a 3D convolutional block, and then through a Batch Normalization (BN) layer to generate a shallow feature map f rich in local information. 11 (x); shallow feature map f 11 (x) is divided into two parts, and each part is processed by a one-dimensional convolutional block to obtain feature maps K and Q; shallow feature map f 11 (x) is then passed through a one-dimensional convolutional block and a BN layer to generate a deep feature map f rich in semantic information. 12 (x); deep features f 12 (x) is processed by a one-dimensional convolutional block to obtain the feature map V; In this context, feature maps K, Q, and V are all four-dimensional matrices B×C×H×W, where B represents the number of data points passed to the program for training in a single pass, C represents the number of channels in the feature map, H represents the length of the feature map, and W represents the width of the feature map. Using feature maps K and Q as input, perform an Affinity operation. For each position in feature map K, sum the pixel values ​​of all channels with the corresponding pixel values ​​of all channels in Q at that position, and then perform Softmax normalization to obtain the shallow feature map f. 11 (x) The weight matrix of the relationships between each pixel; The weight matrix is ​​aggregated with the feature map V, that is, the pixel values ​​of all channels at each position of the weight matrix are weighted and summed with the pixel values ​​of the corresponding positions, rows and columns of each layer of the feature map V to obtain the aggregate matrix. Combine the aggregation matrix and deep feature map f 12 (x) are added together to obtain the feature map f. 21 (x); Feature map f 21 (x) generates deep features f after passing through a one-dimensional convolutional block and a BN layer. 22 (x); With feature map f 21 (x), f 21 (x) is used as the input feature and then input again into the recurrent multi-level cross-attention module to generate the final feature map f. 31 (x).

2. The method according to claim 1, characterized in that, The feature extraction backbone network is a residual network ResNet101. The image x is used as input, and the feature map f(x) is obtained after passing through the feature extraction backbone network.

3. The method according to claim 2, characterized in that, The pseudo-tag generation module adopts a two-branch structure. One branch is used to: extract the features of the original input image x into the backbone network to obtain the feature map f(x), divide each layer of the feature map f(x) into four non-overlapping feature regions, perform global average pooling and expand the feature regions, generate the weight matrix W corresponding to the class through the fully connected layer, multiply the four feature regions with the corresponding weight matrix W and sum them to obtain the spliced ​​class activation map CAM0; Another branch is used to: divide the original input image x into four non-overlapping small images x', pass x' through the feature extraction backbone network, and then obtain feature maps f(x') in the recurrent multi-level cross-attention module. 11 , f(x') 12 , f(x') 21 , f(x') 22 ; Four small feature maps f(x') 11 , f(x') 12 , f(x') 21 , f(x') 22 Global average pooling is performed on the entire region, and a weight matrix W corresponding to the class is generated through a fully connected layer. This weight matrix W is then multiplied by the corresponding small feature map of each layer and summed to obtain four class activation maps CAM. 11 CAM 12 CAM 21 CAM 22 And seamlessly stitch them together, classifying the activation graph CAM1; Finally, the class activation graphs CAM0 and CAM1 obtained from the two branches are added together to obtain the final class activation graph CAM. The corresponding pseudo-labels are then generated from the class activation graph CAM.

4. The method according to claim 1, characterized in that, Also includes: The generated pseudo-labels are fed into the segmentation head as supervision information for model training. During the training process, the weights of the network are updated through gradient descent and backpropagation. A loss function and the optimal evaluation metric are selected. The loss is calculated between the trained image and the labeled image, and backpropagated to the network to update the network weights, gradually reaching the optimal model mode.

5. The method according to claim 1, characterized in that, In the training of the semantic segmentation network model, the loss function selected for calculating the loss of the segmentation image and the label image is the cross-entropy loss function. This cross-entropy loss function is based on the pseudo-labels provided by the pseudo-label generation module, and the formula is as follows: Where L is the loss between the segmentation result image and the label image, N is the number of samples, c is the true class of the sample, C is the number of sample classes, and y ic This is the sample label value. If the label of pixel k is c, then it equals 1; otherwise, it is 0. ic These are the output probabilities of class c, which are obtained from a trained segmentation network and normalized using Softmax.

6. The method according to claim 1, characterized in that, The loss functions L1 and L2 in the pseudo-label generation module are Cross Entropy Loss functions, and the formula is as follows: Where c represents the number of sample categories, C is the number of sample categories, and y c These are sample label values, belonging to class c (y). c It equals 1 if it is equal to 1, otherwise it is 0. The output probability of belonging to class C is predicted by the pseudo-label generation module.

7. The method according to claim 1, characterized in that, The pseudo-label generation module also includes a 1-norm introduced between class activation graphs CAM0 and CAM1, and a loss function L3 for generating activation graph CAM, used to minimize the distance between the two class activation graphs before and after splicing, thereby improving the consistency of network predictions. The formula is as follows: L3 = ||CAM0-CAM1||1.

8. The method according to claim 1, characterized in that Also includes: Save the semantic segmentation network model trained with weights, and obtain the feature map f after passing through the recurrent multi-level cross-attention module. 31 (x) is used as the input to the segmentation head. It first passes through a 3D convolutional block and a ReLU layer, and then through another 3D convolutional block to finally obtain the segmentation map.

Images