A transformer weakly supervised semantic segmentation method combining context attention

Abstract

The application discloses a kind of weakly supervised semantic segmentation methods of combining context attention Transformer, and the application belongs to the field of image processing segmentation, including the following steps: step one, select the semantic segmentation image dataset with weak label;Step two, construct weakly supervised semantic segmentation network model of combining context attention Transformer, the network model constructed includes Transformer encoder, decoder, pseudo-label refinement module and context attention module;Step three, set the loss function suitable for the network model;Step four, use image dataset to train network model, obtain the weakly supervised semantic segmentation network model of final training;Step five, the image to be segmented is input into the network model of training model parameter, and the segmentation result graph of the sky image is output.

Description

Technical Field

[0001] This invention belongs to the field of image processing and segmentation technology, specifically a Transformer weakly supervised semantic segmentation method that combines contextual attention. Background Technology

[0002] Semantic segmentation is a fundamental task in computer vision, requiring the classification of visual input into different semantically interpretable categories. Unlike detection and classification tasks, semantic segmentation needs to predict and classify all pixels in an image, making it one of the key problems in the field of computer vision today. Most existing semantic segmentation algorithms are based on fully supervised learning; however, due to the difficulty in obtaining a large number of pixel-level labels from existing image datasets, weakly supervised semantic segmentation algorithms have been proposed. Weakly supervised semantic segmentation aims to reduce the reliance on pixel-level ground truth labels by using weak supervision. A key step in this task is generating high-quality pseudo-labels using existing weak labels.

[0003] Weakly supervised semantic segmentation, compared to pixel-level labeling, employs a coarser labeling approach, using less cost and weaker labeled information as samples to build a more ideal segmentation network model. Previous weakly supervised semantic segmentation methods typically relied on Class Activation Maps (CAMs) to extract object localization maps from Convolutional Neural Networks (CNNs). Despite using complex CAM expansion strategies and multiple training steps, some existing methods still exhibit limited performance and less precise completeness and accuracy. Vision Transformer (ViT), as the first Transformer model specifically designed for computer vision, has achieved state-of-the-art performance in large-scale image recognition due to its powerful contextual interaction capabilities. ViT segments the input image into non-overlapping patches and transforms them into a series of vectors, also using an additional class token to aggregate information from the entire sequence of patch tokens. Weakly supervised semantic segmentation methods can be broadly categorized into two types: multi-stage methods and end-to-end methods. End-to-end methods are further divided into weakly supervised semantic segmentation based on Convolutional Neural Networks and weakly supervised semantic segmentation based on self-attention.

[0004] Most weakly supervised semantic segmentation methods using image-level labels are performed through a multi-stage process. Typically, these methods use classification networks to generate initial pixel-level pseudo-labels, generally employing CAM methods. Due to the limited amount of supervisory information, training a high-performance end-to-end model for weakly supervised semantic segmentation is challenging. EM proposes an adaptive expectation-maximization framework to infer segmentation pseudo-labels. MIL treats image-level label-based weakly supervised semantic segmentation as a multi-instance learning problem and designs a logarithmic-summary-exponential aggregation function to drive the network to assign correct pixel-level labels. 1Stage achieves performance comparable to multi-stage models by introducing nGWP pooling, pixel-adaptive mask refinement, and random low-level information passing. In RRM, RRM uses CAM as the initial pseudo-labels and employs CRF to generate refined labels as supervision for segmentation. RRM also introduces an auxiliary regularization loss to ensure consistency between the segmentation map and the low-level image appearance. The above methods typically use CNNs, which suffer from the inherent limitation of convolutions failing to capture global information, leading to incomplete activation of the target region. The advantage of Transformers is their ability to capture global information, leading to the development of weakly supervised semantic segmentation methods based on self-attention. By introducing Transformers, the limitation of CNNs in focusing only on local regions can be overcome, capturing more comprehensive features and refining pseudo-labels accordingly, resulting in more accurate performance.

[0005] In weakly supervised semantic segmentation, pseudo-label generation is a crucial step. Pseudo-labels assign a predicted semantic category to each pixel using image-level labels. By generating pseudo-labels, the framework for weakly supervised semantic segmentation can be trained even in the absence of pixel-level labels and achieve segmentation results as accurately as possible. However, the performance of weakly supervised semantic segmentation can be limited by the inaccuracy and class inconsistency of pseudo-label generation. Therefore, improving the pseudo-label generation method is key to improving the performance of weakly supervised semantic segmentation. By improving the pseudo-label generation method, the performance of weakly supervised semantic segmentation can be improved, thereby more accurately assigning each pixel to the correct semantic category. Summary of the Invention

[0006] Purpose of the invention: To address the problems of low accuracy in generating pseudo-labels and limited mIOU performance in traditional weakly supervised semantic segmentation, this invention discloses an end-to-end Transformer weakly supervised semantic segmentation method incorporating contextual attention. Using the Transformer network architecture and enhancement algorithms such as contextual attention, this method addresses the coarse pseudo-label generation characteristic in weakly supervised semantic segmentation by combining attention and diffusion propagation of labels to obtain a network model that effectively generates more accurate pseudo-labels, thus enhancing the adaptability and robustness of this method in weakly supervised semantic segmentation.

[0007] Technical solution: A Transformer weakly supervised semantic segmentation method combining contextual attention, comprising the following steps:

[0008] Step 1: Select a semantic segmentation image dataset with weak labels;

[0009] Step 2: Construct a Transformer weakly supervised semantic segmentation network model that incorporates contextual attention. The constructed network model includes a Transformer encoder, decoder, pseudo-label refinement module, and contextual attention module.

[0010] Step 3: Set the loss function applicable to the network model;

[0011] Step 4: Train the network model using the image dataset to obtain the final trained weakly supervised semantic segmentation network model;

[0012] Step 5: Input the image to be segmented into the network model with trained model parameters, and output the segmentation result image of the image.

[0013] Furthermore, in step one: the semantic segmentation image dataset with weak labels is selected from the PASCALVOC dataset and the SBD dataset, and the images in the dataset are scaled and cropped.

[0014] Furthermore, in step two, the Transformer encoder consists of four Transformer blocks, each of which includes a multi-head self-attention mechanism module and a feedforward neural network. After the image is input into the encoder, it is decomposed into multiple linearly embedded image blocks. These blocks are then processed by the four Transformer blocks to obtain multi-level features of the original image resolution, generating high-resolution coarse features and low-resolution fine-grained features.

[0015] The decoder is a lightweight fully connected MLP decoder used to fuse multi-level features generated by the encoder to generate the final semantic segmentation mask.

[0016] The pseudo-label refinement module is used to optimize pseudo-labels. The pseudo-label refinement module includes a module for generating an affinity matrix, a pixel adaptive module, and a random walk algorithm.

[0017] The contextual attention module is used to optimize the entire network using contextual information, and includes a common attention module and a contrastive attention module.

[0018] Furthermore, in step two, the steps for generating pseudo-labels from the constructed network model are as follows:

[0019] Step (2.1), generation of CAM diagram and initial pseudo-labels;

[0020] 1) Generation of CAM maps: The image first obtains feature maps in the Transformer encoder. The feature maps enter the MLP classification layer of the decoder. In the MLP classification layer, the ReLU function is used to learn weights, which are then weighted and added to the feature maps to generate CAM maps for each category.

[0021] 2) Generation of initial pseudo-labels: The CAM image is processed by using the ReLU function to clear the data to zero and setting a threshold to obtain the initial pseudo-labels;

[0022] Step (2.2): Further processing of the initial pseudo-labels obtained in step (2.1):

[0023] 1) Using the multi-head self-attention of the Transformer encoder and the MLP layer of the decoder, the affinity prediction of the input image is generated to obtain the affinity matrix;

[0024] 2) Use the initial pseudo-labels from step (2.1) to generate affinity relationships and obtain pseudo-affinity labels;

[0025] 3) The affinity prediction of the input image and the affinity relationship generated by the initial pseudo-label are used as prediction and supervision signals, respectively, to optimize the initial pseudo-label and obtain further optimized pseudo-labels;

[0026] Step (2.3) involves using a random walk algorithm to propagate the pseudo-labels obtained in step (2.2), and then using a pixel adaptive module to further optimize the propagated pseudo-labels to obtain the final pseudo-labels.

[0027] Furthermore, in step (2.1):

[0028] 1) The process of generating a CAM drawing can be expressed by the following formula:

[0029]

[0030] Where W is the network layer weight, F is the feature map, and M is the network layer weight. c d represents the activation mapping for a given category c, where c represents the specific category used to generate the response activation mapping; d is the feature vector dimension of each element; and i is a value that ranges from 1 to d.

[0031] 2) The specific process of generating the initial pseudo-labels is as follows: two hyperparameters β, with a range of 0 to 1, are predefined. l and β h The initial pseudo-labels obtained are processed according to the following rules: those below β l The region is set to 0 to represent the background region; regions higher than β... hThe region uses argmax to obtain the category of each pixel location; above β l Below β h The region is set to 255, indicating that the region is uncertain; this process is expressed by the formula:

[0032]

[0033] Where M is the CAM matrix, the two βs are thresholds, and W p The initial pseudo-labels i and j represent the positions of the pixels;

[0034] In step (2.2):

[0035] 1) The specific process of generating affinity prediction for the input image is as follows:

[0036] The specific process of generating affinity prediction for the input image is as follows: Let the multi-head self-attention in the Transformer be denoted as S∈R. hw×hw×n Where h×w is the number of patches, and n is the number of heads in the multi-head self-attention matrix. Since the multi-head attention matrix is ​​asymmetric, while the affinity between image regions is symmetric, the multi-head attention matrix S and its transpose S are used to define the multi-head attention matrix S. T Adding them together yields a symmetric attention matrix, denoted as (S+S). T The symmetric self-attention matrix is ​​passed through an MLP layer, reducing the original n-dimensionality to 1-dimensionality, ultimately yielding the affinity matrix A∈R. hw×hw The entire process can be expressed by the following formula:

[0037] A = MLP(S + S) T )

[0038] 2) The specific process of generating affinity relationships is as follows;

[0039] W p The region with a value of 255 is designated as the uncertain region, and the remaining region represents the certain region. The certain region is further divided into the background region with a value of 0 and the foreground region with values ​​corresponding to the category labels. For each location, the region within its radius r is considered as its neighborhood, and then the relationship between the location and its neighborhood is analyzed. Specifically, assuming a location i is known, and one of its neighbors is j, if i and j are both certain regions, there are three cases, described by the rules below:

[0040] In the first scenario, if and If the values ​​are the same, then... Set it to 1, and mark the corresponding positions i and j as positive position pairs;

[0041] In the second scenario, if and If the values ​​are different, then... Set it to 0, and mark the corresponding positions i and j as negative position pairs;

[0042] In the third case, if either i or j belongs to the uncertain region, then... Setting it to 255 means ignoring this position; if i and j are outside the neighborhood, then... and The values ​​are the same, all set to 255;

[0043] The pseudo-affinity label W is obtained according to the above rules. pr That is, the label W of the affinity matrix. pr ∈R hw×hw ;

[0044] In step (2.3):

[0045] 1) The initial pseudo-labels are propagated using a random walk algorithm. The specific steps are as follows:

[0046] First, calculate the affinity matrix to the power of α, then normalize it by dividing each row by the sum of its elements; finally, sum the affinities to the CAM matrix M∈R. hw×hw×C Matrix multiplication yields the processed CAM matrix M. pr The entire process can be expressed by the following formula:

[0047] T=D -1 A α

[0048]

[0049] M pr =T*vec(M)

[0050] Where α is the set hyperparameter, A is the affinity matrix, M is the CAM matrix, D is the sum of the rows of A, used to normalize A, T is the normalized A, and D is the normalized A. ii Let A represent the normalization factor of the affinity matrix A in the i-th row. ik The element in the i-th row and k-th column of the affinity matrix A is raised to the power of α; the vec(M) operation converts M ∈ R. hw×hw×C Transform into M∈R hw×C M pr These are pseudo-labels that have spread and proliferated.

[0051] 2) The specific steps for further optimizing the initial pseudo-labels of diffusion propagation using the pixel adaptive module are as follows:

[0052] ① Given an image I∈R h×w×3For pixel positions (i,j) and (k,l), the kernel of its RGB and spatial pairs is defined by the following formula:

[0053]

[0054]

[0055] In the formula, I ij I represents the RGB information at pixel position (i,j) in the image. kl P represents the RGB information at pixel position (k,l) in the image; ij P represents the spatial location of pixel (i,j). kl σ represents the spatial location of pixel (k,l); rgb σ represents the standard deviation of RGB values. pos The standard deviation of position; The kernel represents the RGB pair and is used to measure the similarity between image pixels (i,j) and (k,l); The kernel represents the position pair and is used to measure the spatial positional similarity between pixels (i,j) and (k,l); ω1 and ω2 both represent the RGB kernel.

[0056] ② Put k rgb ,k pos After normalization using softmax and then weighting, we obtain k. ij,kl As the affinity core of the pixel adaptive module, the formula is expressed as follows:

[0057]

[0058] In the formula, k ij,kl ω3 represents the affinity nucleus between a specific position (i,j) and another position (k,l); ω3 represents the RGB nucleus.

[0059] ③ Through the affinity kernel of the pixel adaptive module, both the initial CAM and the propagated CAM can be refined simultaneously;

[0060] The refinement operation is implemented through multiple iterative processes, for CAM M∈R h×w×c In the t-th iteration, we have:

[0061]

[0062] In the formula, Let represent the CAM value of the c-th channel at position (i,j) after the t-th iteration. This represents the CAM value of the c-th channel at position (k,l) after t-1 iterations.

[0063] Furthermore, in step three, the specific steps for the loss function of the network model are as follows:

[0064] Step (3.1): Randomly crop a fixed number of local images from the specified uncertain regions and background regions in the input image. Assign these local images as positive or negative samples, with each positive or negative sample forming an image pair. Obtain the category scores s for the positive and negative samples in each image pair. m s n :

[0065] Step (3.2): Obtain the loss of the common attention module.

[0066] The cropped image pairs obtained in step (3.1) are input into the Transformer encoder to extract the feature maps of each image. These feature maps are then fed into a fully convolutional layer of the common attention module to obtain class-aware activation maps. After obtaining the feature maps of each image in the image pair, the affinity matrix between the two images in the image pair is calculated to obtain the common attention features.

[0067] Step (3.3): Obtain the contrastive attention module loss.

[0068] Using the common attention features obtained in step (3.2) as input, the convolutional layer and sigmoid function operation of the contrastive attention module enhance the shared semantic information in the common attention features, resulting in enhanced shared semantic information. Then, the contrastive attention matrix is ​​obtained, which inversely highlights the non-common semantic regions outside the shared semantics. The loss of this step is denoted as...

[0069] Step (3.4): Set the overall loss function for a suitable network model;

[0070] Overall loss of the network model: classification loss Similarity loss and contextual attention loss composition:

[0071] 1) For classification loss, the aggregated features are fed into the MLP classification layer to calculate the class probability vector, and then multi-label soft margin loss is used as the classification function. The calculation formula is as follows:

[0072]

[0073] In the formula, C represents the number of categories; c represents the category index, ranging from 1 to C; y c This represents the tag of class c in the actual tag; This represents the probability that a sample from the model's pre-training belongs to class c;

[0074] 2) For similarity loss The calculation formula is as follows:

[0075]

[0076] In the formula, R + and R - W pr The set of positive and negative samples in N + and N - Used to calculate R + and R - The number of;

[0077] 3) Overall loss function The formula is expressed as follows:

[0078]

[0079] In the formula, λ1 and λ2 represent the values ​​used for adjustment. Loss items and Hyperparameters of the weights of the loss term.

[0080] Beneficial effects:

[0081] (1) This invention proposes a weakly supervised semantic segmentation method that combines contextual attention with Transformer and attention mechanism. By generating and optimizing pseudo-labels through Transformer and contextual content, the network can better extract target features on datasets with weakly supervised labels, thereby completing weakly supervised semantic segmentation.

[0082] (2) The present invention designs a method that combines Transformer network with contextual information to perform detailed processing on the generated CAM, improve the difference between the foreground region and the background region, and help the CAM to activate a more complete target region, thereby optimizing the pseudo-labels generated by weakly supervised semantic segmentation, obtaining better segmentation results, and effectively completing weakly supervised semantic segmentation.

[0083] (3) This invention is applicable to scenarios where image segmentation is required but the cost of labeled data is high or difficult to obtain, such as medical image analysis, remote sensing image analysis and autonomous driving and intelligent transportation. Attached Figure Description

[0084] Figure 1 This is a schematic diagram of the weakly supervised semantic segmentation network model constructed in an embodiment of the present invention;

[0085] Figure 2This is the encoder module structure constructed in the embodiments of the present invention;

[0086] Figure 3 This is an explanation of the input image cropping method in the embodiments of the present invention;

[0087] Figure 4 This is a schematic diagram illustrating the process of calculating category scores in an embodiment of the present invention;

[0088] Figure 5 This is a schematic diagram illustrating the calculation process of the context attention module in an embodiment of the present invention;

[0089] Figure 6 This is an illustration of extracting effective pixel pairs when calculating affinity loss in an embodiment of the present invention;

[0090] Figure 7 This is a visual semantic segmentation result in an embodiment of the present invention. Detailed Implementation

[0091] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described. The Transformer weakly supervised semantic segmentation method combining contextual attention provided by the embodiments of the present invention includes the following steps:

[0092] Step 1: Select a semantic segmentation image dataset with weak labels.

[0093] The semantic segmentation image dataset with weak labels is selected from the PASCAL VOC dataset and the SBD dataset. The images in the dataset are scaled and cropped to obtain the augmented dataset. The augmented dataset is then divided into training set, validation set and test set according to the proportions.

[0094] In this embodiment, the dataset used is PASCAL VOC 2012. PASCAL VOC (Visual Object Classes) is a commonly used dataset for object recognition and segmentation. It contains images in 20 categories, covering a range of different objects such as people, vehicles, animals, and furniture. This invention further adds the SBD (Semantic Boundaries Dataset) dataset to the PASCAL VOC 2012 dataset.

[0095] The augmented dataset consists of 10582, 1449, and 1456 images for training, validation, and testing, respectively. During training, only image-level labels are used. mIOU (mean Intersection over Union) is used as the evaluation metric, measuring the degree of overlap between predicted and ground truth segmented regions. The image to be segmented is input into the trained network model, which outputs the segmented image and evaluates it using technical metrics, including:

[0096] Pixel accuracy (PA) is the total number of correctly segmented pixels divided by the total number of pixels.

[0097]

[0098] The mean Intersection over Union (mIoU) is calculated by taking the IoU (intersection over union of the true label and the predicted result) for each class separately, and then averaging the IoUs for all classes.

[0099]

[0100] Where k is the total number of categories, or k+1 if the background is included; p ij It represents the total number of pixels whose true pixel class is i but are predicted to be class j; in other words, for any given pixel of class i, how many are misclassified as class j. ii The total number of pixels whose true pixel class is i that are predicted to be class i is the total number of pixels that are correctly classified as true class i.

[0101] Step 2: Construct a Transformer weakly supervised semantic segmentation network model that incorporates contextual attention.

[0102] The constructed network model is an end-to-end network model that incorporates a Transformer mechanism with contextual attention; such as Figure 1 As shown, the constructed network model includes a Transformer encoder, a decoder, a pseudo-label refinement module, and a contextual attention module. The pseudo-label refinement module is used to optimize pseudo-labels, including a module for generating affinity matrices, a pixel adaptation module, and a random walk algorithm; the contextual attention module is used to optimize the entire network using contextual information, including a common attention module and a contrastive attention module.

[0103] (1) Transformer encoder, structure as follows Figure 2 As shown;

[0104] The Transformer encoder is a hierarchical Transformer consisting of four Transformer blocks, each including a multi-head self-attention mechanism module and a feedforward neural network. After the image is input into the encoder, it is decomposed into multiple linearly embedded image blocks. These blocks are then processed by the four Transformer blocks to obtain multi-level features at 1 / 4, 1 / 8, 1 / 16, and 1 / 16 of the original image resolution, respectively, generating high-resolution coarse features and low-resolution fine-grained features.

[0105] Specifically, the processing steps after the image is input into the encoder are as follows:

[0106] (1.1) In the first stage of the encoder, the input image is first divided into 4×4 segments, and then the segmented images are merged by overlapping patching.

[0107] Given an image patch, a 2×2×C i The feature patch is unified into a 1×1×C i A vector with +1, where C i The number of feature channels can be understood as a convolution with a kernel of 2, a stride of 2, and padding of 0, where the stride of the last layer is set to 1. This step obtains hierarchical feature maps and uses this process to iterate to other feature maps in the hierarchical structure. Because this process cannot preserve the local continuity around these patches, convolution is set to perform patch merging, thereby generating features of the same size as the non-overlapping process.

[0108] (1.2) In the second stage of the encoder, the computational complexity of the self-attention mechanism module in the Transformer block is reduced by adding a scaling factor.

[0109] To reduce the overall computational complexity of the network, a scaling factor R is added to reduce the computational complexity of each self-attention mechanism module. The traditional self-attention mechanism is shown in the following formula:

[0110]

[0111] In the formula, Q is the query vector, K is the keyword vector, V is the value vector, and d head Let be the vector dimension. For multi-head self-attention, the Q, K, and V of each head are all of the same dimension N×C, where N = H×W is the sequence length. The computational complexity of this process is incalculable for high-resolution images. To reduce computation, a scaling factor R is added, resulting in:

[0112]

[0113]

[0114] In the formula, It is formed by reshaping a K matrix with input dimension N×C. A matrix of size (C·R). Then, through a linear transformation, the dimension of (C·R) is changed to C, thus obtaining an output of size (C·R). K. N represents the number of input vectors, C represents the feature dimension of each input vector, and R represents the scaling factor. First, a reshape operation is performed, which transforms the N×C feature map into a (N / R*C*R) feature map. Then, a fully connected layer transforms (N / R*C*R) into (N / R*C). From Transformer Block 1 to Transformer Block 4, the scaling factor R of the Transformer Block is 64, 16, 4, and 1, respectively.

[0115] (1.3) In the third stage of the encoder, Mix-FFN is introduced, and 3×3 convolution is used in the feedforward network (FFN) to take into account the impact of leaked position information.

[0116] Mix-FFN can be represented as:

[0117] x out =MLP(GELU(Conv 3*3 (MLP(x in ))))+x in

[0118] Where, x in The features are derived from the self-attention module. Mix-FFN mixes a 3×3 convolution and an MLP into each FFN, x out This represents x after Mix-FFN processing. in .

[0119] (2) Decoder

[0120] The decoder is a lightweight, fully connected MLP decoder. This module is used to fuse multi-level features generated by the encoder to generate the final semantic segmentation mask.

[0121] The decoder of this invention is a lightweight decoder, consisting of only MLP layers. The decoder mainly includes four steps: first, the multi-level features F from the encoder... iThe process begins with an MLP layer. The second step unifies the channel dimensions, then upsamples to 1 / 4 the size and concatenates the data. The third step involves feature fusion through an MLP layer, followed by prediction through another MLP layer, effectively fusing features from different resolutions. The decoder can be formalized as follows:

[0122] M=MLP3(MLP2(Concatenate(UpSample(F),F1,F2,F3,F4)))

[0123] Here, UpSample represents the upsampling operation, Concatenate represents the concatenation operation, MLP2 and MLP3 are MLP layers used for fusion and prediction, respectively, and F1 to F4 are features from different resolutions.

[0124] (3) Pseudo-tag refinement module

[0125] The pseudo-label refinement module includes an affinity similarity module, a random walk module, and a pixel adaptation module.

[0126] The overall steps for generating pseudo-labels using the constructed network model are as follows: First, pseudo-labels are generated; then, they pass through an affinity and similarity module; finally, random walks and pixel adaptation are performed to refine the pseudo-labels. The specific steps are as follows:

[0127] Step (2.1), generation of CAM diagram and initial pseudo-label.

[0128] 1) Generation of CAM diagrams:

[0129] The image first obtains a feature map in the Transformer encoder. The feature map enters the MLP classification layer of the decoder. In the MLP classification layer, the weights are learned using the ReLU function and then weighted and added to the feature map to generate a CAM map for each category.

[0130] Specifically, this invention uses a Transformer as the encoder, and each image input to the encoder corresponds to a set of output feature maps F∈R. hw×d F is a two-dimensional matrix, hw represents the height multiplied by the width of the feature map, i.e., the total number of elements in the feature map; d represents the dimension of the feature vector of each element, indicating that the feature is d-dimensional. After the generated feature map passes through the MLP classification layer of the decoder, a c-dimensional CAM map can be obtained, where c refers to the dimension of the category. The obtained CAM map is a multi-scale fused CAM map. Each image is scaled to obtain different sizes, and these images of different sizes form an image sequence. Then, the CAM map of this sequence of images is obtained, scaled again to obtain CAM maps of the same size, and accumulated to obtain a multi-scale fused CAM map.

[0131] The entire process of generating a CAM diagram can be expressed by the following formula:

[0132]

[0133] Where W is the network layer weight, F is the feature map, and M is the network layer weight. c d represents the activation mapping for a given category c, where c represents the specific category used to generate the response activation mapping; d is the feature vector dimension of each element; and i is a value that ranges from 1 to d.

[0134] 2) Generation of initial pseudo-labels: The CAM image is processed by using the ReLU function to clear the data to zero and set a threshold to obtain the initial pseudo-labels;

[0135] The initial pseudo-label generation process is as follows: First, two hyperparameters β, ranging from 0 to 1, are predefined. l and β h The initial pseudo-labels obtained are processed according to the following rules: those below β l The region is set to 0 to represent the background region; regions higher than β... h The region uses argmax to obtain the category of each pixel location; above β l Below β h The region is set to 255, indicating that this region is an uncertain region. The process can be expressed by the formula:

[0136]

[0137] Where M is the CAM matrix, the two βs are thresholds, and W p The initial pseudo-labels i and j represent the positions of the pixels.

[0138] Step (2.2): Further processing of the initial pseudo-labels obtained in step (2.1):

[0139] 1) Using the multi-head self-attention (MHSA) layer of the Transformer encoder and the MLP layer of the decoder, the affinity prediction of the input image is generated to obtain the affinity matrix;

[0140] The specific process of generating affinity prediction for the input image is as follows: Let the multi-head self-attention in the Transformer be denoted as S∈R. hw×hw×n Where h×w is the number of patches, and n is the number of heads in the multi-head self-attention matrix. Since the multi-head attention matrix is ​​asymmetric, while the affinity between image regions is mutually symmetric, the multi-head attention matrix S and its transpose S are... T Adding them together yields a symmetric attention matrix, denoted as (S+S). T The symmetric self-attention matrix is ​​passed through an MLP layer, reducing the original n-dimensionality to 1-dimensionality, ultimately yielding the affinity matrix A∈R.hw×hw The entire process can be expressed by the following formula:

[0141] A = MLP(S + S) T )

[0142] 2) In step (2.1), the initial pseudo-labels generate affinity relationships in the affinity similarity module, resulting in pseudo-affinity labels; the process of generating affinity matrix relationships from the initial pseudo-labels is as follows:

[0143] W p The region with a value of 255 is set as the uncertain region, and the remaining region represents the certain region. The certain region is further divided into the background region with a value of 0 and the foreground region with a value of each category label. For each location, the region within its radius r is regarded as the neighborhood, and then the relationship between the location and the neighborhood is analyzed.

[0144] Specifically, assuming a given location is i and its neighborhood is j, if both i and j are definite regions, there are three cases, described by the following rules:

[0145] In the first scenario, if and If the values ​​are the same, then... Set it to 1, and mark the corresponding positions i and j as positive position pairs;

[0146] In the second scenario, if and If the values ​​are different, then... Set it to 0, and mark the corresponding positions i and j as negative position pairs;

[0147] In the third case, if either i or j belongs to the uncertain region, then... Setting it to 255 means ignoring this position; if i and j are outside the neighborhood, then... and If the values ​​are the same, set them all to 255.

[0148] Following the above rules, the pseudo-affinity label W can be obtained. pr That is, the label W of the affinity matrix. pr ∈R hw×hw The effective pixel pairs used to calculate affinity loss are illustrated in the diagram below. Figure 6 As shown.

[0149] 3) The affinity prediction of the input image and the affinity relationship generated by the initial pseudo-label are used as prediction and supervision signals, respectively, to optimize the initial pseudo-label and obtain further optimized pseudo-labels;

[0150] Step (2.3): The pseudo-labels obtained in step 2.2 are propagated using a random walk algorithm. Then, the pixel adaptive module is used to further optimize the propagated pseudo-labels to obtain the final pseudo-labels.

[0151] The final pseudo-tag generation process is as follows:

[0152] 1) First, use the Random Walk algorithm to spread the initial pseudo-labels.

[0153] Specifically, first consider the affinity matrix A∈R hw×hw To process it, first calculate the affinity matrix to the power of α, then normalize it by dividing each row by the sum of its elements; then sum it with the CAM matrix M∈R. hw×hw×C Matrix multiplication yields the processed CAM matrix M. pr .

[0154] The entire process can be expressed by the following formula:

[0155] T=D -1 A α

[0156]

[0157] M pr =T*vec(M)

[0158] Where α is the set hyperparameter, A is the affinity matrix, M is the CAM matrix, D is the sum of the rows of A, used to normalize A, T is the normalized A, and D is the normalized A. ii Let A represent the normalization factor of the affinity matrix A in the i-th row. ik The element in the i-th row and k-th column of the affinity matrix A is raised to the power of α; the vec(M) operation converts M ∈ R. hw×hw×C Transform into M∈R hw×C M pr These are pseudo-tags after propagation. h refers to height, i.e., the height of the image; w refers to width, i.e., the width of the image; and c refers to channel, i.e., the number of channels in the image.

[0159] 2) The initial pseudo-labels of the diffusion propagation are further optimized using the pixel adaptive module to obtain the final pseudo-labels.

[0160] Due to the pseudo-affinity label W prThe initial pseudo-labels are generated from the initial labels, but these initial labels are coarse and locally inconsistent. For example, adjacent pixels with similar image appearances do not share the same semantics. To ensure local consistency, some methods use dense random fields to refine the initial pseudo-labels, but this is not a good choice due to its low training efficiency. Therefore, this invention employs pixel-adaptive convolution to extract local RGB information for refinement, that is, combining RGB information and spatial information to define low-level pairwise affinity and construct a pixel-adaptive refinement module (PAR).

[0161] The initial pseudo-labels for diffusion propagation are further optimized using a pixel adaptive module to obtain the final pseudo-labels. The specific steps are as follows:

[0162] ① Given an image I∈R h×w×3 For pixel positions (i,j) and (k,l), the kernel of its RGB and spatial pairs is defined by the following formula:

[0163]

[0164]

[0165] In the formula, I ij I represents the RGB information at pixel position (i,j) in the image. kl P represents the RGB information at pixel position (k,l) in the image; ij P represents the spatial location of pixel (i,j). kl σ represents the spatial location of pixel (k,l); rgb σ represents the standard deviation of RGB values. pos The standard deviation of position; The kernel represents the RGB pair and is used to measure the similarity between image pixels (i,j) and (k,l); The kernel represents the position pair and is used to measure the spatial positional similarity between pixels (i,j) and (k,l); ω1 and ω2 both represent the RGB kernel.

[0166] ② Put k rgb ,k pos After normalization using softmax and then weighting, we obtain k. ij,kl As the affinity core of the pixel adaptive module, the formula is expressed as follows:

[0167]

[0168] In the formula, k ij,klω represents the affinity nucleus between a specific position (i,j) and another position (k,l); ω3 represents the RGB nucleus.

[0169] ③ By using the affinity kernel of the pixel adaptive module, both the initial CAM and the propagated CAM can be refined simultaneously.

[0170] This refinement operation is achieved through multiple iterative processes, for CAM M∈R h×w×c In the t-th iteration, we have:

[0171]

[0172] In the formula, Let represent the CAM value of the c-th channel at position (i,j) after the t-th iteration. This represents the CAM value of the c-th channel at position (k,l) after t-1 iterations.

[0173] In order to expand the receptive field, this invention uses multiple dilated convolutions to extract information from multiple nearest neighbor pixels. After obtaining the low-level affinity relationship between pixels, the pseudo-labels are corrected through multiple iterations.

[0174] Step 3: Set the loss function suitable for the Transformer weakly supervised semantic segmentation network model that incorporates contextual attention;

[0175] Given an image, firstly, random cropping of local images is performed from specified uncertain regions. To handle the possibility that the cropped local images may contain few or no foreground objects, some local images are also cropped from the background region. After passing through a common attention module and a contrastive attention module, semantic relationships between the randomly cropped local images are extracted. The computation process of the context attention module is as follows: Figure 5 As shown.

[0176] The specific steps for step three are as follows:

[0177] Step 3.1: Randomly crop a fixed number of local images from the specified uncertain region and background region in the input image. Assign these local images as positive samples (from the uncertain region) or negative samples (from the background region), forming a pair of images. Obtain the class scores s of the positive and negative samples in the image pair. m s n ;

[0178] First, sample images from the cropped image are compared to I. m and I n ,l m and l nThis represents the category attributes present in the image; subsequently, the feature map F of each image data is extracted in the Transformer encoder. m With F n After the feature maps are extracted, they are fed into a fully convolutional layer of the common attention module. Obtain the class activation map S m and S n Then, after global average pooling, I is obtained. m and I n Category scores m s n Cutting method as follows Figure 3 As shown, Figure 4 The category score calculation process is as follows: Figure 4 As shown.

[0179] Step 3.2: Obtain the loss of the common attention module.

[0180] The image pairs obtained from cropping in step 3.1 are input into MiT (MixTransformer, Transformer encoder) to extract the feature maps of each image. These feature maps are then fed into a fully convolutional layer of the common attention module to obtain class-aware activation maps. After obtaining the feature maps of each image in the image pair, the affinity matrix between the two images in the image pair is calculated to obtain the common attention features.

[0181] In this image pair, both images retain the common semantics from each other and locate objects sharing the same semantics in their original feature maps. In the joint attention map, only regions with the same semantic features show significant activation, while regions with different semantic features receive only lower weights. Therefore, shared semantic information can be accurately located; the loss for this step is denoted as...

[0182] The specific steps of step 3.2 are as follows:

[0183] 1) First, calculate the feature map F. m With F n Affinity matrix between them;

[0184] First, put F m With F n Tiled to a size of C×HW, where C represents the number of channels and HW represents the spatial dimension. Calculate F using the formula below. m With F n The similarity between them.

[0185]

[0186] Here, Y pIt is a C×C learnable matrix, and according to this formula, each element P in the affinity matrix P is... i,j All represent F m The i-th element and F n The similarity between the j-th elements.

[0187] 2) Then, perform softmax operations on the affinity matrix P by row and column respectively to obtain... and Normalized attention map A for each element m and F m With F n Normalized attention map A for each element n .

[0188] A m =softmax(P)

[0189] A n =softmax(P T )

[0190] 3) Next, calculate F. n (F m ) for F m (F n Attention, as shown in the following formula:

[0191]

[0192]

[0193] and It preserves the common semantics from each other and marks the shared semantic objects in its own original feature map.

[0194] 4) Therefore, the cross-entropy loss function can be used. The loss of the common attention module is obtained. for:

[0195]

[0196] Step 3.3: Obtain the contrastive attention module loss.

[0197] Using the common attention features obtained in step 3.2 as input, the features are processed through a convolutional layer and a sigmoid function operation in the contrastive attention module to enhance the shared semantic information within the common attention features, resulting in enhanced shared semantic information. This leads to the contrastive attention matrix, which in turn highlights non-common semantic regions outside the shared semantics. The loss in this step is denoted as...

[0198] Specifically, the input to the contrastive attention module is the common attention features extracted by the common attention module. and First, we compare how attention modules enhance shared semantic information in common attention features through convolution operations (C) and the sigmoid function:

[0199]

[0200]

[0201] Subtracting the above values ​​from 1 yields the contrastive attention matrix:

[0202]

[0203]

[0204] get and Then, calculate the contrastive attention features:

[0205]

[0206]

[0207] After obtaining the contrastive attention features, the contrastive attention loss function can be obtained as follows:

[0208]

[0209]

[0210] in, It is the loss of the shared attention module. It compares the loss of the attention module. It represents the loss of the entire context attention module.

[0211] Step 3.4: Set the overall loss function for a suitable network model.

[0212] The overall loss of the network model in this invention consists of three loss terms: classification loss and classification loss. Similarity loss and contextual attention loss

[0213] 1) For classification loss, the aggregated features are fed into the MLP classification layer to calculate the class probability vector, and then the multi-label soft margin loss is used as the classification function. The formula for calculating the classification loss is as follows:

[0214]

[0215] In the formula, C represents the number of categories; c represents the category index, ranging from 1 to C; y c This represents the tag of class c in the actual tag; This represents the probability that a sample from the model's pre-training belongs to class c.

[0216] 2) The similarity loss is calculated using the following formula:

[0217]

[0218] In the formula, R + and R - W pr The set of positive and negative samples in N; + and N - Used to calculate R + and R - The number of.

[0219] 3) The overall loss function formula is expressed as follows:

[0220]

[0221] In the formula, λ1 and λ2 represent the values ​​used for adjustment. Loss items and Hyperparameters of the weights of the loss term.

[0222] Step 4: Train the network model using the image dataset to obtain the final trained weakly supervised semantic segmentation network model;

[0223] Specifically, the training set of the image dataset from step one is preprocessed through operations such as scaling and random cropping, and then fed into the network model constructed in step two for training. During training, the loss function value is calculated according to the loss function set in step three, and the obtained loss is fed back to the network to update the network model weights.

[0224] Furthermore, as a preferred technical solution of this invention, a variable learning rate method is employed during training, adjusting the learning rate differently in each iteration. Adjusting the learning rate value in each iteration is another effective adaptive learning rate method. The basic idea of ​​this type of method is that the further you are from the optimal value, the more you need to move towards the optimal value, i.e., the learning rate should be larger, and vice versa. This leads to the final trained weakly supervised semantic segmentation network model.

[0225] The resulting loss is fed back into the network to update the network model weights, by weighting the different loss terms. The weighting of λ1 and λ2 here (weighting the similarity loss and the contrastive attention module loss) can adjust the degree of influence of each item in model training, ensuring that the model learns the features corresponding to each loss better.

[0226] Step 5: Input the image to be segmented into the network model with trained model parameters, and output the segmentation result image of the image.

[0227] This embodiment is validated using a validation set. Through actual calculations, the network model in this invention can achieve a maximum image segmentation accuracy of 81.0% and a mean IoU of 65.2% for images with only images and labels. Figure 7 Example of visualizing semantic segmentation results.

[0228] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A Transformer weakly supervised semantic segmentation method incorporating contextual attention, characterized in that, Includes the following steps: Step 1: Select a semantic segmentation image dataset with weak labels; Step 2: Constructing a context-sensitive attention mechanism The weakly supervised semantic segmentation network model consists of a Transformer encoder, decoder, pseudo-label refinement module, and context attention module. Step 3: Set the loss function applicable to the network model; Step 4: Train the network model using the image dataset to obtain the final trained weakly supervised semantic segmentation network model; Step 5: Input the image to be segmented into the network model with trained model parameters, and output the segmentation result image of the image. In step two, the Transformer encoder consists of four Transformer blocks, each of which includes a multi-head self-attention mechanism module and a feedforward neural network. After the image is input into the encoder, it is decomposed into multiple linearly embedded image blocks. These blocks are then processed by the four Transformer blocks to obtain multi-level features of the original image resolution, generating high-resolution coarse features and low-resolution fine-grained features. The decoder is a lightweight fully connected MLP decoder used to fuse multi-level features generated by the encoder to generate the final semantic segmentation mask. The pseudo-label refinement module is used to optimize pseudo-labels. The pseudo-label refinement module includes a module for generating an affinity matrix, a pixel adaptive module, and a random walk algorithm. The contextual attention module is used to optimize the entire network using contextual information, including a common attention module and a contrastive attention module; In step two, the steps for generating pseudo-labels from the constructed network model are as follows: Step (2.1), generation of CAM diagram and initial pseudo-labels; 1) Generation of CAM maps: The image first obtains feature maps in the Transformer encoder. The feature maps enter the MLP classification layer of the decoder. In the MLP classification layer, the ReLU function is used to learn weights, which are then weighted and added to the feature maps to generate CAM maps for each category. 2) Generation of initial pseudo-labels: The CAM image is processed by using the ReLU function to clear the data to zero and setting a threshold to obtain the initial pseudo-labels; Step (2.2): Further processing of the initial pseudo-labels obtained in step (2.1): 1) Use Multi-head self-attention and decoder The layer generates affinity predictions for the input image, resulting in an affinity matrix; 2) Use the initial pseudo-labels from step (2.1) to generate affinity relationships and obtain pseudo-affinity labels; 3) The affinity prediction of the input image and the affinity relationship generated by the initial pseudo-label are used as prediction and supervision signals, respectively, to optimize the initial pseudo-label and obtain further optimized pseudo-labels; Step (2.3) The pseudo-labels obtained in step (2.2) are spread using a random walk algorithm, and then the pixel adaptive module is used to further optimize the spread pseudo-labels to obtain the final pseudo-labels.

2. The Transformer weakly supervised semantic segmentation method combining contextual attention as described in claim 1, characterized in that, In step one: the semantic segmentation image dataset with weak labels is selected. The dataset is used to scale and crop the images within it.

3. The Transformer weakly supervised semantic segmentation method combining contextual attention as described in claim 1, characterized in that, In step (2.1): 1) The process of generating a CAM drawing can be expressed by the following formula: ； Where W is the network layer weight and F is the feature map. d represents the activation mapping for a given category c, where c represents the specific category used to generate the response activation mapping; d is the feature vector dimension of each element; and i is a value that ranges from 1 to d. 2) The specific process of generating the initial pseudo-tags is as follows: two hyperparameters with a range of 0 to 1 are predefined. and The initial pseudo-labels obtained are processed according to the following rules: lower than The area is set to 0 to represent the background area; areas higher than 0 are considered background areas. The region uses argmax to obtain the category of each pixel location; higher than Below The region is set to 255, indicating that the region is uncertain; this process is expressed by the formula: ； Where M is the CAM matrix, two It is a threshold. It is the initial pseudo tag Indicates the position of the pixel; In step (2.2): 1) The specific process of generating affinity prediction for the input image is as follows: The specific process of generating affinity predictions for the input image is as follows: The multi-head self-attention in the middle is denoted as S. ,in, is the number of patches, and n is the number of heads in the multi-head self-attention matrix. Since the multi-head attention matrix is ​​asymmetric, while the affinity between image regions is symmetric, the multi-head attention matrix... and its transpose Adding them together yields a symmetric attention matrix, denoted as... The symmetric self-attention matrix is ​​passed through an MLP layer, reducing the original n-dimensionality to 1-dimensionality, ultimately yielding the affinity matrix. The entire process can be expressed by the following formula: ； 2) The specific process of generating affinity relationships is as follows; Will The region with a value of 255 is designated as the uncertain region, and the remaining region represents the certain region. The certain region is further divided into a background region with a value of 0 and a foreground region with values ​​corresponding to the category labels. For each location, the region within its radius r is considered its neighborhood, and the relationship between that location and its neighborhood is analyzed. Specifically, assuming a location i is known, and one of its neighborhoods is j, if... All cases involve defined regions and fall into three categories, with the rules described below: In the first scenario, if and If the values ​​are the same, then... Set it to 1, and mark the corresponding positions i and j as positive position pairs; In the second scenario, if and If the values ​​are different, then... Set it to 0, and mark the corresponding positions i and j as negative position pairs; In the third case, if either i or j belongs to the uncertain region, then... Setting it to 255 means ignoring this position; if i and j are outside the neighborhood, then... and The values ​​are the same, all set to 255; The pseudo-affinity label is obtained according to the above rules. That is, the labels of the affinity matrix. ; In step (2.3): 1) The initial pseudo-labels are propagated using a random walk algorithm. The specific steps are as follows: First, calculate the affinity matrix. The normalization process involves dividing each row by the sum of its elements; then, the result is combined with the CAM matrix. Matrix multiplication yields the processed CAM matrix. The entire process can be expressed by the following formula: ； ； ； in, These are the hyperparameters that are set. A is the affinity matrix, M is the CAM matrix, D is the sum of the rows of A, used to normalize A, and T is the normalized A. This represents the normalization factor of the affinity matrix A in the i-th row. Let represent the element in the i-th row and k-th column of the affinity matrix A, raised to the power of α. Operation will become ; These are pseudo-labels that have spread and proliferated. 2) The specific steps for further optimizing the initial pseudo-labels of diffusion propagation using the pixel adaptive module are as follows: ① Given an image For pixel position Its RGB and space pair kernel is defined by the following formula: ； ； In the formula Represents the pixel position in the image RGB information at the location, Represents the pixel position in the image RGB information at the location; Represents pixels Spatial location, Represents pixels Spatial location; Represents the standard deviation of RGB; The standard deviation of position; The kernel represents the RGB pairs and is used to measure image pixels. and Similarity between them; The kernel represents the position pair and is used to measure pixels. and Spatial similarity between them; , Both represent RGB cores; ② go through Normalize, then weight, to obtain As the affinity core of the pixel adaptive module, the formula is expressed as follows: ； In the formula, Indicates a specific location and another location The affinity nucleus between them; Both represent RGB cores; ③ Through the affinity kernel of the pixel adaptive module, both the initial CAM and the propagated CAM can be refined simultaneously; Refinement is achieved through multiple iterative processes. , In the In the next iteration: ； In the formula, Indicates the position after the t-th iteration. The CAM value of the c-th channel. Indicates the position after t-1 iterations. The CAM value of the c-th channel.

4. The Transformer weakly supervised semantic segmentation method combining contextual attention as described in claim 1, characterized in that, In step three, the loss function of the network model : Step (3.1): Randomly crop a fixed number of local images from the specified uncertain regions and background regions in the input image. Assign these local images as positive or negative samples, with each positive or negative sample forming an image pair. Obtain the category scores for the positive and negative samples in each image pair. , : Step (3.2): Obtain the loss of the common attention module. ; The cropped image pairs obtained in step (3.1) are input into the Transformer encoder to extract the feature maps of each image. These feature maps are then fed into a fully convolutional layer of the common attention module to obtain class-aware activation maps. After obtaining the feature maps of each image in the image pair, the affinity matrix between the two images in the image pair is calculated to obtain the common attention features. ; Step (3.3): Obtain the contrastive attention module loss. ; Using the common attention features obtained in step (3.2) as input, the shared semantic information in the common attention features is enhanced through a convolutional layer and a sigmoid function operation of the contrastive attention module, resulting in enhanced shared semantic information. Then, the contrastive attention matrix is ​​obtained, which inversely highlights the non-common semantic regions outside the shared semantics. The loss in this step is denoted as... ; Step (3.4): Set the overall loss function for a suitable network model; Overall loss of the network model: classification loss Similarity loss and contextual attention loss composition: 1) For classification loss, the aggregated features are fed into the MLP classification layer to calculate the class probability vector, and then multi-label soft margin loss is used as the classification function. The calculation formula is as follows: ； In the formula, C represents the number of categories; c represents the category index, ranging from 1 to C; This represents the tag of class c in the actual tag; This represents the probability that a sample from the model's pre-training belongs to class c; 2) For similarity loss The calculation formula is as follows: ； In the formula and They represent The set of positive and negative samples in the sample. and Used for calculation and The number of; 3) Overall loss function The formula is expressed as follows: ； In the formula, , They represent the functions used for regulation. Loss items and Hyperparameters of the weights of the loss term.

5. The Transformer weakly supervised semantic segmentation method combining contextual attention as described in claim 4, characterized in that, The steps (3.1) are as follows: First, sample images from the cropped image. and , and This represents the category attributes present in the image; subsequently, the feature maps of each image data are extracted in the Transformer encoder. and After the feature maps are extracted, they are fed into a fully convolutional layer of the common attention module. , obtain class activation mapping and Then, after global average pooling, we obtain... and Category score , .

6. The Transformer weakly supervised semantic segmentation method combining contextual attention as described in claim 5, characterized in that, The specific steps of step (3.2) are as follows: 1) First, calculate the feature map. and Affinity matrix between them: First and Lay flat Size, Represents the number of channels. The spatial dimension is calculated using the formula below. and Similarity between them; ； Here, It is The learnable matrix, each element in the affinity matrix P All represent The i-th element and Similarity between the j-th elements; 2) Then, perform row and column operations on the affinity matrix P respectively. Operation, obtain and Normalized attention map for each element and and Normalized attention map for each element : ； ； 3) Next, calculate right Attention, as shown in the following formula: ； ； 4) Therefore, the cross-entropy loss function can be used. The loss of the common attention module is obtained. for: 。 7. The Transformer weakly supervised semantic segmentation method combining contextual attention as described in claim 6, characterized in that, The specific steps of step (3.3) are as follows: The input to the contrastive attention module is the common attention features extracted by the common attention module. and First, we compare the attention module through convolution operations and... Functions enhance shared semantic information in common attention features: ； ； Subtract 1 The contrastive attention matrix is ​​obtained as follows: ； ； get and Then, calculate the contrastive attention features: ； ； After obtaining the contrastive attention features, the contrastive attention loss function can be obtained as follows: ； ； in, It is the loss of the shared attention module. It compares the loss of the attention module. It represents the loss of the entire context attention module.

Images