A weakly supervised semantic segmentation method based on high salient feature step-by-step inactivation

Abstract

The application discloses a weakly supervised semantic segmentation method based on high salient feature step-by-step inactivation, and belongs to the field of computer vision. Firstly, an image is divided into three different scales, including a classification network and a segmentation network. The classification network inputs a medium scale image, and a function of the classification network is to generate a CAM graph. In a training process, an improved PDA method is used to reduce the activation value of a high salient feature in the CAM, so as to reduce the contribution of the most discriminative region, make the classification network pay more attention to other regions, and meanwhile not ignore the features of the most discriminative region. Then, classification is performed through a weighted feature graph, so that the salient region of the CAM graph generated by the classification network is expanded, a more accurate CAM graph is obtained, and finally the output of the classification network is the optimized CAM graph. The input of the segmentation network is three different scale images, and the segmentation network is based on a U-Net network and is composed of 3 encoders and 1 decoder. In a decoding stage, the network fuses the features of the three scales, and better segmentation performance is obtained.

Description

Technical Field

[0001] This invention relates to a weakly supervised semantic segmentation method based on the gradual deactivation of highly salient features, belonging to the field of computer vision. Background Technology

[0002] Gigabit-level images typically have 10 pixels. 5 ×10 5 The size of these gigabyte-level images is on the order of magnitude, with each image exceeding 1GB in size, and some even reaching 3-5GB. Manipulating and processing them incurs enormous computational costs, making semantic segmentation using gigabyte-level images a challenging task. Furthermore, drawing pixel-level labels on gigabyte-level images is extremely expensive and time-consuming. Therefore, we employ a weakly supervised semantic segmentation method based on the gradual deactivation of highly saliency features to segment gigabyte-level images.

[0003] Currently, for gigapixel images, a common semantic segmentation method is to first set a sliding window on the image, cropping it into multiple slices, and then using the resulting slices to train a segmentation network. For example, Ronneberger et al. [1] Semantic segmentation of sliced ​​images using a U-Net-based segmentation method achieved good results. Furthermore, since drawing pixel-level labels for gigapixel images is too costly, weakly supervised semantic segmentation using image-level labels is one solution. Weakly supervised segmentation methods using image-level labels require a good class activation map (CAM); therefore, some studies have optimized the initially generated CAM using different strategies to obtain segmentation results. [2,3] Currently, few studies use weakly supervised algorithms based on class activation mappings for segmenting gigapixel images, among which Chan et al.'s work stands out. [4] The Grad-weighted Class Activation Mapping (Grad-CAM) method was used to generate initial segmentation results, and a fully connected conditional random field was employed to refine these results. However, the results indicate that the accuracy needs improvement. On one hand, the sliding window-based method does not consider the contextual relationships of the image. On the other hand, in the CAM map-based method, the salient regions of the CAM map generated by the classification network gradually shrink to the most discriminative regions as the number of training iterations increases. This results in a significant difference between the pseudo-mask converted from the CAM map and the true mask, leading to low accuracy of the segmentation network trained using the pseudo-mask.

[0004] When observing gigapixel images, people often adjust the image to different magnifications to observe the target and draw conclusions by integrating information from different scales. To conform to this observation logic, this invention utilizes global and local information to supervise segmentation, inputting images of different scales into the segmentation network. Furthermore, because classification networks focus more on discriminative features, salient regions gradually shrink. To address this issue and make the CAM (Camera Map) generated by the classification network closer to the true value, this invention reduces the activation values ​​of salient regions in the activation map, prompting the classification network to focus more on non-dominant regions, thus generating a more accurate CAM and further obtaining a more precise pseudomask.

[0005] References:

[0006] [1]Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." Medical ImageComputing and Computer-Assisted Intervention–MICCAI 2015:18th InternationalConference,Munich,Germany,October 5-9,2015,Proceedings,Part III 18.SpringerInternational Publishing,2015.

[0007] [2]Chang, Yu-Ting, et al. "Weakly-supervised semantic segmentation via sub-category exploration." Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2020.

[0008] [3] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEEinternational conference on computer vision. 2017.

[0009] [4]Chan, Lyndon, et al. "Histosegnet: Semantic segmentation of histological tissue type in whole slide images." Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2019. Summary of the Invention

[0010] Existing semantic segmentation methods for gigabit-level images slice the image using a sliding window and train the segmentation network separately. However, individual slices cannot provide complete semantic information. To address this issue, this invention designs a multi-scale segmentation network architecture that takes three images of different scales as input to provide sufficient semantic information. The largest-scale image provides global background information; the medium-scale image provides structural information and serves as the image to be segmented; and the small-scale image provides detailed information. The features from these three scales are then fused to ensure full utilization of features from different scales. Furthermore, pixel-level annotation of gigabit-level images is very time-consuming and labor-intensive. Therefore, this invention employs a weakly supervised segmentation method, using the CAM image generated by the classification network as a pseudomask for training the segmentation network. However, because the salient regions in the CAM image generated by the classification network shrink to contain only the most discriminative features, the performance of the segmentation network is significantly affected when using the CAM image as a pseudomask. To improve this situation, this invention optimizes the CAM image by reducing the activation values ​​of salient regions, prompting the classification network to focus more on non-dominant regions.

[0011] This invention employs the following technical solution: First, a gigapixel image is divided into three different scales: the large-scale image is defined as the entire image; the slice extracted using a sliding window, i.e., the region to be segmented, is used as the medium-scale image; the small-scale image is obtained by cropping the medium-scale image, with a size half that of the medium-scale image, and the medium and small-scale images share the same center. This method consists of two networks: a classification network and a segmentation network. The classification network takes the medium-scale image as input and generates a CAM (Camspective Mapping Image). During training, an improved Progressive DropoutAttention (PDA) method is used to reduce the activation values ​​of highly salient features in the CAM, thereby reducing the contribution of the most discriminative regions. This allows the classification network to focus on other regions without ignoring the features of the most discriminative regions. Then, the weighted feature map is used for classification, thereby expanding the salient region of the CAM generated by the classification network, resulting in a more accurate CAM. The final output of the classification network is the optimized CAM. The segmentation network takes three images of different scales as input and is based on the U-Net network, consisting of three encoders and one decoder. In the decoding stage, the network fuses the features from the three scales, utilizing the contextual information provided by the large and small-scale images to achieve better segmentation performance. The output of the segmentation network is the final segmentation result, and its specific network structure is shown in the attached figure. Figure 1 As shown. Segmentation results and CAM Figure 2 The valued pseudomask is used to optimize the segmentation network parameters by calculating the loss and backpropagating.

[0012] The specific solution of the present invention is attached. Figure 2 As shown.

[0013] Step 1: Image Preprocessing

[0014] The main task of image preprocessing is to crop each image in the dataset into three images of different resolutions and unify the size of these three images. The dataset referenced in this invention is gigapixel data, such as publicly available breast pathology slide image datasets and remote sensing image datasets. This step first crops the image, using the original image as the large-scale image; then, a sliding window is used to select the image to be segmented as the medium-scale image; within the medium-scale image, a region half the size of the original image and having the same center as the medium-scale image is selected as the small-scale image. Then, the large-scale and small-scale images are unified to the same size as the medium-scale image through downsampling and upsampling, respectively.

[0015] Step 2: Obtain the class activation graph

[0016] This step trains a classification network for medium-scale images and then uses the Grad-CAM algorithm to generate CAM maps.

[0017] Step 2.1 Training the classification network: This invention uses the classic convolutional neural network RseNet101 as the classification network and uses image-level classification labels as supervision information to train the network to predict category information (e.g., predicting whether the target in the remote sensing dataset is an airplane, ship, or car).

[0018] Step 2.2 Calculate the CAM image: This step uses the Grad-CAM algorithm to generate the CAM image. First, we extract the feature map of the last convolutional layer of the classification network from Step 2.1; then, we backpropagate the predicted scores for each predicted category to obtain the gradient information of the predicted scores relative to the output features of the last convolutional layer; next, we average the gradient information to obtain the weight of each channel of the last layer's features for a given category; finally, we sum the weights and feature layers by weight and then activate them using ReLU (Rectified Linear Unit) to obtain the CAM image of the input image.

[0019] Step 3: Optimize the classification network

[0020] This step is performed during the training phase of the classification network. First, this step obtains an attention feature map by reducing the activation values ​​of salient regions in the CAM map generated by the classification network. Then, it uses this attention feature map as weights on the feature map output by the last convolutional layer of the classification network to prevent the model from over-focusing on the most obvious and discriminative regions, thus providing a more comprehensive feature representation. Finally, the weighted feature map is used to make predictions, and the network is trained based on the true classification labels to optimize the network parameters.

[0021] Step 3.1 Reduce the activation values ​​of salient regions in the CAM graph: This step uses an improved PDA algorithm on the CAM to reduce the activation values ​​of salient regions. To make the network pay more attention to regions with non-salient features, this invention sets a threshold and reduces the values ​​of regions with confidence scores greater than the threshold on the CAM graph, thereby reducing the weight of the corresponding regions on the feature map, instead of directly setting the values ​​of regions with confidence scores greater than the threshold to zero. This allows the network to pay more attention to non-dominant regions without ignoring regions with highly significant features. Furthermore, to prevent the salient regions in the CAM graph from gradually shrinking with increasing iterations, this invention gradually reduces the threshold to expand the region whose activation values ​​need to be reduced.

[0022] Step 3.2 Generate Attention Feature Map: For the new CAM map generated in Step 3.1, this step sums up the CAM maps corresponding to different categories and calculates the average value to obtain the attention feature map.

[0023] Step 3.3 Optimize Network Parameters: This step first multiplies the attention feature map generated in Step 3.2 with the feature map output from the last convolutional layer of the classification network to obtain a weighted feature map. Then, this feature map is used for subsequent class prediction. Finally, the loss is calculated with the true class label and backpropagated to update the network parameters. By predicting class information using a weighted feature map, we make the classification network pay more attention to information outside the salient regions. Therefore, the CAM map generated by the optimized network will be closer to the true value, rather than generating a CAM map that only contains the most salient features within highly salient features.

[0024] Step 4: Train and predict the multi-scale segmentation network

[0025] In this step, we first use the classification network trained in step 3 to generate CAM images of medium-scale images, and convert them into binary images as pseudomasks for training the segmentation network. Then, we input the three images of different scales generated in step 1 into the three encoders of the segmentation network, and in the decoding stage, we perform feature fusion and splicing through skip connections and attention fusion modules to output the semantic segmentation results of medium-scale images. Finally, we use the trained segmentation network to perform segmentation prediction on the test set.

[0026] Step 4.1 Generating a pseudo-mask: This step first inputs the image to be segmented, i.e., the mesoscale image, into the trained classification network to obtain the CAM image corresponding to the mesoscale image; then, it selects the class value corresponding to the maximum value at the same pixel position in the CAM images of each class as the value at that position in the final CAM; finally, it uses this CAM image as the CAM image. Figure 2 Value-enhanced pseudomasks are used to train the segmentation network.

[0027] Step 4.2 Obtaining Multi-Scale Features: We input the three images at different scales obtained in Step 1 into three different encoders of the segmentation network to obtain three sets of features at different scales. The three encoders are U-Net network architecture encoders, which consist of convolution and downsampling.

[0028] Step 4.3 Feature Fusion: This step is performed during the decoding stage of the segmentation network. We first upsample the output of the previous convolutional layer, and then use a skip connection structure to input the upsampled features and the features of the corresponding layers in the three encoders into the attention fusion module for fusion. The fused features are then input into the convolutional layer, and so on, layer by layer, to finally obtain the segmentation result.

[0029] Step 4.4 Segmentation Network Training and Prediction: This step uses the pseudomask obtained in Step 4.1 to train the segmentation network, and then uses the trained segmentation network to segment the test set.

[0030] Compared with existing technologies, the advantages of this invention are as follows:

[0031] I. The weakly supervised semantic segmentation method based on the gradual deactivation of highly salient features adopted in this invention effectively avoids the pixel-level labels required for fully supervised learning, reduces the required standards for supervision information, and greatly reduces the workload of data annotation.

[0032] Second, the weakly supervised semantic segmentation method based on the gradual deactivation of highly salient features adopted in this invention uses an improved PDA method for CAM, which makes the classification network pay more attention to the non-salient regions of the image, while not ignoring the most discriminative regions, preventing the salient regions of the CAM map from shrinking, and thus obtaining a pseudo-mask that is closer to the true value.

[0033] Third, the weakly supervised semantic segmentation method based on the gradual deactivation of highly salient features adopted in this invention divides the image into three different scales: large, medium, and small. Feature maps are then stitched and fused in the segmentation network, enabling the segmentation network to obtain as much semantic information as possible. This includes global background information provided by large-scale images; structural information provided by medium-scale images, which are used as segmentation objects; and detailed information provided by small-scale images, thereby obtaining more accurate segmentation results. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the network structure of the segmentation network of the present invention.

[0035] Figure 2 This is an overall flowchart of the method proposed in this invention. Detailed Implementation

[0036] The following detailed description of embodiments of the present invention is provided in conjunction with the accompanying drawings:

[0037] This invention is a weakly supervised semantic segmentation method based on progressive deactivation of highly saliency features, consisting of four stages: The first stage is image preprocessing, where all images in the dataset are divided into three different scales, and then upsampling or downsampling is used to unify the images of different scales to the same size. The second stage is CAM image generation. This stage mainly includes training the classification network and the calculation method for extracting CAM from the classification network. The third stage is optimizing the classification network. During the training of the classification network, this stage first reduces the values ​​of high-confidence regions in the CAM image using an improved PDA method; then, the various CAM images are fused to obtain an attention feature map; next, the attention feature map is multiplied with the feature map output by the last convolutional layer of the classification network to obtain a weighted feature map, which is then used for subsequent prediction to obtain the class score; finally, the classification network parameters are optimized using the real class labels to make the CAM image generated by the classification network more accurate. The fourth stage is the training and prediction of the segmentation network. This stage first uses a classification network to generate a CAM image of a medium-scale image, and binarizes it as a pseudo-mask for supervised training of the segmentation network. Then, the three preprocessed images of different scales are input into the segmentation network, and features of different scales are obtained through three encoders. Next, in the decoding stage, skip connections and attention fusion modules are used to fuse the features of different scales, and finally output the prediction results. Finally, the trained segmentation network is used to segment and predict gigabit-level pixel images.

[0038] Specifically, the method includes the following steps:

[0039] Step 1: Image Preprocessing

[0040] The purpose of this step is to crop the original image into three images with different resolutions and to unify the size of all images in the dataset. The large-scale image p... large Original image; mesoscale image p mid It is a 512*512 image cropped using a sliding window; small-scale image p littleThe images are cropped from mesoscale images, half the size of the mesoscale images, and share the same center. The data referenced in this invention primarily consists of gigapixel image data, such as publicly available breast pathology slide image datasets and remote sensing image datasets. Different images in the datasets, as well as the three different scale images obtained after cropping, have different resolutions. Since the fixed parameters in the classification and segmentation networks cannot adapt to different image sizes, the size of all input images must be standardized. To reduce the loss of image information and prevent excessive subsequent computational costs, the size of all images is set to 512*512. It should be noted that during the downsampling of large-scale images, because the large-scale images are used to provide background information, the loss of some information will not significantly affect the segmentation results.

[0041] Step 2: Obtain the class activation graph

[0042] This step trains a classification network for the input image to be segmented, i.e., the mesoscale image, and then uses the Grad-CAM algorithm to obtain the CAM map of the mesoscale image.

[0043] Step 2.1 Training the classification network: This invention uses the classic convolutional neural network RseNet101 as the classification network, which contains 100 convolutional layers and 1 fully connected layer. This invention trains a classification network, denoted as Cls, on the input mesoscale image based on the class labels.

[0044] Step 2.2 Calculate the CAM map: This step uses the Grad-CAM algorithm to generate the CAM map. For the classification network Cls, we first extract the feature map f output by its last convolutional layer, and then backpropagate the prediction scores of each predicted class to obtain the gradient information g of the prediction score relative to f using formula (1).

[0045]

[0046] Where y c This represents the score predicted by the network for class c, which is the score without softmax activation. The gradient information g is the result of y... c The partial derivative of f with respect to y represents the derivative of each element in f with respect to y. c The contribution of this gradient information g. Next, we average the gradient information g over width w and height h to obtain the weights of channel k corresponding to feature f for class c. It is obtained from formula (2).

[0047]

[0048] Where g k(i, j) represents the gradient information at coordinates (i, j) in channel k, and Z equals the width w multiplied by the height h. Finally, we will assign weights... The input image's CAM map M with respect to category c is obtained by weighted summation of the feature layer f and then ReLU activation, according to formula (3). c .

[0049]

[0050] Where f k This represents the data of the k-th channel of the feature map f. ReLU is the ReLU activation function, which filters out negative activation values.

[0051] Step 3: Optimize the classification network

[0052] This step is performed during the training phase of the classification network. First, this step applies an improved PDA algorithm to the CAM map calculated in step 2.2 to reduce the activation values ​​of its salient regions, thus obtaining an attention feature map. Then, this attention feature map is used as weights on the feature map output by the last convolutional layer of the classification network. Finally, the weighted feature map is input into a fully connected layer to obtain the classification result. The classification network is trained using real classification labels, thereby optimizing its parameters and making the model focus more on information beyond the most discriminative feature regions, enabling the generation of more comprehensive feature representations.

[0053] Step 3.1 Reduce the activation values ​​of salient regions in the CAM image: This step affects the CAM image M c An improved PDA algorithm is used to reduce the activation values ​​of salient regions in the CAM. First, the algorithm sets a threshold β, reducing the activation values ​​of regions in the CAM map that are above the threshold to half their original value, resulting in a new CAM map M′ corresponding to category c. c As shown in formula (4).

[0054]

[0055] Where M c (i, j) represents the value at coordinate (i, j) in the CAM graph corresponding to category c. β is a relative threshold that varies with the CAM graph M. c The value of β changes with the change of the maximum value, and β can be obtained by formula (5).

[0056] β=θ*max(M c )#(5)

[0057] Here, θ is not a fixed value; it gradually decreases with the increase of the number of iterations until it decreases to a pre-given lower bound l. This achieves the effect that the threshold β gradually decreases with the number of iterations. The value of θ is obtained from formula (6).

[0058]

[0059] Where ω represents the attenuation rate of θ, which is a fixed value, taken as ω = 0.985. t Let θ represent the value of θ in the t-th iteration, with an initial value of θ1 of 1. The lower bound l is given in advance as 0.65.

[0060] Step 3.2 Generate attention feature map: For the newly obtained CAM map M′ in step 3.1, this step sums up the CAM maps corresponding to different categories and calculates the average value, and obtains the attention feature map A according to formula (7).

[0061]

[0062] Where N represents the number of all categories. M′ c (i, j) represents the value at coordinate (i, j) in the new CAM diagram for category c.

[0063] Step 3.3 Optimize network parameters: As shown in formula (8), we multiply the attention feature map A generated in step 3.2 with the feature map f output by the last convolutional layer of the classification network Cls to obtain the weighted feature map.

[0064]

[0065] Then we will use the feature map The input is fed into the fully connected layer of the classification network for subsequent class prediction. Finally, we calculate the loss using the predicted information and the true class labels, and then backpropagate to update the network parameters.

[0066] Step 4: Train and predict the multi-scale segmentation network

[0067] This step first uses the classification network trained in step 3 to generate a CAM image of the medium-scale image, and converts it into a binary image as a pseudo-mask for training the segmentation network. Then, the three images of different scales generated in step 1 are input into the three encoders of the segmentation network, and feature fusion and stitching are performed through skip connections and attention fusion modules in the decoding stage to finally output the segmentation result of the medium-scale image. Finally, the trained segmentation network is used to perform segmentation prediction on the test set.

[0068] Step 4.1 Generate a pseudo-mask: This step first generates a pseudo-mask from the image to be segmented, i.e., the medium-scale image p. mid The input is fed into the trained classification network CLS to obtain the CAM map corresponding to the mesoscale image; then, the class value corresponding to the maximum value at the same pixel position in the CAM map of each class is selected as the value at that position in the final CAM, and the final class activation map M is obtained. finalThe calculation formula is shown in formula (9).

[0069] M final =argmax(M 1 M 2 M N )#(9)

[0070] The `argmax` parameter compares the values ​​at the same coordinates across all CAMs and returns the index of the maximum value, which corresponds to the category value. Finally, the resulting M... final This is transformed into a binary pseudomask for training the segmentation network to segment category c. The values ​​of the coordinates (i, j) are determined by formula (10).

[0071]

[0072] Where c represents the category to be segmented.

[0073] Step 4.2 Obtaining Multi-Scale Features: We will obtain three images p at different scales from Step 1. large p mid p little The inputs are fed into three different encoders E1, E2, and E3 of the segmentation network, respectively, to obtain three sets of features f at different scales. large f mid f little The three encoders are structured similarly to the encoders in the U-Net network, consisting of convolutional layers and downsampling layers.

[0074] Step 4.3 Feature Fusion: This step is performed during the decoding stage of the segmentation network. We first fuse the features from the previous convolutional layer... The output is upsampled, and the upsampled result is obtained by formula (11). Then, using a skip connection structure, the upsampled result is... and features of the corresponding layers in the three encoders The input is fused into the attention fusion module, and the fused feature map is then processed. It is obtained from formula (12).

[0075]

[0076]

[0077] UP stands for upsampling operation. This represents the output of the (i-1)th convolutional layer. att For the attention fusion module, its feature fusion process is as follows: First, the features output by the three different encoders are combined... The splicing is performed, and the splicing result is obtained by formula (13).

[0078]

[0079] in The first step is the feature map concatenation operation, which merges the feature maps along the channel dimension. Next, the attention fusion module performs convolution, pooling, and activation operations on the concatenated feature maps, resulting in an attention map using formula (14).

[0080]

[0081] Here, W represents a series of operations including convolution, pooling, and activation. Finally, an attention graph is used. For mesoscale features Weighting is applied, and the result is compared with the upsampled features. By splicing the components, the attention fusion module Block is obtained from formula (15). att The final output.

[0082]

[0083] The fused features are then passed through a convolutional layer to obtain the output of the i-th layer. This process is repeated layer by layer upwards until the final segmentation result is obtained.

[0084] Step 4.4 Segmentation Network Training and Prediction: This step first uses the pseudomask obtained in Step 4.1 to train the segmentation network, and then uses the trained segmentation network to segment the test set.

Claims

1. A weakly supervised semantic segmentation method based on the gradual deactivation of highly saliency features, characterized in that: Step 1: Image Preprocessing The original image was cropped into three images with different resolutions, and the size of all images in the dataset was standardized; among them, the large-scale image... Original image; mesoscale image It is a 512x512 pixel image cropped using a sliding window; a small-scale image. It was cropped from a mesoscale image, and all images were sized to 512*512. Step 2: Obtain the class activation graph This step trains a classification network for the input image to be segmented, i.e., the mesoscale image, and then uses the Grad-CAM algorithm to obtain the CAM image of the mesoscale image; Step 2.1 Training the classification network: The classic convolutional neural network RseNet101 is used as the classification network, containing 100 convolutional layers and 1 fully connected layer; a classification network is trained on the input mesoscale image based on the class labels, denoted as... ; Step 2.2 Calculate the CAM map: This step uses the Grad-CAM algorithm to generate the CAM map; for classification networks... First, extract the feature map output from its last convolutional layer. By backpropagating the predicted scores for each predicted category, the relative predicted scores can be obtained from formula (1). gradient information ; ； in The network represents the category The predicted score, here is the score without softmax activation; gradient information. yes right The partial derivative represents Each element pair The contribution of gradient information; then the gradient information The average value over width w and height h is obtained for the category. In terms of characteristics Corresponding channel weight This is obtained from formula (2); ； in Represents gradient information in the channel The median coordinate is The value at that location, It equals the width w multiplied by the height h; finally, the weight is... and feature layer Weighted summation followed by ReLU activation yields the CAM map of the input image with respect to category c, obtained by formula (3). ; ； in Representative feature map No. Data from each channel; ReLU is the ReLU activation function, which can filter out negative activation values; Step 3: Optimize the classification network This step is performed during the training phase of the classification network. First, this step uses an improved PDA algorithm on the CAM map calculated in step 2.2 to reduce the activation values ​​of its salient regions, thereby obtaining an attention feature map. Then, the attention feature map is used as weights on the feature map output by the last convolutional layer of the classification network. Finally, the weighted feature map is input into the fully connected layer to obtain the classification result. The classification network is trained using real classification labels, thereby optimizing the parameters of the classification network so that the model pays more attention to information outside the most discriminative feature regions and can generate more comprehensive feature representations. Step 3.1 Reduce the activation values ​​of salient regions in the CAM diagram: This step affects the CAM diagram. An improved PDA algorithm is used to reduce the activation values ​​of salient regions in CAM; firstly, the algorithm sets a threshold of... The activation values ​​of regions in the CAM image above a threshold are reduced to half their original values ​​to obtain the category. The corresponding new CAM diagram As shown in formula (4); ； in Indicates category Corresponding coordinates in the CAM drawing The value at; It is a threshold that varies with the CAM map. It changes with the change of its maximum value. Obtained from formula (5); ； in It is not a fixed value; it gradually decreases with each iteration until it reaches a pre-defined lower bound. Thus reaching the threshold As the number of iterations gradually decreases, the effect is... The value is obtained from formula (6); ； in represent The attenuation rate is a fixed value, taken as... Indicates the first During the next iteration The value of , its initial value =1; a pre-defined lower bound Take 0.65; Step 3.2 Generate Attention Feature Map: Based on the newly obtained CAM map in Step 3.1 This step sums the CAM maps corresponding to different categories and calculates the average value, and obtains the attention feature map according to formula (7). ; ； in Represents the number of all categories; Representative category Coordinates in the new CAM drawing The value at; Step 3.3 Optimize network parameters: As shown in formula (8), optimize the attention feature map generated in step 3.

2. With classification networks Feature map output by the last convolutional layer Multiply to obtain the weighted feature map. ; ； Then feature map The input is fed into the fully connected layer of the classification network for subsequent class prediction; finally, the predicted information is compared with the true class label to calculate the loss and backpropagation is used to update the network parameters. Step 4: Train and predict the multi-scale segmentation network This step first uses the classification network trained in step 3 to generate a CAM image of the medium-scale image, and converts it into a binary image as a pseudo-mask for training the segmentation network; then, the three images of different scales generated in step 1 are input into the three encoders of the segmentation network, and feature fusion and stitching are performed through skip connections and attention fusion modules in the decoding stage, finally outputting the segmentation result of the medium-scale image; finally, the trained segmentation network is used to perform segmentation prediction on the test set. Step 4.1 Generate a pseudo-mask: This step first generates a pseudo-mask from the image to be segmented, i.e., the medium-scale image. Input into the trained classification network The CAM map corresponding to the mesoscale image is obtained; then, the class value corresponding to the maximum value at the same pixel position in the CAM map of each class is selected as the value at that position in the final CAM, resulting in the final class activation map. The calculation formula is shown in formula (9); ； in Its function is to compare the values ​​at the same coordinates in all CAMs and return the index, i.e., the category value, corresponding to the maximum value; finally, the obtained value is... Transform into segmentation categories for training the segmentation network Binary pseudomask; pseudomask median coordinate The value of is determined by formula (10). ； in The categories to be segmented; Step 4.2 Obtain multi-scale features: Combine the three images obtained in Step 1 at different scales. , , Three different encoders are input into the segmentation network. In the process, three sets of features at different scales were obtained. , , The three encoders are structured similarly to the encoders in the U-Net network, consisting of convolutional layers and downsampling layers. Step 4.3 Feature Fusion: This step is performed during the decoding stage of the segmentation network. First, the feature fusion of the previous convolutional layer... The output is upsampled, and the upsampled result is obtained by formula (11). ; Then, using a skip connection structure, the upsampled result is... and features of the corresponding layers in the three encoders The input is fused into the attention fusion module, and the fused feature map is then processed. It is obtained from formula (12); ； ； in This is an upsampling operation; Representing the The output of the convolutional layer; For the attention fusion module, its feature fusion process is as follows: First, the features output by the three different encoders are combined... The splicing is performed, and the splicing result is obtained by formula (13). ; ； in The first step is the feature map concatenation operation, which merges the feature maps along the channel dimension. Then, the attention fusion module performs convolution, pooling, and activation operations on the concatenated feature maps to obtain an attention map using formula (14). ; ； Where W represents a series of operations including convolution, pooling, and activation; finally, an attention graph is used. For mesoscale features Weighting is applied, and the result is compared with the upsampled features. By splicing the components, the attention fusion module is obtained from formula (15). The final output; ； The fused features are then passed through a convolutional layer to obtain the first... Layer output This process is repeated layer by layer upwards until the final segmentation result is obtained; Step 4.4 Segmentation Network Training and Prediction: This step first uses the pseudomask obtained in Step 4.1 to train the segmentation network, and then uses the trained segmentation network to segment the test set.

Images