A Weakly Supervised Image Target Localization Method Based on Multi-Scale Satisfactory Feature Fusion

Abstract

This invention relates to a weakly supervised image target localization method based on multi-scale salient feature fusion, belonging to the field of computer vision. To address the problems of cumbersome ROI annotation and insufficient CAM activation in small target images, this invention focuses on optimizing the output class activation map of a classification network under weak supervision. This invention relates to information fusion at two levels: ① Since the feature maps at the lowest level of a convolutional neural network have weak semantic information but strong positional information, they can be fused with the feature maps at the highest level to obtain the final feature map of the classification network; ② Because the classification network has different sensitivities to ROIs at different scales, the resulting class activation maps also differ. Therefore, fusing complementary object information from different activation maps can improve the localization of target regions in the image, thereby generating more accurate pseudo-labels for segmentation tasks.

Description

Technical Field

[0001] This invention relates to a weakly supervised image target localization method based on multi-scale salient feature fusion, belonging to the field of computer vision. Background Technology

[0002] Localizing and segmenting regions of interest (ROIs) in images is a classic challenge in computer vision research. Significant progress has been made in ROI localization and segmentation based on natural images. However, for non-natural images in certain domains (e.g., medical images, pollen grain images), their ROIs are smaller than those in natural images, so ROI localization and segmentation methods based on natural images are not entirely applicable to these types of images. Therefore, research on localizing and segmenting small objects in images based on specific domains is of great significance.

[0003] Currently, the mainstream deep learning-based methods for small object localization and segmentation fall into two categories: fully supervised learning and weakly supervised learning. Z. Ning et al. [1] This paper proposes using the saliency maps of the foreground and background in breast ultrasound images to guide the main and auxiliary networks to learn foreground and background saliency representations respectively, and finally fusing the features of both to enhance the morphological learning ability of the segmentation network. However, this type of fully supervised deep learning method generally requires a large amount of labeled datasets, and obtaining pixel-level labels for images is a complicated and time-consuming task. In contrast, obtaining datasets with only category information is easier. Therefore, many works only use weakly supervised methods such as image-level labels to achieve target localization and segmentation. However, the class activation map (CAM) obtained by the classification network in weakly supervised learning can only cover the most salient part of the image and cannot indicate the complete target region. That is, the localization accuracy of CAM is low (insufficient activation). To address this, Li Y et al. [2] First, prior knowledge of breast anatomy is used to constrain the search space of the classification network for breast lesions, and then the level set algorithm is used to correct the CAM. However, it overlooks an important piece of information: the discriminative regions captured by the classification network are not consistent for targets of different scales.

[0004] To address the issues of cumbersome ROI annotation and insufficient CAM activation in small target images, this invention focuses on optimizing the output class activation maps of classification networks under weak supervision. This invention relates to information fusion at two levels: ① Since the feature maps at the lowest level of a convolutional neural network have weak semantic information but strong positional information, they can be fused with the feature maps at the highest level to obtain the final feature map of the classification network; ② Because classification networks have different sensitivities to ROIs of different scales, the resulting class activation maps also differ. Therefore, fusing complementary object information from different activation maps can improve the localization of target regions in the image, thereby generating more accurate pseudo-labels for segmentation tasks.

[0005] References:

[0006] [1]Z.Ning, S.Zhong, Q.Feng, W.Chen and Y.Zhang, "SMU-Net: Saliency-GuidedMorphology-Aware U-Net for Breast Lesion Segmentation in Ultrasound Image," inIEEE Transactions on Medical Imaging, vol.41, no.2, pp.476-490, Feb.2022, doi:10.1109 / TMI.2021.3116087.

[0007] [2]Li Y, Liu Y, Huang L, Wang Z, Luo J. Deep weakly-supervised breasttumor segmentation in ultrasound images with explicit anatomicalconstraints. Med Image Anal. 2022Feb; 76:102315.doi:10.1016 / j.media.2021.102315.Epub 2021Nov 28.PMID:34902792. Summary of the Invention

[0008] To address the issues of cumbersome annotation work in existing image small target localization and segmentation research based on fully supervised learning, and insufficient CAM activation in single-scale image small target localization and segmentation research based on weakly supervised learning, this invention designs a weakly supervised image target localization method based on multi-scale salient feature fusion. Specifically, we construct an image pyramid to obtain images at three different scales, thereby obtaining multi-scale CAMs for the same image, then fuse them, and finally use the fused CAM as weakly supervised information to train the segmentation network.

[0009] The weakly supervised image target localization method based on multi-scale salient feature fusion described in this invention consists of five stages: The first stage is image preprocessing, mainly unifying the resolution of images in the dataset. The second stage is the construction of the image pyramid. This stage mainly includes three parts: downsampling from the input image as the source image to construct the top layer of the image pyramid, upsampling to construct the bottom layer of the image pyramid, and determining the final number of layers of the image pyramid. The third stage is the acquisition of classifier feature maps. In this stage, a classifier is first trained for each layer of the image pyramid, and then for any layer of the classifier, the feature maps of the highest and lowest layers are concatenated to obtain the fused feature map. The fourth stage is the fusion of multi-scale CAMs. In this stage, a multi-scale CAM for the same image is obtained by weighted summing of the feature maps of each layer, then all CAMs are aligned and finally fused to obtain the final CAM of the source image. The fifth stage is the prediction of the target region. In this stage, the fused CAM is first converted into a pseudo-binary label, then the pseudo-label is used to train a segmentation network, and finally the target region is predicted by the segmentation network.

[0010] The specific solution of the present invention is attached. Figure 2 As shown.

[0011] Step 1: Image Preprocessing

[0012] The purpose of image preprocessing is to standardize the size of all images within the dataset. The data referenced in this invention mainly consists of small-target image data, such as publicly available breast image datasets and pollen image datasets. If the image resolution within the dataset is inconsistent, the feature map sizes obtained by the subsequent classification network will also be inconsistent. Since the parameters of the fully connected layers in the classification network cannot adapt to feature maps of different sizes, it is necessary to fix the size of all input images to a uniform size.

[0013] Step 2: Image Pyramid Construction

[0014] This step uses images within the dataset as source images and constructs a Gaussian pyramid to obtain three scale transformations of the input image. To simultaneously obtain more global and finer-grained information than the original image, the Gaussian pyramid constructed in this invention adopts a pyramid structure that combines downsampling and upsampling.

[0015] Step 2.1 Image Pyramid Top Layer Construction: Using the input image as the source image, firstly, Gaussian smoothing is performed on it using a 5*5 template Gaussian kernel. Then, the processed image is downsampled by removing even-numbered rows and columns from the image matrix. Finally, an image of 1 / 4 the size of the input image is obtained, and this is used as the top layer of the image pyramid.

[0016] Step 2.2 Image Pyramid Bottom Layer Construction: Using the input image as the source image, the image is first enlarged to twice its original size in each direction, with the newly added rows and columns filled with the value 0. Then, a 5*5 Gaussian kernel template is multiplied by 4 and convolved with the enlarged image to obtain approximate values ​​for the newly added pixels. Finally, an image four times the size of the input image is obtained, and this is used as the bottom layer of the image pyramid.

[0017] Step 2.3 Determining the Image Pyramid Layers: Images on different layers of the image pyramid are assigned numbers, starting from 0. As the number of pyramid layers increases, the image resolution decreases accordingly. The image pyramid constructed in this invention has 3 layers, with the original image at layer 2, corresponding to layer number 1.

[0018] Step 3: Obtaining Classifier Feature Maps

[0019] This step trains a classifier for each of the three different scales of images in the image pyramid to obtain class activation maps for the same image at the three different scales.

[0020] Step 3.1 Classification Network Training: This invention uses the classic ResNet50 as the classification network to determine the category of the input image. Since there are three different scales of images in the image pyramid, a classifier needs to be trained for each of the three different scale image datasets.

[0021] Step 3.2 Fusion of High- and Low-Level Feature Maps: For each classification network, the shallow receptive field is small, extracting low-level geometric information such as texture and edges; while the high-level receptive field is large, extracting more global and deeper semantic information. Therefore, this invention aligns and concatenates the highest-level features and lowest-level features of each classification network, prompting the network to enhance the low-level features of small target objects to obtain the final fused feature map of the network.

[0022] Step 4: Multi-scale CAM fusion

[0023] This step obtains the CAMs of the three classification networks, aligns them, and then fuses them to finally obtain the fused CAM image corresponding to the image.

[0024] Step 4.1 Classification Network CAM Acquisition: For the final fused feature map obtained in Step 3.2, the CAM is obtained by multiplying it with the weight matrix of the fully connected layer in the classification network. Since this invention uses three classification networks, for each source image, three CAMs of different scales will be obtained, forming a CAM pyramid.

[0025] Step 4.2 Alignment of Multiple CAMs: Align CAMs of different scales based on the size of the source image to facilitate subsequent fusion operations.

[0026] Step 4.3 Multiple CAM Fusion: For any pixel in the fused CAM, the present invention adopts the following judgment mechanism: if at least two independent CAMs have activation values ​​of a certain category at this point that are greater than or equal to a threshold, then the pixel is considered to belong to that category. If the pixel is not assigned to any category after the judgment mechanism, then the pixel is ignored; if the pixel is assigned to multiple categories, then the pixel is assigned to the category corresponding to the maximum average activation value of the three independent CAMs at this point.

[0027] Step 5: ROI Prediction

[0028] This step first converts the fused CAM obtained in step 4.3 into pseudo-labels, then trains a localization and segmentation network for image ROIs based on the pseudo-labels, and finally uses the network to predict ROIs.

[0029] Step 5.1 CAM Fusion and Pseudo-Label Conversion: The fused CAM is converted into a pseudo-binary mask for training the segmentation network. This invention employs the following judgment mechanism: if any pixel in the fused CAM belongs to a non-target class, its pixel value is assigned to 0; otherwise, it is assigned to 1.

[0030] Step 5.2 Segmentation Network Training and Prediction: Based on the pseudo-binary labels obtained in Step 5.1, an image segmentation network is trained. The segmentation network architecture selected in this invention is U-Net. Finally, the trained network is used to perform ROI segmentation prediction on the test set.

[0031] Compared with existing technologies, the advantages of this invention are as follows:

[0032] I. The weakly supervised image target localization method based on multi-scale salient feature fusion adopted in this invention effectively avoids the pixel-level annotation of ROI required under fully supervised learning, which greatly reduces the workload of data annotation.

[0033] Second, the weakly supervised image target localization method based on multi-scale salient feature fusion adopted in this invention concatenates the highest and lowest layer feature maps obtained from each classification network, which enhances the network's learning of low-level features of small target objects, thereby enabling the network to pay attention to more features of small target objects.

[0034] Third, the weakly supervised image target localization method based on multi-scale salient feature fusion adopted in this invention constructs a hybrid pyramid structure of downsampling and upsampling based on the source image, thereby obtaining more global and finer-grained features than the original image. By fusing these features, a more complete CAM is obtained. Attached Figure Description

[0035] Figure 1A schematic diagram of the image pyramid constructed for this invention.

[0036] Figure 2 This is an overall flowchart of the method proposed in this invention. Detailed Implementation

[0037] The following is in conjunction with the instruction manual. Figure 2 The following is a detailed description of embodiments of the present invention:

[0038] The weakly supervised image target localization method based on multi-scale salient feature fusion described in this invention consists of five stages: The first stage is image preprocessing, mainly to unify the resolution of the dataset. The second stage is the construction of the image pyramid. This stage mainly includes three parts: downsampling the input image as the source image to construct the top layer of the image pyramid, upsampling to construct the bottom layer of the image pyramid, and determining the final number of layers of the image pyramid. The third stage is the acquisition of classifier feature maps. In this stage, a classifier is first trained for each layer of the image pyramid, and then for any layer of the classifier, the feature maps of the highest and lowest layers are concatenated to obtain the fused feature map. The fourth stage is the fusion of multi-scale CAMs. In this stage, a multi-scale CAM for the same image is obtained by weighted summing of the feature maps of each layer, then all CAMs are aligned and finally fused to obtain the final CAM of the source image. The fifth stage is the prediction of ROIs. In this stage, the fused CAM is first converted into pseudo-binary labels, then the pseudo-labels are used to train a segmentation network, and finally the segmentation network is used to predict ROIs.

[0039] Specifically, the method includes the following steps:

[0040] Step 1: Image Preprocessing

[0041] The purpose of image preprocessing is to standardize the size of all images within the dataset. The data referenced in this invention mainly consists of small-target image data, such as publicly available breast image datasets and pollen image datasets. If the image resolution in the dataset is not unique, the feature map sizes obtained by the last convolutional layer of the subsequent classification network will be inconsistent. Since the parameter dimensions of the fully connected layer and the previous layer are fixed in advance, the parameters of the fully connected layer cannot adapt to different feature map sizes. Therefore, the size of all input images must be fixed. To minimize image information loss and facilitate convolutional operations in the subsequent classification network, we set the size of all images to 512*512.

[0042] Step 2: Image Pyramid Construction

[0043] This step obtains three scale transformations of the input image by constructing a Gaussian pyramid. To simultaneously acquire more global and finer-grained information than the original image, the Gaussian pyramid constructed in this invention employs a hybrid downsampling and upsampling pyramid structure. Specifically, the construction process includes two parts: first, the width and height of the input original image are downsampled to 50% of the original image using the Gaussian pyramid, resulting in a 256*256 resolution image as the top layer of the pyramid; second, the width and height of the input original image are upsampled to 200% of the original image using the Gaussian pyramid, resulting in a 1024*1024 resolution image as the bottom layer of the pyramid.

[0044] Step 2.1 Image Pyramid Top Layer Construction: For a given original image of size 512*512, we downsample to the original size... Figure 1 The top layer of the Gaussian pyramid is constructed from an image of size 4, corresponding to a resolution of 256*256. The specific process is shown in formula (1): First, the original 512*512 image is subjected to Gaussian smoothing. Unlike simple smoothing, Gaussian smoothing assigns higher weight values ​​to pixels near the center point when calculating the weighted average of surrounding pixels. Then, the processed image is downsampled by removing even-numbered rows and columns from the image matrix to obtain an image with a resolution of 256*256.

[0045]

[0046] (1≤l≤L,0≤x≤R l ,0≤y≤C l )

[0047] Among them G l This is an image of the l-th level of the Gaussian Pyramid (the number of levels in the Gaussian Pyramid starts from 0), where L is the level number of the top level of the Gaussian Pyramid, and R... l and C l The numbers are the row and column number of the l-th layer image, respectively. W(m,n) is the value of the m-th row and n-th column of the Gaussian filter template, which is generally 5*5 in size. In this invention, the two-dimensional separable 5*5 Gaussian kernel widely used in the anti-sharpening mask algorithm is selected to smooth the original image. Its value is shown in (2).

[0048]

[0049] Step 2.2 Image Pyramid Bottom Layer Construction: For a given 512*512 original image, we upsample to a size four times that of the original image to construct the lowest layer of the Gaussian pyramid, corresponding to a resolution of 1024*1024. The specific process is as follows: First, the image is enlarged to twice its original size in each direction, with the newly added rows and columns filled with zeros. Then, the Gaussian kernel used in downsampling is multiplied by 4 and convolved with the enlarged image to obtain approximate values ​​for the newly added pixels, ultimately resulting in a 1024*1024 resolution image.

[0050] Step 2.3 Determining the number of layers in the image pyramid: After the image pyramid is constructed, the number of layers l in the Gaussian pyramid corresponding to the image with resolution w*h is determined by formula (3).

[0051]

[0052] Where l0 is the layer number of the 512*512 original image in the image pyramid. Since the three scales of the image in the Gaussian pyramid are 1024, 512, and 256, the layer number corresponding to the original image is l0 = 1. According to formula (3), the 1024*1024 image corresponds to the layer number 0, that is, it is located at the lowest layer of the Gaussian pyramid; the 256*256 image corresponds to the layer number 2, that is, it is located at the top layer of the Gaussian pyramid.

[0053] Step 3: Obtaining Classifier Feature Maps

[0054] This step trains a classifier for each of the three different scales of images in the image pyramid to obtain class activation maps for the same image at the three different scales.

[0055] Step 3.1 Classification Network Training: The classification network is used to determine the category of the input image. The classification network selected in this invention is ResNet50. The ResNet50 network contains 49 convolutional layers and 1 fully connected layer. Each residual block has three convolutional layers. The residual structure of this network can directly connect the input to the subsequent network layers to avoid information loss. In this invention, a classifier is trained for images with three resolutions in the image pyramid: 256*256, 512*512, and 1024*1024, denoted as R1, R2, and R3 respectively.

[0056] Step 3.2 Fusion of High- and Low-Level Feature Maps: For each classification network, shallow feature maps have small receptive fields, extracting local and general features such as image texture and edges, i.e., low-level geometric information. As the network depth increases, higher-level feature maps have larger receptive fields, extracting deeper and more global features, i.e., high-level semantic information. Therefore, this invention concatenates the highest-level and lowest-level features in each classification network to obtain the final feature map output by the network, thereby enhancing the low-level features of small target objects. For classifier R1, let its highest-level feature map be... The lowest layer feature map is The final feature map f1 of classifier R1 can be obtained by formula (4).

[0057]

[0058] UP stands for upsampling, which involves upsampling the highest-level feature map to achieve the same size as the lowest-level feature map for easier subsequent processing. This is a lateral concatenation operation of the feature maps, i.e., element-wise addition. Similarly, the final feature maps f2 and f3 of classifiers R2 and R3 can be obtained from formulas (5) and (6).

[0059]

[0060]

[0061] in These are the highest and lowest layer feature maps obtained from classifier R2, respectively. These are the highest and lowest layer feature maps obtained by classifier R3, respectively, and UP represents the upsampling operation. This is a lateral connection operation on the feature map, i.e., element-wise addition.

[0062] Step 4: Multi-scale CAM fusion

[0063] This step obtains the CAMs of the three classification networks, aligns them, and then fuses them. The final output is the fused CAM image corresponding to the image.

[0064] Step 4.1 Obtaining the Classification Network (CAM): For classifier R1, the activation values ​​of spatial pixels u(x,y) with respect to class c in a 256*256 resolution image. It can be obtained from formula (7).

[0065]

[0066] Where i is the channel number of the last convolutional layer in the classification network, and K is the number of channels in the last convolutional layer of the classification network. f represents the weight corresponding to category c in channel i. i 1 (x,y) represents the feature value at position (x,y) on channel i in the final fused feature map of classifier R1. Similarly, for classifiers R2 and R3, the activation value of pixel u(x,y) with respect to class c is... It can be obtained from formulas (8) and (9) respectively.

[0067]

[0068]

[0069] Where i is the channel number of the last convolutional layer in the classification network, and K is the number of channels in the last convolutional layer of the classification network. f represents the weight corresponding to category c in channel i. i 2 (x,y) and f i 3 (x, y) are the feature values ​​at position (x, y) on channel i in the final fused feature map of classifiers R2 and R3, respectively.

[0070] Step 4.2 Alignment of Multiple CAMs: Since the inputs to classifiers R1, R2, and R3 are three-layer image pyramids, the sizes of the three class activation maps they produce also constitute an activation map pyramid. To fuse the three CAMs of different scales, they need to be aligned. In this invention, all CAMs are set to the same size as the original input image, i.e., 512*512 resolution.

[0071] Step 4.3 Multiple CAM Fusion: The three aligned CAMs are fused into the final CAM. For the fusion class activation map M... agg For pixel u(x,y) in the image, the fusion mechanism of this invention is as follows: if at least two independent activation maps have activation values ​​at this point with respect to class c that are greater than or equal to a threshold θ (θ∈[0.5, 0.7]), then M is considered to be... agg The pixel belongs to category c. If the pixel is not assigned to any category after the fusion mechanism, the pixel is ignored; if the pixel is assigned to multiple categories, its final category cla(x,y) is determined according to formula (10).

[0072]

[0073] Where j is the pyramid level number, P is the total number of pyramid levels (P = 3 here), and N is the number of categories in the dataset (excluding the background class). Let be the activation value of pixel u(x,y) with respect to category c in the feature map obtained at the j-th level of the pyramid. It refers to the average activation value of pixel u(x,y) in P feature maps with respect to the background class (class number 0). It refers to the average activation value of pixel u(x,y) with respect to class c in P feature maps. This refers to the average activation value of pixel u(x,y) across P feature maps with respect to category N. `index` is the index operation, which retrieves the index number corresponding to the maximum value in the array. Here, it also indicates the category to which the pixel belongs. For example, if the 0th pixel in the array has the largest average activation value, then the retrieved index value is 0, indicating that its category is 0.

[0074] Step 5: ROI Region Prediction

[0075] This step first converts the fused CAM obtained in step 4.3 into pseudo-labels, then trains a segmentation network based on the pseudo-labels, and finally uses the network to predict the ROI.

[0076] Step 5.1 CAM Fusion and Pseudo-Label Conversion: The fused CAM is converted into a pseudo-binary mask for training the segmentation network. Pseudo-binary mask The value of the pixel u(x,y) is determined by formula (11).

[0077]

[0078] Where cla(x,y) refers to the category to which pixel u(x,y) belongs, and cla(x,y) = 0 indicates that the pixel belongs to a non-target class.

[0079] Step 5.2 Segmentation Network Training and Prediction: Based on the pseudo-binary labels obtained in Step 5.1, an image segmentation network is trained. The segmentation network architecture selected in this invention is U-Net. Finally, the trained network is used to perform ROI segmentation prediction on the test set.

[0080] This invention primarily targets small target image data, such as medical image data with small lesion areas and pollen image data. By fusing multi-scale salient features obtained from image pyramids, it can enhance the location and contour information of small target objects, thereby improving the performance of small target object localization and segmentation tasks in weakly supervised scenarios. The steps described in specific embodiments of this invention can be modified, but the system architecture does not depart from the basic spirit of this invention. Therefore, the current embodiments should be considered exemplary rather than limiting, and the scope of this invention is defined by the appended claims rather than the foregoing description.

Claims

1. A weakly supervised image target localization method based on multi-scale salient feature fusion, characterized in that: Step 1: Image Preprocessing The purpose of image preprocessing is to standardize the dimensions of all images within the dataset; Step 2: Image Pyramid Construction This step uses the images in the dataset as the original images and constructs a Gaussian pyramid to obtain three scale transformations of the input images. In order to obtain more global and finer-grained information than the original images at the same time, the constructed Gaussian pyramid adopts a pyramid structure that combines downsampling and upsampling. Step 2.1 Image Pyramid Top Layer Construction: Using the input image as the original image, firstly, Gaussian smoothing is performed on it using a 5*5 template Gaussian kernel. Then, the processed image is downsampled by removing even-numbered rows and columns from the image matrix. Finally, an image of 1 / 4 the size of the input image is obtained, and this is used as the top layer of the image pyramid. Step 2.2 Image Pyramid Bottom Layer Construction: Using the input image as the original image, firstly, the image is enlarged to twice its original size in each direction, with the newly added rows and columns filled with the value 0; then, a 5*5 Gaussian kernel template is multiplied by 4 and convolved with the enlarged image to obtain an approximate value of the newly added pixels; finally, an image four times the size of the input image is obtained, and this is used as the bottom layer of the image pyramid. Step 2.3 Determining the number of image pyramid layers: Assigning numbers to the images on different layers of the image pyramid, where the image pyramid layer numbering starts from 0, and the image resolution decreases accordingly as the pyramid layer number increases; the constructed image pyramid has 3 layers, with the original image located on the 2nd layer, and the corresponding pyramid layer number is 1; Step 3: Obtaining Classifier Feature Maps This step trains a classifier for each of the three different scales of images in the image pyramid to obtain class activation maps for the same image at the three different scales. Step 3.1 Classification Network Training: The classic ResNet50 is selected as the classification network to determine the category of the input image. Since there are three different scales of images in the image pyramid, a classifier needs to be trained for each of the three different scale image datasets. Step 3.2 Fusion of high and low layer feature maps: Align and concatenate the highest and lowest level features of each classification network to enhance the low-level features of small target objects, thereby obtaining the final fused feature map of the network. Step 4: Multi-scale CAM fusion This step obtains the CAMs of the three classification networks, aligns them, and then fuses them to finally obtain the fused CAM image corresponding to the image. Step 4.1 Classification Network CAM Acquisition: For the fused feature map obtained in Step 3.2, the CAM is obtained by multiplying it with the weight matrix of the fully connected layer in the classification network. Since three classification networks are used, for each original image, three CAMs of different scales will be obtained, forming a CAM pyramid. Step 4.2 Aligning Multiple CAMs: Align CAMs of different scales based on the dimensions of the original image to facilitate subsequent fusion operations; Step 4.3 Multiple CAM Fusion: For any pixel in the fused CAM, the following judgment mechanism is adopted: if at least two independent CAMs have activation values ​​of the pixel with respect to a certain category that are greater than or equal to a threshold, then the pixel is considered to belong to that category; if the pixel is not assigned to any category after the judgment mechanism, then the pixel is ignored; if the pixel is assigned to multiple categories, then the pixel is assigned to the category corresponding to the maximum average activation value of the three independent CAMs at that pixel. Step 5: ROI Prediction This step first converts the fused CAM obtained in step 4.3 into pseudo-labels, then trains a localization and segmentation network for image ROI based on the pseudo-labels, and finally uses the network to predict ROI. Step 5.1 CAM Fusion and Pseudo-Label Conversion: Convert the fused CAM into a pseudo-binary mask for training the segmentation network; The following judgment mechanism is adopted: if any pixel in the fused CAM belongs to a non-target class, then the pixel value of that point is assigned to 0; otherwise, it is assigned to 1. Step 5.2 Segmentation Network Training and Prediction: Based on the pseudo-binary labels obtained in Step 5.1, an image segmentation network is trained. The segmentation network architecture selected is U-Net. Finally, the trained network is used to perform ROI segmentation prediction on the test set.

2. The weakly supervised image target localization method based on multi-scale salient feature fusion according to claim 1, characterized in that: Step 1: Image Preprocessing The purpose of image preprocessing is to standardize the size of all images in the dataset; the size of all images is set to 512*512. Step 2: Image Pyramid Construction The construction process consists of two parts: First, the width and height of the input original image are downsampled to 50% of the original image using a Gaussian pyramid, resulting in a 256*256 resolution image as the top layer of the pyramid; second, the width and height of the input original image are upsampled to 200% of the original image using a Gaussian pyramid, resulting in a 1024*1024 resolution image as the bottom layer of the pyramid; as detailed below: Step 2.1 Building the top layer of the image pyramid: For a given original image of size 512*512, downsampling is performed to construct the top layer of a Gaussian pyramid with an image size of 1 / 4 of the original image, resulting in an image resolution of 256*256. The specific process is shown in formula (1): First, the original image of size 512*512 is subjected to Gaussian smoothing. Unlike simple smoothing, Gaussian smoothing assigns higher weight values ​​to pixels near the center point when calculating the weighted average of surrounding pixels. Then, the processed image is downsampled by removing even-numbered rows and columns from the image matrix to obtain an image with a resolution of 256*256. (1) ； in The first of the Gaussian pyramids Layered image, Gaussian pyramid layers start from 0. This is the level number of the top layer of the Gaussian pyramid. and The first The number of rows and columns in the layer image. The first Gaussian filter template Line number The values ​​are listed, and the size is 5*5. The two-dimensional separable 5*5 Gaussian kernel, which is widely used in the anti-sharpening mask algorithm, is used to smooth the original image. The values ​​are shown in (2). (2) Step 2.2 Building the bottom layer of the image pyramid: Given a 512*512 original image, upsampling is used to construct the lowest layer of a Gaussian pyramid with an image size four times that of the original image, corresponding to a resolution of 1024*1024. The specific process is as follows: First, the image is enlarged to twice the size of the original image in each direction, with the newly added rows and columns filled with the value 0. Then, the Gaussian kernel used in downsampling is multiplied by 4 and then convolved with the enlarged image to obtain an approximate value for the newly added pixels, finally resulting in an image with a resolution of 1024*1024. Step 2.3 Determining the number of layers in the image pyramid: Once the image pyramid is constructed, the image with resolution w*h corresponds to the number of layers in the Gaussian pyramid. Determined by formula (3); (3) in This represents the layer number in the image pyramid for the original 512x512 image. Since the three image scales in the Gaussian pyramid are 1024, 512, and 256, the corresponding layer number for the original image is... From formula (3), we can see that the image size of 1024*1024 corresponds to the number of layers 0, which means it is located at the lowest layer of the Gaussian pyramid; the image size of 256*256 corresponds to the number of layers 2, which means it is located at the top layer of the Gaussian pyramid. Step 3: Obtaining Classifier Feature Maps This step trains a classifier for each of the three different scales of images in the image pyramid to obtain class activation maps for the same image at three different scales, as detailed below; Step 3.1 Classification Network Training: The classification network is used to determine the category of the input image. The selected classification network is ResNet50, which contains 49 convolutional layers and 1 fully connected layer. Each residual block has three convolutional layers. A classifier is trained for each of the three resolutions in the image pyramid: 256*256, 512*512, and 1024*1024. , , ; Step 3.2 Fusion of high- and low-level feature maps: The highest and lowest layer features in each classification network are concatenated to form the final feature map output by the network, thereby enhancing the low-level features of small objects; for the classifier Let the highest-level feature map obtained by its network be... The lowest layer feature map is Then the classifier Final feature map It can be obtained from formula (4); (4) UP stands for upsampling, which involves upsampling the highest-level feature map to achieve the same size as the lowest-level feature map for easier subsequent processing. The lateral connection operation for the feature maps is essentially element-wise addition; similarly, the classifier... and classifier Final feature map , Then it can be obtained from formulas (5) and (6); (5) (6) in , Classifiers The obtained highest-level feature map and lowest-level feature map, , These are respectively classifiers The obtained highest and lowest layer feature maps, where UP represents the upsampling operation. This is a lateral connection operation of the feature map, i.e., element-wise addition; Step 4: Multi-scale CAM fusion The CAMs of the three classification networks are obtained, aligned, and then fused. The final output is the fused CAM image corresponding to the image, as shown below: Step 4.1 Obtaining the Classification Network (CAM): For the classifier In terms of spatial pixels in a 256*256 resolution image About classes activation value It can be obtained from formula (7); (7) in This refers to the channel number of the last convolutional layer in the classification network. The number of channels in the last convolutional layer of the classification network. For channel Medium category The corresponding weights For classifier Finally, the channels in the fused feature map are merged. Upper position eigenvalues; Similarly, for classifiers and classifier Pixels on the image About classes activation value , It can be obtained from formulas (8) and (9) respectively; (8) (9) in This refers to the channel number of the last convolutional layer in the classification network. The number of channels in the last convolutional layer of the classification network. For channel Medium category The corresponding weights and Classifiers and Finally, the channels in the fused feature map are merged. Upper position eigenvalues; Step 4.2 Multiple CAM Alignment: Due to the classifier , , The input is a three-layer image pyramid, so the sizes of the three class activation maps obtained also constitute the activation map pyramid. In order to fuse the three CAMs of different scales, they need to be aligned and all CAMs are set to the same size as the original input image, i.e., 512*512 resolution. Step 4.3 Multiple CAM Fusion: The three aligned CAMs are fused into the final CAM; for fusion-type activation maps... pixels in The fusion mechanism is as follows: if at least two independent activation graphs exist at this point with respect to class... The activation value is greater than or equal to the threshold. , If ∈ [0.5, 0.7], then it is considered This pixel belongs to the category If the pixel is not assigned to any category after the fusion mechanism, then the pixel is ignored. If the pixel is assigned to multiple categories, its final category is determined according to formula (10). ; ； (10) in Number the pyramid levels. This represents the total number of levels in the pyramid. , The number of categories that divide the dataset, excluding the background class. For pixels In the The feature map obtained from the layer pyramid is related to the category The activation value; pixel exist The average activation value of the background class in each feature map, where the background class number is 0; pixel exist Regarding the category in the feature map The average activation value, pixel exist Regarding the category in the feature map The average activation value; This is an indexing operation, which retrieves the index number corresponding to the maximum value in the array, and in this case, it also refers to the category to which the pixel belongs. Step 5: ROI Region Prediction This step first converts the fused CAM obtained in step 4.3 into pseudo-labels, then trains a segmentation network based on the pseudo-labels, and finally uses the network to predict the ROI; the details are as follows: Step 5.1 CAM Fusion and Pseudo-Label Conversion: Convert the fused CAM into a pseudo-binary mask for training the segmentation network; pseudo-binary mask medium pixel The value of is determined by formula (11); (11) in pixel Category to which it belongs This indicates that the pixel belongs to a non-target class; Step 5.2 Segmentation Network Training and Prediction: Based on the pseudo-binary labels obtained in Step 5.1, an image segmentation network is trained. The segmentation network architecture selected is U-Net. Finally, the trained network is used to perform ROI segmentation prediction on the test set.

Images