A weakly supervised object localization method based on category correction

Abstract

The present application belongs to the field of computer vision, and particularly relates to a weakly supervised object localization method based on category correction. In order to solve the problem of inaccurate localization of the CAM technology, the rough-to-fine process is adopted instead of using the category feature map for localization. The model of the present application is composed of a backbone network, a localization network and a classification network. First, the localization network generates a class-independent segmentation map by using unsupervised segmentation technology, so as to determine the rough position of the target object. Then, the classification network is used for fine-grained correction through the category label. The method based on category correction can accurately locate the object and well identify the contour details.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a weakly supervised target localization method based on category correction. Background Technology

[0002] Object localization is a fundamental perceptual task in computer vision, aiming to pinpoint the exact location of a target object in an image and determine its category. However, in practical applications, achieving good generalization performance for the algorithm often requires significant manual effort to annotate the target bounding boxes, sometimes down to the pixel level. Due to the cost of annotation, weakly supervised object localization tasks often rely on easily obtainable and labeled category labels to enable the model to locate objects. For weakly supervised object localization, mainstream research both domestically and internationally is based on CAM (Collection of Objects) techniques, using category-related attention highlighting regions in the image to determine the object's location. However, such methods typically only locate the category-discriminating parts of the object, resulting in inaccurate bounding boxes that are often smaller than the target object. Therefore, obtaining accurate bounding boxes has become a pressing problem in the field of weakly supervised object localization. Summary of the Invention

[0003] This invention primarily provides a weakly supervised target localization method based on category correction. To address the inaccuracy of CAM (Computer-Aided Localization) technology, we abandon the use of category feature maps for localization and instead employ a coarse-to-fine process. The algorithm consists of a localization network and a classification network. First, the localization network uses unsupervised segmentation techniques to generate category-independent segmentation maps, thereby determining the coarse location of the target object. Then, the classification network performs fine-grained correction using category labels. This category-correction-based method can accurately locate objects and also effectively identify contour details.

[0004] The technical solution provided by this invention includes a coarse-to-fine target localization method comprising a training phase and a testing phase, wherein the training phase includes the following steps:

[0005] Step 1: Construct a target localization model, which includes a backbone network, a classification network, and a localization network. The backbone network extracts features from the input image, and the classification network and localization network are dual networks that predict the category and mask of the features extracted from the backbone network.

[0006] Step 2: For the input image I, generate a synthetic image I that has a distributional similarity to the training samples. s and foreground mask M s Then synthesize the image I s Inputting the target localization model yields the mask predicted by the localization network.

[0007] Step 3, Image-level Fine-tuning Stage: Increase the difference between foreground and background in the image hierarchy to allow the localization network to locate more accurately; includes the following sub-steps:

[0008] Step 3.1: Obtain the real image I from the localization network trained in Step 2, which has coarse localization capabilities. r Foreground mask prediction

[0009] Step 3.2, predict the foreground mask. Compared to real pictures I r Performing the Hadamard product yields a category-independent foreground attention image I. f Simultaneously, the foreground mask is converted from 0 to 1. Real Image I r and Performing the Hadamard product yields a category-independent background attention image I. f ;

[0010] Step 3.3, foreground attention image I f and background attention image I b The data is fed into a classification network for prediction, yielding the predicted probability features. as well as

[0011] Step 4, Feature-level fine localization stage: After the differences between the foreground and background at the image level are amplified, the differences between the foreground and background at the feature level are increased using the same method as in Step 3, allowing the localization network to further correct the details that have been mislocalized and output the final localization result.

[0012] The testing phase process is as follows:

[0013] Disconnect the localization network and the classification network, and obtain the final localization bounding box by thresholding the foreground mask of the localization network:

[0014]

[0015] in This represents the mask predicted by the localization network for the test sample, where θ is the filtering threshold, and the Select function selects... For portions exceeding the threshold, return a minimum bounding box containing all foreground coordinates as the final determined positioning box.

[0016] Furthermore, in step 1, the backbone network adopts the U-Net network structure, and the localization network adopts the CNN convolutional network structure.

[0017] Furthermore, in step 2, the BigBiGAN method is used to generate the synthesized image and mask.

[0018] Furthermore, the mask is obtained in step 2. The specific formula is as follows:

[0019]

[0020] Where θ B and θ L These represent the parameters of the backbone network and the positioning network, respectively.

[0021] Furthermore, the localization network is optimized using a bisection cross-entropy function, with the following loss function:

[0022]

[0023] Where m and n are the width and height of the mask, Foreground mask M s The element in row i and column j, For predicting the mask The element in row i and column j.

[0024] Furthermore, the predicted probability features in step 3.3 as well as The calculation formula is as follows:

[0025]

[0026] Where θ B and θ C Here, represent the parameters of the backbone network and the classifier, respectively. The loss functions for the foreground and background attention images are as follows:

[0027]

[0028] in It is the cross-entropy function of the foreground attention image. is the negative of the entropy of the background attention image, and K is the total number of categories in the dataset. The overall loss function in the image-level fine-tuning stage can be expressed as:

[0029]

[0030] α and B are equilibrium parameters.

[0031] Furthermore, the specific implementation method of step 4 is as follows;

[0032] Step 4.1, for real image I r The feature map is obtained using the localization network trained in step 3. and mask The formula for calculating the feature map is as follows:

[0033]

[0034] Where θ B These are the parameters of the backbone network;

[0035] Step 4.2, transfer the feature map With mask and the 0-1 conversion of the mask Perform Hadamard product on each side to obtain the foreground feature map. and background feature map The formula is as follows:

[0036]

[0037] Step 4.3: Fix the classification network weights trained in Step 3 and use them as the mask quality judge to analyze the foreground feature maps. and background feature map The data is fed into a classification network for prediction, yielding the predicted probability features. as well as

[0038] Where θ C The parameters represent the classifier; the specific loss function is as follows:

[0039]

[0040] in It is the cross-entropy of the foreground features. It is the negative of the entropy of the background features, and K is the total number of classes in the training samples. The overall loss function in the feature-level fine localization stage can be expressed as:

[0041]

[0042] α and β are equilibrium parameters.

[0043] Furthermore, the threshold θ is set to 0.55 ± 0.05.

[0044] Furthermore, both α and β take the value 1.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention avoids the drawback of CAM technology, which suffers from limited localization. CAM relies entirely on category information for training to obtain category attention images, but it neglects object regions with low category discrimination, resulting in only coarse localization, particularly poor performance on fine-grained datasets. This invention employs a combined approach of category-independent and category-related information. It trains the network using category-independent segmentation maps and then refines details using category information, achieving fine-grained localization. This invention can completely locate the object's outline, and the feature maps clearly delineate the target object's contour information. In the fine-grained localization stage, category information plays an auxiliary corrective role, preventing the network from ignoring foreground regions with low category discrimination, thus overcoming the shortcomings of CAM technology. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the training process in an embodiment of the present invention.

[0048] Figure 2 This is a test flowchart in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0050] This invention proposes a weakly supervised target localization algorithm based on category correction. This algorithm utilizes both category-independent and category-dependent information while avoiding CAM (Class-Independent Mapping). [1] The present invention addresses the drawbacks of the technology by proposing a dual-head network structure of locator and classifier to learn both class-independent and class-related information. The locator consists of a segmentation network that predicts the foreground mask of the input image. The classifier then refines the locator's prediction structure at both the image level and the feature level.

[0051] [1] B.Zhou, A.Khosla, A.Lapedriza, A.Oliva, and A.Torralba, “Learning deepfeatures for discriminative localization,” in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2016, pp.2921–2929.

[0052] This invention proposes a coarse-to-fine training process to optimize the algorithm model, as shown in the appendix. Figure 1 As shown. In the coarse localization stage, for category-independent information, we generate a synthetic image I using an unsupervised method. sand foreground mask M s Related unsupervised methods include [2],[3] The locator predicts the synthetic image I. s The mask and the unsupervised generated M s It can be supervised to achieve feature segmentation capabilities that are independent of category.

[0053] [2] A.Voynov, S.Morozov, and A.Babenko, “Object segmentation without labels with large-scale generative models,” in International Conference onMachine Learning.PMLR, 2021, pp.10 596–10 606

[0054] [3] M.Chen, T.Artieres, and L.Denoyer, "Unsupervised object segmentation by redrawing," Advances in Neural Information Processing Systems, vol.32, 2019.

[0055] In the fine localization stage, the locator with coarse localization capability first predicts the true image I. r foreground mask To further enhance the difference between foreground and background, we perform a Hadamard product between the foreground mask and the original image to obtain foreground and background attention images. The classifier then performs different classification tasks on the two attention images to optimize them: the foreground image is supervised by class labels, while the background image should not be assigned to any class and is supervised by suppressing its highly assigned class. This further increases the difference between foreground and background at the feature level. The Hadamard product of the localizer's feature map and its predicted foreground mask yields the separated foreground and background feature maps. After fine-grained localization at the image level, the classifier learns a certain foreground classification ability. At this point, the classifier's weights are fixed so that it no longer participates in gradient backpropagation. The classifier is then used to judge the separation quality of the foreground and background feature maps, thereby increasing the foreground-background discriminability at the feature level.

[0056] After fine-grained localization training at the image and feature levels, the locator, through category correction, can better determine foreground and background regions from a semantic perspective. This avoids the limitation of CAM technology, which cannot detect foregrounds lacking category discrimination. This invention can accurately predict the contours of objects. This is because the locator is trained based on category-independent features, thus possessing excellent recognition capabilities for contour and texture information. After category correction, the algorithm can judge both semantically relevant and semantically irrelevant foreground information.

[0057] As attached Figure 2 As shown, during the testing phase, this invention uses a trained locator to complete the target localization task. First, the locator predicts the target location based on the real image I. r mask Foreground mask based on a pre-determined threshold Binarization is performed to obtain a binary foreground mask. The foreground value is 1, and the background value is 0. The foreground values ​​may be discontinuous or clustered. This invention selects the largest continuous foreground value cluster as the predicted foreground, and the rest are considered background. For the selected foreground, the tightest bounding box containing it (i.e., the smallest bounding box containing all foreground coordinates) is taken as the target object's location box.

[0058] The process provided in the embodiment specifically includes the following steps:

[0059] Step 1: Construct a target localization model, which includes a backbone network, a classification network, and a localization network. The backbone network extracts features from the input image, and the classification network and localization network are dual networks that predict the category and mask of the features extracted from the backbone network.

[0060] Step 2: Generate a synthetic image I that has a similar distribution to the training samples using unsupervised algorithms such as GAN. s and foreground mask M s BigBiGAN was chosen in the example. [2] The method generates a composite image and a mask. Then the composite image I... s Inputting the target localization model yields the foreground mask predicted by the localization network. The formula is as follows:

[0061]

[0062] Where θ B and θ L These represent the parameters of the backbone network and the localization network, respectively. In the example, θ B Using the U-Net network structure, θ L A CNN convolutional network structure is used. f represents the process of the network predicting the mask. Mask The closer a pixel value is to 1, the more likely the localization network is to classify it as foreground; conversely, the closer a pixel value is to 0, the more likely the localization network is to classify it as background. The algorithm then uses a binary cross-entropy function to optimize the localization network. The loss function is as follows:

[0063]

[0064] Where m and n are the width and height of the mask, Foreground mask M s The element in row i and column j, For predicting the mask The element in row i and column j. Through step 2, the localization network has a coarse, class-independent localization capability. Next, class information is used to perform class-related corrections on the localization network.

[0065] Step 3: Image-level fine-tuning stage: This stage increases the difference between the foreground and background at the image level to correct the image. Step 3 can be divided into the following sub-steps:

[0066] Step 3.1: Obtain the real image I through a localization network with coarse localization capabilities. r Prediction mask

[0067] Step 3.2: Evaluate the mask using category information The quality of the sample I is adjusted and corrected during training. r With mask and the 0-1 conversion of the mask Hadamard accumulation was performed separately. The foreground attention image and background attention image are calculated using the following formula:

[0068]

[0069] Step 3.3: Transfer the foreground attention image I to the foreground image. f and background attention image I b The data is fed into a classification network for prediction, yielding the predicted probability features. as well as The formula in the example is expressed as follows:

[0070]

[0071] Where θ B and θ C These represent the parameters of the backbone network and the classification network, respectively. For I... f Supervision is performed using category labels, and the loss function is the cross-entropy function. For Ib It does not belong to any category, therefore it is hoped that the model will be applicable to I. b The predicted class probabilities tend to be average, resulting in neither excessively high nor excessively low predicted class probabilities. In the example, this manifests as letting I... b The entropy of the predicted probability should be as large as possible. The loss function for the foreground and background attention image is as follows:

[0072]

[0073] in It is the cross-entropy function of the foreground attention image. is the negative of the entropy of the background attention image, and K is the total number of classes in the training samples. The overall loss function in the image-level fine-tuning stage can be expressed as:

[0074]

[0075] Here, α and β are balancing parameters. Extensive experimental testing in practice has shown that setting both to 1 yields excellent algorithm performance. In this step, the locator undergoes category-related correction at the image level, and the classifier is trained to have classification capabilities, preparing for category correction at the feature level in the next step.

[0076] Step 4: After increasing the foreground and background difference at the image level, this step further ensures that the foreground and background still have differences at the feature level. This is more conducive to the localization network's foreground localization. Step 4 can be further subdivided into the following sub-steps:

[0077] Step 4.1: For real image I r The feature map is obtained using the localization network trained in step 3. and mask The formula for calculating the mask is the same as in (1), and the formula for calculating the feature map is as follows:

[0078]

[0079] Where θ B These are the parameters of the backbone network.

[0080] Step 4.2: Transfer the feature map With mask and the 0-1 conversion of the mask Perform Hadamard product on each side to obtain the foreground feature map. and background feature map The formula is as follows:

[0081]

[0082] Step 4.3: Fix the weights of the classification network trained in Step 3 and use them as the judge of mask quality. Then, process the foreground feature maps... and background feature map The data is fed into a classification network for prediction, yielding the predicted probability features. as well as The formula in the example is expressed as follows:

[0083]

[0084] Where θ C These represent the parameters of the classification network. The functions for foreground and background probability features are consistent with those in step 3.3. The algorithm optimizes the foreground probability features by minimizing the cross-entropy function and optimizes the background probability features by maximizing the entropy function. In this example, the specific loss function is as follows:

[0085]

[0086] in It is the cross-entropy of the foreground features. It is the negative of the entropy of the background features, and K is the total number of classes in the training samples. The overall loss function in the feature-level fine-tuning stage can be expressed as:

[0087]

[0088] α and β are balancing parameters. Through extensive experimental verification in practice, consistent with step 3, it was found that setting both to 1 can achieve good results for the algorithm.

[0089] Although steps 4 and 3 are similar, step 4 is essential. In step 3, the classification network is also involved in training; that is, while part of the loss function corrects the features of the localization model, a larger portion adjusts the classification network. Step 4, by fixing the classification network and making adjustments at the feature level, allows category information to be more fully transmitted to the localization network, enabling fine-grained correction of the network's localization results. At the same time, step 3 is also indispensable; without it, a classification network with generalization capabilities cannot be obtained.

[0090] The following points should be noted during implementation:

[0091] During the testing phase of this invention, extensive experiments demonstrated that a threshold of 0.55 yielded excellent results in the CUB dataset for the selection of the threshold θ. It should be noted that, compared to similar methods, the method in this invention is less sensitive to threshold values, achieving good results within a range of ±0.15, while the threshold tolerance range of similar methods is often less than ±0.05.

[0092] It should be emphasized that the embodiments described in this invention are illustrative, not limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation details. Other implementations derived by those skilled in the art based on the technical solutions of this invention also fall within the scope of protection of this invention.

Claims

1. A weakly supervised target localization method based on category correction, characterized in that: It includes a training phase and a testing phase, with the training phase comprising the following steps: Step 1: Construct a target localization model, which includes a backbone network, a classification network, and a localization network. The backbone network extracts features from the input image, and the classification network and localization network are dual networks that predict the category and mask of the features extracted from the backbone network. Step 2: For the input image I, generate a synthetic image I that has a distributional similarity to the training samples. s and foreground mask M s Then synthesize the image I s Inputting the target localization model yields the mask predicted by the localization network. Step 3, Image-level fine-tuning stage: Increase the difference between the foreground and background in the image layers to enable the localization network to locate more accurately; Includes the following sub-steps: Step 3.1: Obtain the real image I from the localization network trained in Step 2, which has coarse localization capabilities. r Foreground mask prediction Step 3.2, predict the foreground mask. Compared to real pictures I r Performing the Hadamard product yields a category-independent foreground attention image I. f Simultaneously, the foreground mask is converted from 0 to 1. Real Image I r and Performing the Hadamard product yields a category-independent background attention image I. b ; Step 3.3, foreground attention image I f and background attention image I b The data is fed into a classification network for prediction, yielding the predicted probability features. as well as Step 4, Feature-level fine localization stage: After the differences between the foreground and background at the image level are amplified, the differences between the foreground and background at the feature level are increased using the same method as in Step 3, allowing the localization network to further correct the details that have been mislocalized and output the final localization result. The testing phase process is as follows: Disconnect the localization network and the classification network, and obtain the final localization bounding box by thresholding the foreground mask of the localization network: in This represents the mask predicted by the localization network for the test sample, where θ is the filtering threshold, and the Select function selects... For portions exceeding the threshold, return a minimum bounding box containing all foreground coordinates as the final determined positioning box.

2. The weakly supervised target localization method based on category correction as described in claim 1, characterized in that: In step 1, the backbone network adopts the U-Net network structure, and the localization network adopts the CNN convolutional network structure.

3. The weakly supervised target localization method based on category correction as described in claim 1, characterized in that: In step 2, the BigBiGAN method is used to generate the synthesized image and mask.

4. The weakly supervised target localization method based on category correction as described in claim 1, characterized in that: Obtain the mask in step 2 The specific formula is as follows: Where θ B and θ L These represent the parameters of the backbone network and the positioning network, respectively.

5. The weakly supervised target localization method based on category correction as described in claim 4, characterized in that: The localization network is optimized using a bisection cross-entropy function, with the following loss function: Where m and n are the width and height of the mask, Foreground mask M s The element in row i and column j, For predicting the mask The element in row i and column j.

6. The weakly supervised target localization method based on category correction as described in claim 1, characterized in that: The predicted probability features in step 3.3 as well as The calculation formula is as follows: Where θ B and θ C Here, represent the parameters of the backbone network and the classifier, respectively. The loss functions for the foreground and background attention images are as follows: in It is the cross-entropy function of the foreground attention image. is the negative of the entropy of the background attention image, and K is the total number of categories in the dataset. The overall loss function in the image-level fine-tuning stage can be expressed as: α and β are equilibrium parameters.

7. The weakly supervised target localization method based on category correction as described in claim 1, characterized in that: The specific implementation method of step 4 is as follows; Step 4.1, for real image I r The feature map is obtained using the localization network trained in step 3. and mask The formula for calculating the feature map is as follows: Where θ B These are the parameters of the backbone network; Step 4.2, transfer the feature map With mask and the 0-1 conversion of the mask Perform Hadamard product on each part to obtain the foreground feature map. and background feature map The formula is as follows: Step 4.3: Fix the classification network weights trained in Step 3 and use them as the mask quality judge to analyze the foreground feature maps. and background feature map The data is fed into a classification network for prediction, yielding the predicted probability features. as well as Where θ C The parameters represent the classifier; the specific loss function is as follows: in It is the cross-entropy of the foreground features. It is the negative of the entropy of the background features, and K is the total number of classes in the training samples. The overall loss function in the feature-level fine localization stage can be expressed as: α and β are equilibrium parameters.

8. The weakly supervised target localization method based on category correction as described in claim 1, characterized in that: The threshold θ is set to 0.55 ± 0.

05.

9. A weakly supervised target localization method based on category correction as described in claim 6 or 7, characterized in that: Both α and β take the value 1.

Images