A Weakly Supervised Object Localization Method Based on Adaptive Domain Learning

Abstract

A weakly supervised object localization method based on adaptive domain learning, which involves computer vision technology. Model training stage: Step 1, preprocess the image to be detected, and then send the preprocessed image to be detected and its corresponding category into the neural network; Step 2, the neural network processes the image and outputs the probability value of each category corresponding to the image to be detected during the training process. Model testing stage: During the testing process, the coordinates, category, and score of the predicted box of the image to be detected are output. The ALM induces the network to adversarially generate the foreground and background of the image to the greatest extent, and the OLM refines the foreground and background on this basis to obtain refined object and background information. For images with unsatisfactory foreground and background discrimination effects, the RLM extracts the key information in the foreground and enables the network to further learn from it. Experiments on the CUB-200-2011 and ILSVRC datasets can achieve significant and consistent performance improvements.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a weakly supervised object localization method based on adaptive domain learning. Background Art

[0002] In recent years, with the improvement of computer performance and the development of big data, the visual information data has increased rapidly. Multimedia data such as static images, dynamic images, video files, and audio files are all spreading at a very fast speed on various social media. As one of the most basic problems in the field of computer vision, object detection has been widely applied in many fields such as object tracking, behavior understanding, human-computer interaction, and face recognition, and has attracted extensive attention and research from many scholars in the early 20th century. Humans mainly receive external information through vision, so the application technology based on visual information will be a forward-looking research point of artificial intelligence. Among them, technologies such as face recognition, video surveillance, object detection, Internet image content review, and biometric recognition have become current research hotspots. These technologies are also widely applied in fields such as medical care, elderly care, transportation, urban operation, and security. For example: medical image diagnosis, pose estimation, station security inspection, autonomous driving, vehicle speed detection, video surveillance behavior analysis, etc.

[0003] Object localization is an extremely important research field in computer vision and machine learning, integrating cutting-edge knowledge in multiple fields such as image processing, pattern recognition, artificial intelligence, and automatic control. Object localization usually has only one relatively large object located in the middle of the picture, and it identifies and locates this object. With the development of video websites and social networks, people can access a large number of multimedia resources such as images and videos, and object localization has also begun to be widely applied in the above fields. For example, face detection in pictures on social websites, pedestrian detection in images or video sequences, vehicle detection in traffic monitoring, and helping people with visual impairments understand visual content, etc.

[0004] Object localization focuses on research on convolutional neural networks (CNNs). In the process of training the detector, this convolutional neural network uses a large amount of data with instance-level labels (i.e., bounding box annotations). However, collecting bounding box annotations for specific classes is obviously a time-consuming and laborious task, which limits the practical use of the detector. Compared with bounding box annotations, collecting image-level labels is much easier. For example, by querying an image search engine (such as Google Image) or a photo sharing website (such as Flickr), it is possible to easily manually check whether a target object exists in the collected images. Therefore, the weakly supervised object localization (WSOL) task, that is, training an object detector only with image-level supervision, has recently attracted more and more attention.

[0005] To utilize image tags for localization, most existing WSOL methods adopt a CNN classifier to mine the localization map of the target class, form a localization head, and then generate predicted bounding boxes to tightly cover the high-response regions. There are two challenging problems that create a significant performance gap between WSOL and its fully supervised counterparts. First, the classifier often ignores insignificant image features and overly relies on significant regions for classification; and since the backbone network is usually pre-trained through image classification, it is difficult to learn a successful localization head. Second, the localization head may only capture the discriminative parts of the target object because the bounding boxes are generated only around the most discriminative parts of the map. The above problems may further interact with each other. For example, given an image of a bird, even the head or feathers of the bird can provide sufficient clues to achieve a high classification accuracy. Taking this image as an example, however, the boxes generated around the head or feathers do not localize the entire area of the bird, resulting in poor localization performance.

[0006] To address WSOL with only image-level labels, most methods rely on class activation maps (CAMs) to discover discriminative image regions for localizing potential target objects. However, CAMs only activate the most discriminative clues related to the object class by using the classification loss, thus severely underestimating the object regions, i.e., producing bounding boxes that are much smaller than the actual extent. To capture more object parts, different kinds of techniques have been proposed to enhance CAMs, such as graph propagation, data augmentation, adversarial erasing, and spatial relation activation. Although existing methods have achieved encouraging results, due to the inherent characteristics of convolutional neural networks (CNNs), they fail to correctly explore the global feature relationships and thus still exhibit limited performance in fully localizing objects.

[0007] Recently, a brand-new weakly supervised object localization paradigm has been proposed, which locates objects by learning foreground prediction maps (FPMs) from the feature extraction network, avoiding the CAM method and overcoming the drawback that CAM can only locate the most prominent part of the object locally. FPM provides a new perspective for WSOL by jointly optimizing the tasks of object localization and classification in a unified model. Meng et al. (M. Meng, et al. “Foreground activation maps for weakly supervised object localization,” in IEEE International Conference on Computer Vision, 2021) designed a foreground memory mechanism to generate foreground prediction maps. Xie et al. (J. Xie, et al. “Online refinement of low-level feature based activation map for weakly supervised object localization,” in IEEE Conference on Computer Vision and Pattern Recognition, 2021) adopted a two-stage method, first generating foreground prediction maps based on low-level features and then refining them with a validator. In addition, Wu et al. (P. Wu, W. Zhai, and Y. Cao, “Background activation suppression for weakly supervised object localization,” in IEEE Conference on Computer Vision and Pattern Recognition, 2022) proposed a background activation suppression method to suppress the background and obtain better foreground prediction maps. However, these methods only focus on activating the foreground or suppressing the background, being limited to improving the detection effect of the foreground and ignoring the connection between the foreground and the background.) Summary of the Invention

[0008] The object of the present invention is to provide a weakly supervised object localization method based on adaptive domain learning, which can obtain better features with only weak label information through a weakly supervised training method with low-cost image annotation, achieving better training results.)

[0009] The present invention provides a weakly supervised object localization method based on adaptive domain learning (hereinafter referred to as AZL), comprising the following steps:

[0010] 1) Preprocess the image to be detected, and send the preprocessed image to be detected and its corresponding image-level label into the neural network; the neural network includes: a backbone feature extraction network, an adversarial learning module ALM, an oriented learning module OLM, and a reinforcement learning module RLM;

[0011] 2) The neural network processes the image, and outputs the probability value of each category corresponding to the image to be detected during the training process; during the testing process, it outputs the coordinates, category, and score of the predicted box of the image to be detected. The adversarial learning module ALM induces the network to adversarially generate the foreground and background of the image to the greatest extent, and the oriented learning module OLM refines the foreground and background on this basis to obtain refined object and background information.

[0012] In step 1), for the preprocessing, the image can be first normalized, then scaled to a size of 256×256, and finally randomly cropped to a size of 224×224;

[0013] The neural network includes: a backbone feature extraction network (VGG16, MobileNetV1, ResNet50, InceptionV3), an adversarial learning module (Adversarial Learning Mechanism, ALM), an oriented learning module (Oriented Learning Mechanism, OLM), and a reinforcement learning module (Reinforcement Learning Mechanism, RLM). The adversarial learning module (ALM) on the one hand encourages the activation of the foreground while suppressing the background, and on the other hand controls the size of the foreground activation, thereby inducing the network to adversarially generate the foreground and background of the image to the greatest extent; the oriented learning module (OLM) is used to further refine the already generated foreground and background to obtain more fine-grained object and background information; the reinforcement learning module (RLM) is used for the situation where the distinction effect of some foreground and background is not obvious, and enhances the learning effect of the network by further extracting the key information of the foreground.

[0014] In step 2), the neural network processes the image, including the following steps:

[0015] Step a1, given a dataset set with image-level labels, divide the set into a training image sample set and a test image sample set;

[0016] Step a2, randomly select an image I from the training image sample set, and input the image I and its corresponding image-level label y into the shallow feature extraction backbone network of the neural network Obtain the output of the network regarding the shallow features of the image

[0017] Step a3, shallow features After the generator network, we get the prospect prediction graph X fpm ;

[0018] Step a4: Determine the foreground prediction map M of the corresponding class of the image based on the label y fpm , and perform the inverse operation to obtain the corresponding background prediction map M bpm ;

[0019] Step a5, shallow features Continue through the deep feature extraction backbone network of the neural network Obtain image features Image classification loss obtained by global average pooling at the same time, By M fpm After point-by-point multiplication to obtain foreground features, the foreground classification loss is obtained by global average pooling.

[0020] Step a6, With M bpm After point-by-point multiplication to obtain shallow background features, the backbone network is extracted through deep features. Get background image feature X bg ,after Subtract X bg Get the foreground image feature X fg . To X fg and X bg The Adversarial Learning Module (ALM) is applied to adaptively activate the target area. The specific operations are as follows:

[0021]

[0022] Among them, h, w represent the height and width corresponding to the feature map, (i, j) represents the value of the i-th row and j-th column in the feature map, Γ * The operation represents selecting the feature map of the corresponding class of the image according to the label y. ξ takes a very small value to avoid the situation where the equation is meaningless. λ5 is a hyperparameter.

[0023] Step a7, forecasting the future graph M fpm Flatten and sort in descending order to get a one-dimensional vector V fpm , and then sent to the Oriented Learning Module (OLM) to calculate the loss:

[0024]

[0025]

[0026]

[0027] Where L represents the vector V fpm Length, m f and m b Represent the minimum foreground area ratio and the maximum background area ratio, respectively. and Represents vector P respectively f and vector P b The i-th element of ;

[0028] Step a8, based on the image features X of the foreground and background fg and X bg Each of them is subjected to global average pooling and Softmax operation to obtain the classification score S fg and S bg , the reinforcement learning module (RLM) will calculate the loss based on these two scores:

[0029]

[0030]

[0031] In step a9, the total loss of the network is calculated as:

[0032]

[0033] Among them, λ1-λ4 are the weights of the four losses;

[0034] Step a10: In the test phase, the image is input into the network to obtain the foreground prediction image X fpm , select the Top-k prediction maps according to the predicted category results, and take the average value to obtain the positioning map M lpoc , use the linear interpolation method to loc After adjusting to the original image size, the smallest border that can completely cover the foreground pixels is taken as the positioning result.

[0035] Compared with the prior art, the present invention has the following outstanding advantages:

[0036] 1. We develop a new adaptive region learning framework (AZL) for weakly supervised object localization, which strengthens the connection between the object foreground and background of the foreground prediction map.

[0037] 2. The present invention designs an adversarial learning module (ALM), which puts the foreground and background into an adversarial state and adaptively activates the object area.

[0038] 3. The present invention introduces an oriented learning module (OLM) to guide the network to highlight the accurate foreground and background of the target, thereby obtaining a better foreground prediction map.

[0039] 4. The present invention proposes a reinforcement learning module (RLM), which further explores the area where the target is located for the poorly performing foreground prediction map to obtain a more accurate foreground prediction result.

[0040] 5. Extensive experiments conducted on the CUB-200-2011 and ILSVRC datasets show that the proposed AZL of the present invention achieves significant and consistent performance improvements compared to other competing methods. Brief Description of the Drawings

[0041] Figure 1 It is a schematic diagram of the network structure of the neural network of the present invention. Detailed Embodiments

[0042] The following embodiments will, in conjunction with the accompanying drawings, elaborate on the technical solutions and beneficial effects of the present invention in detail.

[0043] The method of the present invention is abbreviated as AZL, which enhances the connection between the object foreground and background in weakly supervised object localization to improve the detection performance. First, the designed adversarial learning module (ALM) encourages activating the foreground while suppressing the background on the one hand, and controls the size of foreground activation on the other hand, thereby inducing the network to adversarially generate the foreground and background of the picture to the greatest extent. Second, an oriented learning module (OLM) is introduced to further refine the generated foreground and background to obtain more fine-grained object and background information. Third, a reinforcement learning module (RLM) is further proposed for the situation where the distinction effect between part of the foreground and background is not obvious, and the learning effect of the network is enhanced by further extracting the key information of the foreground.

[0044] The embodiments of the present invention specifically include the following steps:

[0045] Step 1, send the preprocessed picture to be detected and its corresponding candidate bounding boxes into the neural network; as Figure 1 shown, the neural network in the present invention mainly includes four parts: a backbone feature extraction network (VGG16, MobileNetV1, ResNet50, InceptionV3), an adversarial learning module (Adversarial Learning Mechanism, ALM), an oriented learning module (Oriented Learning Mechanism, OLM), and a reinforcement learning module (Reinforcement Learning Mechanism, RLM). In the preprocessing of the picture, first perform standardization processing on the picture, then scale the picture to a size of 256×256, and finally randomly crop it to a size of 224×224.

[0046] Step 2, the neural network processes the picture, outputs the probability values of each category corresponding to the picture to be detected during the training process, and outputs the coordinates, category, and score of the predicted box of the picture to be detected during the testing process.

[0047] The training method of the neural network includes the following steps:

[0048] Step a1, given a dataset set with image-level labels, divide the set into a training picture sample set and a testing picture sample set;

[0049] Step a2, arbitrarily select an image I from the training picture sample set, and input the image I and its corresponding image-level label y into the shallow feature extraction backbone network of the neural network to obtain the output of the network regarding the shallow features of the image

[0050] Step a3, the shallow features pass through the generator network to obtain the foreground prediction map X fpm ;

[0051] Step a4, determine the foreground prediction map M of the corresponding class of this image according to the label y fpm , and perform an inversion operation on it to obtain the corresponding background prediction map M bpm ;

[0052] Step a5, the shallow features continue to pass through the deep feature extraction backbone network of the neural network to obtain the image features The image classification loss is obtained through global average pooling Meanwhile, after obtaining the foreground features by performing element-wise multiplication with M fpm , the foreground classification loss is obtained through global average pooling

[0053] Step a6, perform element-wise multiplication with M bpm to obtain the background shallow features, and then pass through the deep feature extraction backbone network to obtain the background image features X bg , then subtract X bg to obtain the foreground image features X fg . Apply the adversarial learning module (ALM) to adaptively activate the target area for X fg and X bg . The specific operation is as follows:

[0054]

[0055] Among them, h and w represent the height and width of the feature map, (i, j) represents the value at the i-th row and j-th column in the feature map, and Γ * operation represents selecting the feature map of the corresponding class of the image according to the label y. ξ takes a very small value to avoid the situation where the equation is meaningless, and λ5 is a hyperparameter;

[0056] Step a7, flatten the foreground prediction map M fpm and sort it in descending order to obtain a one-dimensional vector V fpm , and then send it to the Orientation Learning Module (OLM) to calculate the loss:

[0057]

[0058]

[0059]

[0060] Among them, L represents the length of the vector V fpm of, m f and m b respectively represent the minimum foreground region ratio and the maximum background region ratio, and respectively represent the vector P f and the vector P b of the i-th element;

[0061] Step a8, from the image features X of the foreground and background fg and X bg each pass through global average pooling and Softmax operations to obtain classification scores S fg and S bg , and the Reinforcement Learning Module (RLM) will calculate the loss according to these two scores:

[0062]

[0063]

[0064] Step a9, the total loss of the network is calculated as:

[0065]

[0066] Among them, λ1 - λ4 are the weights of the four losses;

[0067] Step a10, in the test stage, the image is input into the network to obtain the foreground prediction map X fpm , select the top-k prediction maps according to the predicted class results, and take the average of them to obtain the localization map M loc , and use the method of linear interpolation to interpolate M locAfter adjusting to the original image size, the smallest bounding box that can completely cover the foreground pixels is taken as the positioning result.

[0068] The effects of the present invention are further illustrated by the following simulation experiments.

[0069] 1) Simulation conditions

[0070] The present invention is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the present invention is Python.

[0071] 2) Simulation content

[0072] The CUB-200-2011 and ILSVRC 2012 datasets are taken, and the network is trained according to the above steps and tested using the test set. Tables 1 and 2 show the localization and classification results of the present invention and other methods on the two datasets respectively. It can be found that compared with other methods, the present invention has the best effect. Among them, AZL (Ours) is the result of the present invention. The evaluation index Loc.Acc. represents the Top-1 / 5 and GT-known localization accuracy, and Cls.Acc. represents the classification accuracy. The present method achieves 77.56% / 81.31% (ResNet50) in terms of classification / localization performance on the CUB-200-2011 dataset and 78.00% / 58.62% (InceptionV3) in terms of classification / localization performance on the ILSVRC dataset. Compared with other methods, the localization accuracy is higher, which proves the better effect of classification and localization of the present invention.

[0073] Table 1 Comparison of localization accuracy and classification accuracy with the latest technical methods on CUB-200-2011

[0074]

[0075] Table 2 Comparison of localization accuracy and classification accuracy with the latest technical methods on ILSVRC

[0076]

[0077] Extensive experiments conducted on the CUB-200-2011 and ILSVRC datasets show that compared with other competing methods, the proposed AZL achieves significant and consistent performance improvements.

[0078] References:

[0079] [1]B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[0080] [2]X. Zhang, Y. Wei, J. Feng, Y. Yang, and T. S. Huang, “Adversarial complementary learning for weakly supervised object localization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[0081] [3]H. Xue, C. Liu, F. Wan, J. Jiao, X. Ji, and Q. Ye, “Danet: Divergent activation for weakly supervised object localization,” in IEEE International Conference on Computer Vision (ICCV), 2019.

[0082] [4]J. Mai, M. Yang, and W. Luo, “Erasing integrated learning: A simple yet effective approach for weakly supervised object localization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0083] [5]J.Choe, S.Lee, and H.Shim, “Attention-based dropout layer for weakly supervised single object localization and semantic segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.

[0084] [6]M.Meng, T.Zhang, Q.Tian, Y.Zhang, and F.Wu, “Foreground activation maps for weakly supervised object localization,” in IEEE International Conference on Computer Vision (ICCV), 2021.

[0085] [7]J.Xie, C.Luo, X.Zhu, Z.Jin, W.Lu, and L.Shen, “Online refinement of low-level feature based activation map for weakly supervised object localization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

[0086] [8]P.Wu, W.Zhai, and Y.Cao, “Background activation suppression for weakly supervised object localization,” 2022.

[0087] [9]C.Tan, G.Gu, T.Ruan, S.Wei, and Y.Zhao, “Dual-gradients localization framework with skip-layer connections for weakly supervised object localization,” IEEE Transactions on Multimedia (TMM), 2022.

[0088]

[10] M. Meng, T. Zhang, W. Yang, J. Zhao, Y. Zhang, and F. Wu, “Diverse complementary part mining for weakly supervised object localization,” IEEE Transactions on Image Processing (TIP), 2022.

[0089]

[11] K. Kumar Singh and Y. Jae Lee, “Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization,” in IEEE International Conference on Computer Vision (ICCV), 2017.

[0090]

[12] X. Zhang, Y. Wei, G. Kang, Y. Yang, and T. Huang, “Self-produced guidance for weakly-supervised object localization,” in European Conference on Computer Vision (ECCV), 2018.

[0091] The above embodiments are only for illustrating the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A weakly supervised object localization method based on adaptive domain learning, characterized in that Including the following steps: 1) Preprocess the image to be detected, and send the preprocessed image to be detected and its corresponding image-level label into the neural network; the neural network includes: a backbone feature extraction network, an adversarial learning module ALM, an orientation learning module OLM, and a reinforcement learning module RLM; 2) The neural network processes the image, outputs the probability value of each category corresponding to the image to be detected during the training process; outputs the coordinates, category, and score of the predicted box of the image to be detected during the test process; the adversarial learning module ALM induces the network to adversarially generate the foreground and background of the image to the greatest extent, and the orientation learning module OLM refines the foreground and background on this basis to obtain refined object and background information; The neural network processes the image, including the following steps: Step a1, given a dataset set with image-level labels, divide the set into a training image sample set and a test image sample set; Step a2, arbitrarily select an image I from the training image sample set, and input the image I and its corresponding image-level label y into the shallow feature extraction backbone network of the neural network Obtain the output of the network regarding the shallow features of the image Step a3, shallow features Through the generator network, a foreground prediction map X is obtained fpm ; Step a4, determine the foreground prediction map M of the corresponding class of the image according to the label y fpm , and perform a negation operation on it to obtain the corresponding background prediction map M bpm ; Step a5, shallow features Continue through the deep feature extraction backbone network of the neural network Obtain image features Obtain the image classification loss through global average pooling Meanwhile, After obtaining the foreground features by performing element-wise multiplication with M fpm and then passing through global average pooling to obtain the foreground classification loss Step a6, Perform element-wise multiplication with M bpm to obtain background shallow features, and then use the backbone network for deep feature extraction to obtain background image feature X bg , and then subtract X bg to obtain foreground image feature X fg ; Apply the adversarial learning module ALM to adaptively activate the region where the target is located for X fg and X bg . The specific operation is as follows: Among them, h and w represent the height and width corresponding to the feature map, (i, j) represents the value of the i-th row and the j-th column in the feature map, and Γ * The operation means selecting the feature map of the corresponding class of the image according to the label y. ξ takes a very small value to avoid the situation where the equation is meaningless, and λ5 is a hyperparameter; Step a7, flatten the foreground prediction map M fpm and sort it in descending order to obtain a one-dimensional vector V fpm , and then send it to the orientation learning module OLM to calculate the loss: Among them, L represents the length of vector V fpm , m f and m b respectively represent the minimum foreground region ratio and the maximum background region ratio, and respectively represent the i-th element of vector P f and vector P b ; Step a8, from the image features X of the foreground and background fg and X bg respectively obtain classification scores S through global average pooling and Softmax operations fg and S bg , and the reinforcement learning module RLM will calculate the loss based on these two scores: Step a9, the total loss of the network is calculated as: where λ1-λ4 are the weights of the four losses; Step a10, in the testing phase, the image is input into the network to obtain the foreground prediction map X fpm , select the top-k prediction maps according to the predicted class results, and take their average to obtain the localization map M loc , use the method of linear interpolation to process M loc , after adjusting it to the original image size, take the smallest bounding box that can completely cover the foreground pixels as the localization result.

2. The weakly supervised object localization method based on adaptive domain learning according to claim 1, characterized in that In step 1), for the preprocessing, first standardize the image, then scale the image to a size of 256×256, and finally randomly crop it to a size of 224×224.

3. The weakly supervised object localization method based on adaptive domain learning according to claim 1, wherein In step 1), the adversarial learning module ALM is used on the one hand to encourage the activation of the foreground while suppressing the background, and on the other hand to control the size of the foreground activation, so as to induce the network to adversarially generate the foreground and background of the image to the greatest extent; the orientation learning module OLM is used to refine on the already generated foreground and background to obtain more fine-grained object and background information; The reinforcement learning module RLM is used for the situation where the distinction effect between the foreground and background is not obvious in part, and enhances the learning effect of the network by extracting the key information of the foreground.

Images