A Weakly Supervised Object Localization Method Based on Hybrid Erasure Strategy

Abstract

The present invention relates to a weakly supervised object localization method based on a hybrid erasing strategy, including: constructing a neural network, inputting an image into a first deep learning classification model to obtain a first feature map and generate a first activation map, and obtaining an erasing threshold according to the first activation map; performing a normalization and zero replacement operation on the first feature map to obtain a second feature map and generate a second activation map; obtaining the target region and target prediction probability of the second activation map; adding the second activation map and the first activation map to obtain a localization map, and screening the localization map to obtain a first-stage mask; inputting the image and the first-stage mask into a deep learning semantic segmentation model to obtain a third feature map and generate a third activation map; flipping and erasing the third activation map to obtain a fourth activation map, guiding the training of the deep learning semantic segmentation model, and continuing the normalization operation on the output result to obtain a set of localization boxes. The weakly supervised localization method of the present invention has strong scalability and high localization accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of object detection, and particularly to a weakly supervised object localization method based on a hybrid erasure strategy. Background Art

[0002] Vision is an important source for humans to obtain information. Research shows that humans obtain more than 80% of the external information through the visual channel. Images are an important information form in multimedia, vividly and intuitively carrying a large amount of information. Therefore, it is of great significance to study how to use computer systems to interpret images and achieve a similar understanding of the external world as the human visual system. Among them, object detection is a basic and important problem in computer vision. The purpose of object detection is to accurately find the location of the object and identify the category of the object in a given picture through a computer system. Object detection has application requirements in many fields, and widely studied object detections include face detection, pedestrian detection, vehicle detection and other important objects.

[0003] In recent years, deep learning technology has developed rapidly and received more and more attention in the field of object detection, overcoming many traditional object detection methods based on features. Currently, there are two major categories of fully supervised object detection algorithms based on deep learning: regression-based one-stage object detection algorithms and candidate box-based two-stage object detection algorithms. The candidate box-based two-stage object detection algorithm is divided into a candidate box extraction module and a classification module. Representative algorithms include methods such as R-CNN and Faster-RCNN. The regression-based one-stage object detection algorithm directly learns the location and category of the object through regression. Representative algorithms include algorithms such as YOLO and SSD. Since the regression-based object detection algorithm has only one step compared to the candidate box-based object detection algorithm, the speed of object detection is greatly accelerated. However, the above-mentioned fully supervised object detection requires a large amount of expensive bounding box annotation information. Although annotated data improves the performance of deep learning-based object detection algorithms, it also consumes a large amount of time and labor costs. There is a huge amount of unannotated data in the real world, which cannot be used to train fully supervised object detection algorithms.

[0004] Weakly-supervised localization algorithms do not require expensive bounding box annotation information. They only need image-level class labels to learn the location information of the objects in the image. Therefore, there is great research and application value in how to achieve high-precision weakly-supervised localization. Currently, research on weakly-supervised object localization trains a deep learning classification model and then weights the last convolutional feature map through the special class weights of the last classifier to obtain the localization map. However, since the classification model tends to identify the most discriminative regions of the objects in the image, it leads to unsuccessful localization. To address this problem, a simple and effective idea is to cover the most discriminative regions of the object according to the weights of the localization map, forcing the network to use other positions of the object to learn classification information, thus shifting from localizing to partial regions to localizing to the overall target region. Currently, some research teams have conducted corresponding research, but there are still the following defects in the existing technologies: ignoring the target spatial information contained in the class activation mapping generated by the classification network; ignoring the problem that the erasing branch in the erasing strategy may carry noise; ignoring the problem that the labels of the pseudo-label-based strategy are inaccurate; ignoring other possible application scenarios of the erasing strategy; low localization accuracy and long time. Summary of the Invention

[0005] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a weakly-supervised localization method based on a hybrid erasing strategy to solve the problems of narrow target position search domain, low accuracy, and small localization area in existing weakly-supervised localization methods.

[0006] To achieve the above purpose, the present invention provides a weakly-supervised object localization method based on a hybrid erasing strategy, including:

[0007] Construct a neural network, where the neural network includes a first deep learning classification model, a second deep learning classification model, and a deep learning semantic segmentation model;

[0008] Input an image into the first deep learning classification model, obtain the first feature map output by the fully convolutional network in the first deep learning classification model, and input the first feature map into the first classifier through a global average pooling layer to generate a first activation map;

[0009] Obtain an erasing threshold according to the sum of the pixel activation values of the first activation map;

[0010] Perform a normalization operation on the first feature map, and perform a zero-replacement operation on the pixel values in the first feature map that are greater than the erasing threshold to obtain a second feature map. Input the second feature map into the second classifier through a global average pooling layer to generate a second activation map;

[0011] Obtain a mask according to the second activation map, and overlay the mask on the original image of the image to obtain the target region of the second activation map;

[0012] Input the second activation map into the second deep learning classification model, and obtain the target prediction probability of the target region of the second activation map through the sigmoid function;

[0013] Use the target prediction probability as the weight of the second activation map, and add the second activation map to the first activation map to obtain a localization map;

[0014] Perform screening according to the localization map to obtain a first-stage mask;

[0015] Input the image and the first-stage mask into the deep learning semantic segmentation model, obtain the third feature map output by the fully convolutional network in the deep learning semantic segmentation model, and generate a third activation map according to the third feature map;

[0016] Flip and erase the third activation map to obtain a fourth activation map, and use the quotient of the third activation map and the fourth activation map as a loss function to guide the training of the deep learning semantic segmentation model;

[0017] Perform a normalization operation on the output result of the deep learning semantic segmentation model to obtain a set of localization boxes.

[0018] Preferably, the first deep learning classification model includes: a fully convolutional network, a global average pooling layer, and two independent fully connected layer networks.

[0019] Preferably, the deep learning semantic segmentation model includes: a fully convolutional network and a fully connected layer network.

[0020] Preferably, before inputting the image into the first deep learning classification model, it further includes:

[0021] Train the first deep learning classification model so that the first deep learning classification model has classification ability.

[0022] Further preferably, training the first deep learning classification model so that the first deep learning classification model has classification ability specifically includes:

[0023] Use an image dataset with class labels to train the first deep learning classification model, and use the cross-entropy function as a loss function so that the first deep learning classification model has classification ability.

[0024] An embodiment of the present invention provides a weakly supervised object localization method based on a hybrid erasure strategy. First, a neural network is constructed, and then an image is input into a first deep learning classification model to obtain a first feature map. A first activation map is generated based on the first feature map, and an erasure threshold is obtained according to the first activation map. A normalization operation and a zero replacement operation are performed on the first feature map to obtain a second feature map, and a second activation map is generated through a global average pooling layer. A mask is obtained according to the second activation map, and the mask is superimposed on the original image to obtain the target region of the second activation map. The second activation map is input into a second deep learning classification model, and the target prediction probability of the second activation map is obtained through a normalization function. The second activation map is added to the first activation map to obtain a localization map. The localization map is screened to obtain a first-stage mask. The image and the first-stage mask are input into a deep learning semantic segmentation model to obtain a third feature map, and a third activation map is generated according to the third feature map. The third activation map is flipped and erased to obtain a fourth activation map, and the quotient of the third activation map and the fourth activation map is used as a loss function to guide the training of the deep learning semantic segmentation model. A normalization operation is performed on the output result of the deep learning semantic segmentation model to obtain a set of localization boxes. The present invention introduces an adaptive threshold strategy to guide erasure, avoiding manual threshold selection. At the same time, through the flipped erasure strategy, the problem of localization errors caused by unclear class information that may occur in the pseudo-label training stage is avoided. The solution of the present invention can be easily integrated with existing weakly supervised localization erasure technologies, and has the characteristics of strong scalability, high localization accuracy, and low computational resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 FIG. is a schematic flowchart of a weakly supervised object localization method based on a hybrid erasure strategy provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0027] An embodiment of the present invention provides a weakly supervised object localization method based on a hybrid erasure strategy. First, a neural network is constructed, and then an image is input into a first deep learning classification model to obtain a first feature map. A first activation map is generated based on the first feature map, and an erasure threshold is obtained according to the first activation map. A normalization operation and a zero replacement operation are performed on the first feature map to obtain a second feature map, and a second activation map is generated through a global average pooling layer. A mask is obtained according to the second activation map, and the mask is superimposed on the original image to obtain the target area of the second activation map. The second activation map is input into a second deep learning classification model, and the target prediction probability of the second activation map is obtained through a normalization function. The second activation map is added to the first activation map to obtain a localization map. The localization map is screened to obtain a first-stage mask. The image and the first-stage mask are input into a deep learning semantic segmentation model to obtain a third feature map, and a third activation map is generated according to the third feature map. A flipped erasure is performed on the third activation map to obtain a fourth activation map, and the quotient of the third activation map and the fourth activation map is used as a loss function to guide the training of the deep learning semantic segmentation model. A normalization operation is performed on the output result of the deep learning semantic segmentation model to obtain a set of localization boxes. The present invention introduces an adaptive threshold strategy to guide erasure, avoiding manual selection of thresholds. At the same time, through the flipped erasure strategy, the problem of localization errors caused by unclear category information that may occur in the pseudo-label training stage is avoided. The solution of the present invention can be easily integrated with existing weakly supervised localization erasure technologies, and has the characteristics of strong scalability, high localization accuracy, and low computational resources.

[0028] Figure 1 It is a schematic flowchart of a weakly supervised localization method based on a hybrid erasure strategy provided by an embodiment of the present invention. As Figure 1 shown, the method includes the following steps:

[0029] Step 101, construct a neural network.

[0030] Specifically, the solution of the present invention needs to first construct a neural network, which includes: a first deep learning classification model, a second deep learning classification model, and a deep learning semantic segmentation model; wherein, the first deep learning classification model includes: a fully convolutional network, a global average pooling layer, and two independent fully connected layer networks; the second deep learning classification model is a pre-trained deep learning classification model; the deep learning semantic segmentation model is used for pseudo-label training and includes: a fully convolutional network and a fully connected layer network for outputting semantic segmentation results. The fully convolutional network can use any classic network, such as network models like VGG, inception, resnet, etc. When using a network structure with multiple fully connected layers, such as VGG, the fully connected layers need to be removed and then used as a fully convolutional network.

[0031] Step 102: Input the image into the first deep learning classification model to obtain the first feature map output by the fully convolutional network in the first deep learning classification model, and input the first feature map into the first classifier through the global average pooling layer to generate the first activation map.

[0032] Specifically, input the image into the first deep learning classification model. The image passes through the fully convolutional network in the first deep learning classification model to extract the first feature map, and then the first feature map is input into the global average pooling layer. The global average pooling layer technology is used to input the first feature map into the weighted sum of the weights of the first classifier on specific categories to generate the target first activation map.

[0033] Before inputting the image into the first deep learning classification model, it is also necessary to train the first deep learning classification model to enable it to have classification capabilities. The specific method is as follows: Use an image dataset with class annotations to train the first deep learning classification model, and use the cross-entropy function as the loss function during the training process. The model parameters can be initialized using a pre-trained model trained on a large dataset.

[0034] Step 103: Obtain the erasure threshold based on the sum of the pixel activation values of the first activation map, perform a normalization operation on the first feature map, and perform a zero replacement operation on the pixel values in the first feature map that are greater than the erasure threshold to obtain the second feature map. Input the second feature map into the second classifier through the global average pooling layer to generate the second activation map.

[0035] Specifically, obtain the erasure threshold based on the sum of the pixel activation values of the first activation map, perform a normalization operation on the first feature map, and perform a zero replacement operation on the pixel values in the first feature map that are greater than the erasure threshold to obtain the erased second feature map. Then input the second feature map into the second classifier through the global average pooling layer in the same way to generate the second activation map.

[0036] Step 104: Obtain the mask according to the second activation map, and overlay the mask on the original image of the image to obtain the target area of the second activation map.

[0037] Specifically, after restoring the second activation map to the original size of the image through bilinear interpolation, take the pixels with a confidence greater than a pre-set threshold to obtain the filtered mask, and overlay the mask on the original image of the image to obtain the target area indicated by the second activation map.

[0038] Step 105: Input the second activation map into the second deep learning classification model, and obtain the target prediction probability of the target area of the second activation map through the softmax function; use the target prediction probability as the weight of the second activation map, and add the second activation map and the first activation map to obtain the localization map.

[0039] Specifically, input the second activation map into the pre-trained second deep learning classification model, obtain the target prediction probability of the target region of the second activation map through the softmax function, and use the target prediction probability as the weight of the second activation map. Finally, add the weighted second activation map and the first activation map to obtain the final localization map.

[0040] Step 106: Screen according to the localization map to obtain the first-stage mask; input the image and the first-stage mask into the deep learning semantic segmentation model, obtain the third feature map output by the fully convolutional network in the deep learning semantic segmentation model, and generate the third activation map according to the third feature map.

[0041] Specifically, screen according to the localization map, further perform a normalization operation on the localization map, take the pixels with a confidence greater than a preset threshold, and obtain the filtered first-stage mask as the supervision information of the deep learning semantic segmentation model. Input the image and the first-stage mask into the deep learning semantic segmentation model to train the semantic segmentation network. During the training process, the present invention introduces a flipping and erasing strategy to enhance the semantic information of the target, thereby solving the problem of fuzzy foreground semantics of the target. Obtain the third feature map output by the fully convolutional network in the deep learning semantic segmentation model, and generate the third activation map according to the third feature map using the global average pooling layer technology as well.

[0042] Step 107: Perform flipping and erasing on the third activation map to obtain the fourth activation map, and use the quotient of the third activation map and the fourth activation map as the loss function to guide the training of the deep learning semantic segmentation model.

[0043] Specifically, perform flipping and erasing on the third activation map to obtain the fourth activation map. The specific operation is to subtract a full-1 matrix of the same size from the normalized third activation map to obtain the flipped fourth activation map. Calculate the quotient of the third activation map and the fourth activation map, and use it as an additional loss function to guide the training of the deep learning semantic segmentation model.

[0044] Step 108: Perform a normalization operation on the output result of the deep learning semantic segmentation model to obtain a set of localization boxes.

[0045] Specifically, in the testing stage, after performing a normalization operation on the output result of the deep learning semantic segmentation model, take the pixels with a confidence greater than a preset threshold, and obtain the set of localization boxes by taking the minimum bounding box of the largest connected region.

[0046] The following elaborates on the processing steps of the present invention through specific examples:

[0047] S1: Use the image dataset with class labels as the training set to train the first deep learning classification model, where N is the number of dataset samples, xi and y i respectively represent the i-th sample data and its annotation vector. The composition and order of the first deep learning classification model are: a fully convolutional network, a global average pooling layer, and two fully connected networks. The last fully connected network outputs the probability of predicting each category. During the training process, the input images are preprocessed, including multi-scale transformation, random cropping, horizontal flipping, etc.

[0048] S2. In the present invention, an erasure strategy is used to guide the neural network to learn the overall region of the target. First, it is assumed that the total number of predicted categories by the fully convolutional network is d categories, the probability vector output by the fully convolutional network is p, and the annotation vector of the c-th category is y c . Without loss of generality, it is assumed that the localization of the c-th category target is achieved on the first feature map S1 output by the fully convolutional network. S1 is sent to the global average pooling layer and the fully connected layer to predict the target category, and the weight w of the fully connected layer L1 on the target category c is obtained. c Multiply w c by the first feature map S1 to obtain the first activation map M1 of the first feature map S1 on the target category c. The specific formula can be expressed as:

[0049]

[0050] Subsequently, according to the sum of the pixel values in the first activation map M1, the original erasure threshold is adjusted, and an erasure operation of zero replacement is performed, which can be specifically expressed as:

[0051]

[0052] where γ is the initially set erasure threshold, and γ ′ is the updated erasure threshold. Subsequently, the second feature map S2 obtained after erasure through the fully convolutional network is input into the fully connected layer L2, and the target category is also predicted, and the classification loss function is calculated. For the classification result output by L2, the present invention adopts a strategy for noisy labels to calculate the loss. Specifically, the second activation map M2 obtained by multiplying the second feature map S2 and the weight of the fully connected layer L2 will be upsampled to the original image size, and dot-multiplied with the original image. The classification result is predicted by an additional classification network, and the probability of this classification result on the target class is used as the weight of the loss function of the L2 part to participate in the training. Additionally, in this embodiment, the cross-entropy function L is used as the loss function for classification. Then the cross-entropy loss between the probability vector P and the annotation vector y of the c-th category c can be expressed as:

[0053]

[0054] where, since the probability vector p and the annotation vector y of the c-th category cThey are all d-dimensional vectors. Therefore, when calculating the cross-entropy loss, it is necessary to calculate from the first dimension to the d-th dimension. Then, y ck represents the k-th element of the vector y c , and p k represents the k-th element of the vector p. In addition, log in the formula represents the log function. Finally, in the test stage, the final positioning map M is obtained by adding M1 and M2

[0055] mask = abs(sign(M - th))

[0056] The present invention obtains the mask mask through a threshold strategy. Mask is a binary matrix of 0 / 1. Specifically, first subtract the threshold th from the positioning map M, then input it into the sign function, map the values greater than 0 to 1, the values less than 0 to -1, and finally input it into the absolute value function abs to remove the negative numbers, thereby obtaining the binary mask mask for use as a pseudo-label

[0057] S3. Use the pseudo-label obtained in the above strategy as a pixel-level label and combine it with the initial image dataset as the training set to train the first deep learning classification model, where N is the number of dataset samples, x i and y ′ i represent the i-th sample data and its label respectively. The composition and order of the model are: fully convolutional network, global average pooling layer, and fully connected network. The global average pooling layer is used to generate the third activation map M3 after the fully convolutional network. Subtract a matrix of all 1s of the same size as M3 from the normalized M3, then multiply it with the original image, and generate the fourth activation map M4 after flipping and erasing in the same way. Use M3 and M4 to represent the activation of the foreground and background respectively, and use the quotient of the sum of the two as an additional loss function L scale to guide the training of the deep learning semantic segmentation model, which can be specifically expressed as:

[0058]

[0059] In addition, the fully connected layer is used to output the semantic segmentation prediction result after the fully convolutional network and calculate the loss function. This embodiment uses the cross-entropy loss function L mask to guide, which can be specifically expressed as:

[0060]

[0061] where y ij represents the network prediction result, The pseudo label is denoted as, and log represents the log function. Finally, the prediction result of the model is output in the test phase, and a binary mask is obtained through threshold segmentation. After the model predicts the mask, it can not only be used for localization but also for segmentation because the mask is at the pixel level. Further, the final bounding box is obtained through the boundaries of different connected components in the mask. Specifically, the present invention uses the cv2.findContours function in opencv to obtain different connected components in the mask, and takes the upper, lower, left, and right boundary lines of the connected components as the final bounding box.

[0062] An embodiment of the present invention provides a weakly supervised object localization method based on a hybrid erasing strategy. First, a neural network is constructed, and then the image is input into the first deep learning classification model to obtain the first feature map. The first activation map is generated according to the first feature map, and the erasing threshold is obtained according to the first activation map; a normalization operation and a zero replacement operation are performed on the first feature map to obtain the second feature map, and the second activation map is generated through the global average pooling layer; the mask is obtained according to the second activation map, and the mask is superimposed on the original image to obtain the target area of the second activation map; the second activation map is input into the second deep learning classification model, and the target prediction probability of the second activation map is obtained through the normalization function; the second activation map is added to the first activation map to obtain the localization map; the first stage mask is obtained by screening the localization map; the image and the first stage mask are input into the deep learning semantic segmentation model to obtain the third feature map, and the third activation map is generated according to the third feature map; the fourth activation map is obtained by flipping and erasing the third activation map, and the quotient of the third activation map and the fourth activation map is used as the loss function to guide the training of the deep learning semantic segmentation model; a normalization operation is performed on the output result of the deep learning semantic segmentation model to obtain the set of localization boxes. The present invention introduces an adaptive threshold strategy to guide erasing, avoiding manual selection of the threshold. At the same time, through the flipping erasing strategy, the problem of incorrect localization caused by unclear class information that may occur in the pseudo label training phase is avoided. The solution of the present invention can be easily integrated with the existing weakly supervised localization erasing technology, and has the characteristics of strong scalability, high localization accuracy, and low computing resources.

[0063] The above specific embodiments have further detailed the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above is only the specific embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A weakly supervised object localization method based on a hybrid erasure strategy, characterized in that The method includes: Constructing a neural network, which includes a first deep learning classification model, a second deep learning classification model, and a deep learning semantic segmentation model; Inputting an image into the first deep learning classification model, obtaining a first feature map output by the fully convolutional network in the first deep learning classification model, and inputting the first feature map into a first classifier through a global average pooling layer to generate a first activation map; Obtaining an erasure threshold according to the sum of pixel activation values of the first activation map; Performing a normalization operation on the first feature map, and performing a zero replacement operation on the pixel values in the first feature map that are greater than the erasure threshold to obtain a second feature map. Inputting the second feature map into a second classifier through a global average pooling layer to generate a second activation map; Obtaining a mask according to the second activation map, and superimposing the mask on the original image of the image to obtain the target region of the second activation map; Inputting the second activation map into the second deep learning classification model, and obtaining the target prediction probability of the target region of the second activation map through a sigmoid function; Taking the target prediction probability as the weight of the second activation map, and adding the second activation map and the first activation map to obtain a localization map; Performing screening according to the localization map to obtain a first-stage mask; Inputting the image and the first-stage mask into the deep learning semantic segmentation model, obtaining a third feature map output by the fully convolutional network in the deep learning semantic segmentation model, and generating a third activation map according to the third feature map; Performing flip erasure on the third activation map to obtain a fourth activation map, and using the quotient of the third activation map and the fourth activation map as a loss function to guide the training of the deep learning semantic segmentation model; Performing a normalization operation on the output result of the deep learning semantic segmentation model to obtain a set of localization boxes.

2. The weakly-supervised object localization method based on the hybrid erasure strategy according to claim 1, wherein The first deep learning classification model includes: a fully convolutional network, a global average pooling layer, and two independent fully connected layer networks.

3. The weakly supervised object localization method based on the hybrid erasure strategy according to claim 1, wherein, The deep learning semantic segmentation model includes: a fully convolutional network and a fully connected layer network.

4. The weakly supervised object localization method based on the hybrid erasure strategy according to claim 1, wherein Before inputting the image into the first deep learning classification model, it further includes: Training the first deep learning classification model so that the first deep learning classification model has classification ability.

5. The weakly supervised object localization method based on the hybrid erasure strategy according to claim 4, wherein Training the first deep learning classification model so that the first deep learning classification model has classification ability specifically includes: Training the first deep learning classification model using an image dataset with class labels, and using a cross-entropy function as a loss function so that the first deep learning classification model has classification ability.

Images