Weakly supervised object detection method and system based on adaptive instance classifier refinement
Through the adaptive instance classifier refinement method, supervision information is generated using the fractional distribution and spatial similarity of candidate regions, which solves the problem of local optimal state in weakly supervised target detection and improves the detection ability of complete targets.
Patent Information
- Application Number
- CN202011454250.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-10
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-12-10
AI Technical Summary
The existing weak supervision object detection methods are prone to falling into the local optimal state and cannot effectively detect the complete target, mainly due to the inconsistency between weak supervision at image level and instance-level learning.
Adaptive instance classifier refinement method is adopted to extract the characteristics of the target candidate region through the backbone network, combine the multi-instance detection network and the adaptive instance classifier refinement framework, and gradually train the classifier to detect the complete target, and utilize the score distribution of the candidate region and Spatial similarity generates supervision information.
It effectively prevents the detector from falling into a local optimal state, improves the detection ability of complete targets in the image, and ensures that the detector can identify the target more accurately.
Smart Images

Figure CN112464877B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a computer vision recognition method, and in particular to a weakly supervised target detection method and system with adaptive instance classifier refinement. Background Art
[0002] Object detection is a basic and key research topic in the field of computer vision, which aims to generate bounding boxes of objects of a specific category in the input image. Therefore, object detection can be divided into two subtasks: object localization and object classification. The object localization task is responsible for determining the location of the object of interest, represented by a rectangular bounding box; the object classification task is responsible for determining the category to which the object belongs, using a series of scored labels to represent the probability that the object belongs to each category.
[0003] At present, deep learning-based object detectors have achieved great success, thanks to the development of convolutional neural networks and the availability of large-scale annotated data. However, the production of large-scale annotated datasets requires a lot of manpower and time, which limits the development of such detectors in practical applications.
[0004] To overcome this limitation, weakly supervised object detection methods that only require image-level annotations have attracted widespread attention. In practical applications, image-level labels are relatively easy to obtain or annotate. For example, after searching keywords on the Internet to obtain a large number of relevant images, a large number of images and corresponding labels can be obtained through manual screening. Current methods usually model the weakly supervised object detection problem as a multiple instance learning problem. In the multiple instance learning problem, a set of labeled bags is given, and each bag contains multiple unlabeled instances. In this way, the weakly supervised object detection problem is transformed into an instance classification problem (i.e., generated target candidate regions) given only the bag (i.e., image) labels. Inspired by the great success of convolutional neural networks, most of the latest methods combine multiple instance learning with convolutional neural networks and train them in an end-to-end manner, i.e., multiple instance detection networks. Usually, these methods train convolutional neural network-based classifiers by selecting the candidate regions with the highest confidence and their adjacent regions.
[0005] However, compared with fully supervised object detectors, multi-instance detection networks still have a large gap. The core problem is the inconsistency between image-level weak supervision and instance-level learning objectives. Convolutional neural networks only use part of the target to classify it, while target detection requires the positioning of the entire target. Therefore, convolutional neural network-based classifiers usually have a higher response to partial areas of the target, which results in candidate regions with higher confidence usually unable to surround the complete target, that is, falling into a local optimal state.
[0006] Therefore, how to provide a weakly supervised target detection method that avoids falling into a local optimal state is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention
[0007] In order to solve the problems in the prior art, the present invention provides a weakly supervised target detection method and system based on adaptive instance classifier refinement for computer vision recognition.
[0008] The present invention provides a weakly supervised target detection method with adaptive instance classifier refinement, comprising the following steps:
[0009] 1) Extract the features of the target candidate area in the image through the backbone network as the input of the classifier;
[0010] 2) Train the basic classifier through the multi-instance detection network to generate the supervision information for the next stage;
[0011] 3) The refined classifier is gradually trained through an adaptive instance classifier refinement framework to detect as many complete objects as possible.
[0012] As a further improvement of the present invention, the backbone network is modified by adding a spatial pyramid pooling layer and two fully connected layers to the original VGGNet.
[0013] As a further improvement of the present invention, in step 1), the backbone network extracts features of the target candidate area in the image, including the following process:
[0014] a1) Extract the convolution features of the image through the original VGGNet, and generate the target candidate region in the image through the selective search algorithm;
[0015] b1) Generate convolutional features corresponding to each candidate region through a spatial pyramid pooling layer;
[0016] c1) Generate feature vectors of candidate regions through two fully connected layers.
[0017] As a further improvement of the present invention, in step 2), the method for establishing the multi-instance detection network is as follows:
[0018] a2), feed the two feature vectors generated in step c1) into two branches with fully connected layers respectively to generate two matrices;
[0019] b2) Input the two matrices generated in step a2) into two Softmax layers respectively, and calculate along two different dimensions of category and candidate region to obtain two calculation results;
[0020] c2) perform element-wise multiplication of the two calculation results obtained in step b2) to generate a score for each candidate region, and calculate the possibility that the image contains each category of targets.
[0021] As a further improvement of the present invention, if it is in the training phase, step d2) is performed, the likelihood of each category of targets and image labels generated in step c2) are input, and the basic classifier is trained using the loss function; if it is in the testing phase, this step d2) is skipped.
[0022] As a further improvement of the present invention, in step 3), the method for establishing the adaptive instance classifier refinement framework is as follows:
[0023] a3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region;
[0024] b3), using the score distribution and spatial similarity of the candidate regions to generate supervision information, if in the training phase, proceed to step c3), if in the testing phase, proceed to step d3);
[0025] c3), input the score of each candidate region generated in step a3) and the supervision information generated in step b3), and use the loss function to train the classifier of each stage;
[0026] d3), repeating step a3), step b3) and step c3) K times, i.e. forming K instance classifier refinement stages;
[0027] e3), feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region;
[0028] f3), setting corresponding learnable parameters for the scores of each candidate region in step a3), obtaining a weighted average region score, and generating supervision information in the same manner as step b3);
[0029] g3), input the score of each candidate region generated in step e3) and the supervision information generated in step f3), and train in the same way as step c3).
[0030] As a further improvement of the present invention, the backbone network extracts target candidate region features in the following manner:
[0031] Input: X∈R H×W×3 For training / testing RGB images of fixed size, H represents the height of the image and W represents the width of the image;
[0032] Output: F∈R P×4096 is the feature vector extracted by the backbone network, P represents the number of candidate regions in the image, and 4096 is the dimension of the feature;
[0033] a1) Extract the convolution features of the image through the original VGGNet, and generate the target candidate region in the image through the selective search algorithm;
[0034] b1) Generate convolutional features corresponding to each candidate region through a spatial pyramid pooling layer;
[0035] c1) Generate feature vectors of candidate regions through two fully connected layers.
[0036] As a further improvement of the present invention, the method for establishing the multi-instance detection network is as follows:
[0037] Input: F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image, C is the number of target categories, when the image contains the c′th target, x c′ =1, otherwise x c′ =0;
[0038] Output: is the score of each candidate region, that is, the possibility that the region belongs to each category; φ c The possibility that the image contains each category of targets;
[0039] a2) Feed the feature vectors generated in step c1) into two branches with fully connected layers respectively to generate two matrices x cls and x det ;
[0040] b2) The matrix x generated in step a2) cls and x det They are input into two Softmax layers respectively and calculated along two different dimensions: category and candidate region: and represents the probability that the p′th candidate region belongs to the c′th class, [σ(x det )] c′p′ represents the normalized contribution of the p′th candidate region to the image containing the c′th class target;
[0041] c2) and convert the σ(x cls ) and σ(x det ) performs element-wise multiplication to generate the score for each candidate region And calculate the possibility that the image contains each category of target
[0042] d2) If in the training phase, input φ generated in step c2) c and image label X, using the loss function Train the base classifier; skip this step if you are testing.
[0043] As a further improvement of the present invention, the adaptive instance classifier refinement framework includes K instance classifier refinement stages and 1 adaptive knowledge refinement stage.
[0044] As a further improvement of the present invention, the method for establishing the adaptive instance classifier refinement framework is as follows:
[0045] Input: For the kth instance classifier refinement stage: F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image; is the score of each candidate region in the previous stage, where the {C+1}th class represents the background; for the adaptive knowledge refinement stage (the {K+1}th stage): F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image; is the score of each candidate region in the refinement stage of the K instance classifier;
[0046] Output: For the kth instance classifier refinement stage: is the score of each candidate region; for the adaptive knowledge refinement stage (the {K+1}th stage): is the score for each candidate region;
[0047] a3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region
[0048] b3) Generate supervision information using the score distribution and spatial similarity of candidate regions and The specific calculation method is as follows: First, Sort along the regional dimension from large to small to obtain the sorted score matrix ψ k-1 and the corresponding order matrix ω k-1 ,Right now Then, for each positive class (i.e., x c′ =1) number of core instances If the value is greater than the threshold N t ,but Annotation and c″≠c′; Finally, the area adjacent to the core instance is also marked as a positive example, that is, the IOU is greater than the threshold I t ;
[0049] c3) If in the training phase, input the generated and supervision generated in step b3) Using loss function Train the classifier at each stage, where Equal to the regional score; if in the testing phase, skip this step;
[0050] d3), repeating step a3), step b3) and step c3) K times, i.e. forming K instance classifier refinement stages;
[0051] e3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region
[0052] f3) Set the corresponding learnable parameter α k , and obtain the weighted average regional score Where ε is a small positive value to ensure that the denominator is meaningful and generate supervision information The steps are the same as step b3);
[0053] g3) If in the training phase, input the generated and supervision generated in step f3) Using loss function Train the classifier at each stage, where Equal to the regional score; if in the testing phase, skip this step.
[0054] The present invention also provides a weakly supervised target detection system with adaptive instance classifier refinement, including a readable storage medium, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, they are used to implement any of the methods described above.
[0055] The beneficial effect of the present invention is that by utilizing the score distribution and spatial similarity of the candidate regions, as many complete objects in the image as possible can be detected, thereby alleviating the problem of the detector falling into a local optimal state. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 It is a flow chart of a weakly supervised target detection method of the present invention with adaptive instance classifier refinement. DETAILED DESCRIPTION
[0057] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.
[0058] like Figure 1 As shown in FIG, a weakly supervised target detection method based on adaptive instance classifier refinement includes three parts: a backbone network, a multi-instance detection network, and an adaptive instance classifier refinement framework. The specific steps are as follows:
[0059] 1) Extract the features of the target candidate area in the image through the backbone network as the input of the classifier;
[0060] 2) Train the basic classifier through the multi-instance detection network to generate the supervision information for the next stage;
[0061] 3) The refined classifier is gradually trained through an adaptive instance classifier refinement framework to detect as many complete objects as possible.
[0062] The backbone network described in step 1) is modified by adding a spatial pyramid pooling layer and two fully connected layers to the original VGGNet (Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." ArXiv. 2014.).
[0063] The backbone network extracts the features of the target candidate region as follows:
[0064] Input: X∈R H×W×3 For training / testing RGB images of fixed size, H represents the height of the image and W represents the width of the image.
[0065] Output: F∈R P×4096 is the feature vector extracted by the backbone network, P represents the number of candidate regions in the image, and 4096 is the dimension of the feature.
[0066] Step a1) extract the convolution features of the image through the original VGGNet, and generate the target candidate region in the image through the selective search algorithm;
[0067] Step b1) generating convolutional features corresponding to each candidate region through a spatial pyramid pooling layer;
[0068] Step c1) generates feature vectors of candidate regions through two fully connected layers.
[0069] At this point, the backbone network has been established, the feature vectors of the candidate regions have been extracted, and input has been provided for the classifier.
[0070] The multi-instance detection network described in step 2) is modified from the original WSDDN (Bilen, Hakan, and Andrea Vedaldi. "Weakly supervised deep detection networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.).
[0071] The specific method of establishing a multi-instance detection network is as follows:
[0072] Input: F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image, C is the number of target categories, when the image contains the c′th target, x c′ =1, otherwise x c′ =0.
[0073] Output: is the score of each candidate region, that is, the possibility that the region belongs to each category; φ c is the probability that the image contains each category of objects.
[0074] Step a2) Feed the feature vectors generated in step c1) into two branches with fully connected layers to generate two matrices x cls and x det ;
[0075] Step b2) converts the matrix x generated in step a2) into cls and x det They are input into two Softmax layers respectively and calculated along two different dimensions: category and candidate region: and represents the probability that the p′th candidate region belongs to the c′th class, [σ(x det )] c′p′ represents the normalized contribution of the p′th candidate region to the image containing the c′th class target;
[0076] Step c2) converts the σ(x) generated in step b2) into cls ) and σ(x det ) performs element-wise multiplication to generate the score for each candidate region And calculate the possibility that the image contains each category of target
[0077] Step d2) During the training phase, input φ generated in step c2) cand image label X, using the loss function Train the base classifier; during testing, skip this step.
[0078] At this point, the establishment of the multi-instance detection network has been completed, the basic classifier has been trained, and supervision information has been provided for the next stage.
[0079] The adaptive instance classifier refinement framework described in step 3) includes K instance classifier refinement stages and 1 adaptive knowledge refinement stage.
[0080] The specific method of establishing the adaptive instance classifier refinement framework is as follows:
[0081] Input: For the kth instance classifier refinement stage: F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image; is the score of each candidate region in the previous stage, where the {C+1}th class represents the background. For the adaptive knowledge refinement stage (the {K+1}th stage): F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image; is the score of each region proposal in the refinement stage of the K-instance classifier.
[0082] Output: For the kth instance classifier refinement stage: is the score of each candidate region. For the adaptive knowledge refinement stage (the {K+1}th stage): is the score for each candidate region.
[0083] Step a3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer and calculate along the category dimension to obtain the score of each candidate region
[0084] Step b3) Generate supervision information using the score distribution and spatial similarity of the candidate regions and The specific calculation method is as follows: First, Sort along the regional dimension from large to small to obtain the sorted score matrix ψ k-1 and the corresponding order matrix ω k-1 ,Right now Then, for each positive class (i.e., x c′ =1) number of core instances If the value is greater than the threshold Nt ,but Annotation c″≠c′; Finally, the area adjacent to the core instance is also marked as a positive example, that is, the IOU is greater than the threshold I t ;
[0085] Step c3) During the training phase, input the generated and supervision generated in step b3) Using loss function Train the classifier at each stage, where Equal to the regional score; during the testing phase, skip this step;
[0086] Step d3) repeats step a3), step b3) and step c3) K times, thus forming K instance classifier refinement stages.
[0087] Step e3) feeds the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculates along the category dimension to obtain the score of each candidate region
[0088] Step f3) is Set the corresponding learnable parameter α k , and obtain the weighted average regional score Where ε is a small positive value to ensure that the denominator is meaningful and generate supervision information The steps are the same as step b3);
[0089] Step f3) Input the data generated in step e3) and supervision generated in step f3) The rest is the same as step c3).
[0090] This completes the establishment of the adaptive instance classifier refinement framework.
[0091] Figure 1 The implementation process of the method of the present invention includes three parts: backbone network, multi-instance detection network and adaptive instance classifier refinement framework. Among them, the backbone network is responsible for extracting the features of the target candidate area in the image, the multi-instance detection network is used to train the basic classifier and generate the supervision information for the next stage, and the adaptive instance classifier refinement framework gradually trains the refined classifier to detect as many complete targets as possible.
[0092] The present invention provides a weakly supervised target detection method based on adaptive instance classifier refinement. The backbone network extracts features of target candidate regions in an image as input to the classifier. The multi-instance detection network trains basic classifiers and generates supervision information for the next stage. The adaptive instance classifier refinement framework uses region score distribution and spatial similarity to gradually train refined classifiers to detect as many complete targets as possible. This provides a solution for alleviating the weakly supervised target detector from falling into a local optimal state, and has great application and commercial value.
[0093] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A weakly supervised object detection method with adaptive instance classifier refinement, characterized in that: The following steps are involved: 1) Extract the features of the target candidate area in the image through the backbone network as the input of the classifier; 2) Train the basic classifier through the multi-instance detection network to generate the supervision information for the next stage; 3) gradually train the refined classifiers through an adaptive instance classifier refinement framework to detect as many complete objects as possible; The backbone network is modified by adding a spatial pyramid pooling layer and two fully connected layers to the original VGGNet; In step 2), the method for establishing the multi-instance detection network is as follows: a2), feed the two feature vectors generated in step c1) into two branches with fully connected layers respectively to generate two matrices; b2) Input the two matrices generated in step a2) into two Softmax layers respectively, and calculate along two different dimensions of category and candidate region to obtain two calculation results; c2), performing element-by-element multiplication of the two calculation results obtained in step b2) to generate a score for each candidate region, and calculating the possibility that the image contains each category of objects; If it is in the training phase, proceed to step d2), input the probability of each category of targets and image labels generated in step c2), and use the loss function to train the basic classifier; if it is in the testing phase, skip this step d2); In step 3), the method for establishing the adaptive instance classifier refinement framework is as follows: a3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region; b3), using the score distribution and spatial similarity of the candidate regions to generate supervision information, if in the training phase, proceed to step c3), if in the testing phase, proceed to step d3); c3), input the score of each candidate region generated in step a3) and the supervision information generated in step b3), and use the loss function to train the classifier of each stage; d3), repeating step a3), step b3) and step c3) K times, i.e. forming K instance classifier refinement stages; e3), feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region; f3), setting corresponding learnable parameters for the scores of each candidate region in step a3), obtaining a weighted average region score, and generating supervision information in the same manner as step b3); g3), input the score of each candidate region generated in step e3) and the supervision information generated in step f3), and train in the same way as step c3); The backbone network extracts target candidate region features in the following way: Input: X∈R H×W×3 For training / testing RGB images of fixed size, H represents the height of the image and W represents the width of the image; Output: F∈R P×4096 is the feature vector extracted by the backbone network, P represents the number of candidate regions in the image, and 4096 is the dimension of the feature; a1) Extract the convolution features of the image through the original VGGNet, and generate the target candidate region in the image through the selective search algorithm; b1) Generate convolutional features corresponding to each candidate region through a spatial pyramid pooling layer; c1) Generate feature vectors of candidate regions through two fully connected layers.
2. The method for weakly supervised target detection using adaptive instance classifier refinement according to claim 1, characterized in that: The method for establishing the multi-instance detection network is as follows: Input: F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image, C is the number of target categories, when the image contains the c′th target, x c′ =1, otherwise x c′ =0; Output: is the score of each candidate region, that is, the possibility that the region belongs to each category; φ c The possibility that the image contains each category of targets; a2) Feed the feature vectors generated in step c1) into two branches with fully connected layers respectively to generate two matrices x cls and x det ; b2) The matrix x generated in step a2) cls and x det They are input into two Softmax layers respectively and calculated along two different dimensions: category and candidate region: and represents the probability that the p′th candidate region belongs to the c′th class, [σ(x det )] c′p′ represents the normalized contribution of the p′th candidate region to the image containing the c′th class target; c2) and convert the σ(x cls ) and σ(x det ) performs element-wise multiplication to generate the score for each candidate region And calculate the possibility that the image contains each category of target d2) If in the training phase, input φ generated in step c2) c and image label X, using the loss function Train the base classifier; If you are in the testing phase, skip this step.
3. The method for weakly supervised target detection using adaptive instance classifier refinement according to claim 2, characterized in that: The adaptive instance classifier refinement framework includes K instance classifier refinement stages and 1 adaptive knowledge refinement stage; The method for establishing the adaptive instance classifier refinement framework is as follows: Input: For the kth instance classifier refinement stage: F∈R P×4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image; is the score of each candidate region in the previous stage, where the {C+1}th class represents the background; for the adaptive knowledge refinement stage, the {K+1}th stage: F∈R P × 4096 is the feature vector extracted by the backbone network; X = [x1, x2, ..., x C ] T is the label of the image; is the score of each candidate region in the refinement stage of the K instance classifier; Output: For the kth instance classifier refinement stage: is the score of each candidate region; for the adaptive knowledge refinement stage, that is, the {K+1}th stage: is the score for each candidate region; a3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region b3) Generate supervision information using the score distribution and spatial similarity of candidate regions and The specific calculation method is as follows: First, Sort along the regional dimension from large to small to obtain the sorted score matrix ψ k-1 and the corresponding order matrix ω k-1 ,Right now Then, calculate each positive class, that is, x c′ =1 number of core instances If the value is greater than the threshold N t ,but Annotation and Finally, the area adjacent to the core instance is also marked as a positive example, that is, the IOU is greater than the threshold I t ; c3) If in the training phase, input the generated and supervision generated in step b3) Using loss function Train the classifier at each stage, where Equal to the regional score; if in the testing phase, skip this step; d3), repeating step a3), step b3) and step c3) K times, i.e. forming K instance classifier refinement stages; e3) Feed the feature vector generated in step c1) into a fully connected layer and a Softmax layer, and calculate along the category dimension to obtain the score of each candidate region f3) Set the corresponding learnable parameter α k , and obtain the weighted average regional score Where ε is a small positive value to ensure that the denominator is meaningful and generate supervision information The steps are the same as step b3); g3) If in the training phase, input the generated and supervision generated in step f3) Using loss function Train the classifier at each stage, where Equal to the regional score; if in the testing phase, skip this step.
4. A weakly supervised object detection system with adaptive instance classifier refinement, characterized by: The method comprises a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement the method according to any one of claims 1 to 3 when executed by a processor.
