A weakly supervised target detection method and system
By screening and optimizing the proposal box, combining low-level supervision information and similarity concepts, local optimization and noise problems in weak supervision target detection are solved, detection accuracy and stability are improved, and high-quality target detection is achieved.
Patent Information
- Application Number
- CN202210302852.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-03-25
AI Technical Summary
The existing weakly supervised object detection method is prone to falling into local optimal solutions, and due to the lack of instance-level constraints, the detector only focuses on local areas, making it difficult to accurately locate non-rigid multi-pose objects, and there is a large amount of noise in the proposed box, which affects training stability. and computing resources.
Before multi-example learning, filter high-quality proposal boxes, add low-level supervision information, generate high-quality proposal boxes through Grad-CAM, and introduce the concept of similarity in the iterative optimization process, filter out the proposal boxes of complete objects as pseudo- Supervise information and optimize bounding box regression network.
It improves the accuracy of target detection, reduces noise interference, avoids local optimal problems, improves the integrity and stability of detection, and saves the cost of labeling instance-level information.
Smart Images

Figure CN114648665B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision image processing, and in particular relates to a weakly supervised target detection method and system. Background Art
[0002] The purpose of weakly supervised object detection is to train an object detector with only image-level annotations. It is different from fully supervised object detection that requires instance-level annotations (the center coordinates, height, and width of the maximum circumscribed rectangle of the object in the image need to be annotated). Annotating instance-level information requires a lot of manpower, material resources, and financial resources. However, the cost of annotating with image categories is obviously lower, and we can also crawl a large number of images with category annotations from web search engines, social media, etc. A large amount of training data can improve the performance of object detection. Obviously, these cheap and easily available images with only category annotations are a good thing for the field of object detection. Therefore, using weakly supervised methods to learn object detectors has received more and more attention in academia, and this is also an urgent need in the industry.
[0003] At present, weakly supervised object detection is usually based on the process of multiple instance learning (MIL), which makes weakly supervised object detection easy to fall into the problem of local optimal solution. Generally speaking, due to the lack of instance-level constraints, the use of image category-level constraints will cause weakly supervised object detection to focus only on local areas, because classification only requires local information (for example, for a person or cat in a picture, the classifier only needs to focus on their faces), while the goal of detection is to accurately locate the maximum enclosing rectangle of the object, which is the gap between classification and detection. This local focus problem is particularly serious in the detection of objects with large intra-class differences. These categories generally include non-rigid multi-pose objects such as people and animals, because such objects generally have certain invariant appearances, such as faces.
[0004] At the same time, due to the lack of instance-level rectangular annotation information, current methods all use a large number of object proposals to ensure the recall rate, which will result in a large amount of noise in the proposal frame (a small part of the object, background, etc.), which will not only lead to unstable training, but also consume a lot of GPU computing resources. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a weakly supervised target detection method and system in view of the deficiencies in the above-mentioned prior art. Before entering the MIL multi-instance learning, the proposal boxes generated by the traditional method SS (Selective Search) are first screened, and the high-quality proposal boxes are screened and sent to the subsequent multi-instance learning process. Then, in the multi-instance learning process, low-level (color, texture, etc.) supervision information is added on the basis of the original image-level label supervision information to better train the network, thereby improving the detection accuracy of the network.
[0006] The present invention adopts the following technical solutions:
[0007] A weakly supervised target detection method comprises the following steps:
[0008] S1, read image data and image-level labels, where the image-level labels are image-level classification labels that only contain object categories in the image, and divide the image data into a training data set and a test data set;
[0009] S2, using the selective search algorithm to generate candidate boxes in the training data set read in step S1, and then generating high-quality object proposal boxes based on the gradient weighted class activation mapping method;
[0010] S3, input the training data set read in step S1 into the VGG16 convolutional neural network for feature extraction, and generate a feature matrix of the same shape through the ROI Pooling layer for the extracted proposal box features;
[0011] S4, the image-level label read in step S1 and the feature matrix obtained in step S3 are matched one by one, multiple-instance learning is performed, and a MIL detector is constructed;
[0012] S5, adding low-level supervision information to the object proposal box obtained in step S2, and establishing an instance classifier for optimization iteration, using the proposal box with the highest score as a pseudo label to train the bounding box regression network;
[0013] S6, determining the loss function of the MIL detector obtained in step S4 and the bounding box regression network obtained in step S5;
[0014] S7, adjusting the hyperparameters of the bounding box regression network in step S6 to obtain a weakly supervised detection model;
[0015] S8, using the training data set obtained in step S1 to train the weak supervision detection model obtained in step S7, to obtain a trained weak supervision detection model;
[0016] S9. The test data set obtained in step S1 uses a selective search algorithm to generate candidate boxes, and then uses the weakly supervised detection model trained in step S8 to classify and regress the candidate boxes to obtain the final target detection box.
[0017] Specifically, in step S2, the first-stage image-level classifier is first used to predict the n potential object categories with the highest scores in each test image, and then the second-stage proposal box level classifier is used to obtain the final proposal box for detection. The loss function of the first stage is as follows:
[0018]
[0019] Where C is the total number of image categories, y i is the label of the i-th image, P i is the prediction result of the i-th sigmoid classifier.
[0020] Specifically, step S4 is as follows:
[0021] S401, for a given image and its corresponding label, a series of object proposal frames are obtained through step S2; then the object proposal frames are sent to the classification data stream and the detection data stream, and two data matrices are obtained through two fully connected layers respectively; the two data matrices are respectively passed through two softmax operators to generate the classification score and detection score of each proposal frame;
[0022] S402: perform element-wise multiplication of the classification score and the detection score obtained in step S401 to obtain the final proposal box score, and add the scores of all proposal boxes in the dimension of R to obtain the image-level prediction score of each category.
[0023] Furthermore, in step S402, the loss function of the MIL detector is:
[0024]
[0025] Among them, P c is the image-level prediction score for the c-th category, y c The label of the image.
[0026] Specifically, step S5 is as follows:
[0027] S501: Establish K instance classifiers based on the MIL detector and calculate the similarity O of the low-level features of the proposal box r. bu (r), the similarity O bu (r) and the category score obtained by the previous classifier Perform weighted addition, take the proposal box with the highest score after addition as the proposal box with the complete object target, and take the first n proposal boxes with the highest scores as the pseudo-supervisory information for the next iteration of training the instance classifier. Train K instance classifiers for a total of K iterations;
[0028] S502. Connect a bounding box regressor after the K instance classifiers. The goal of the bounding box regressor is to output a correction value for each box, and to correct the four parameters x, y, w, and h respectively.
[0029] Furthermore, in step S501, the loss function of iteratively training K instance classifiers is for:
[0030]
[0031] Among them, R is the number of proposal boxes generated in step 3, is the loss weight for each proposal box, CE is the cross entropy loss function, Represents the classification probability of C+1 categories of the r-th proposal box, is its classification label.
[0032] Furthermore, in step S503, pseudo supervision information and positive sample frames are generated as follows: S5031, category probability based on k-1 branches Perform NMS on the proposal set R, and the threshold is the predefined T nms , the set of frames after NMS is recorded as R keep ;
[0033] S5032. For each category c, if The set R obtained in step S5031 keep Score higher than T in the search category conf The proposal is then assigned a label c. If no box satisfies the requirement, the one with the highest score is assigned a label c. All boxes found are recorded as R. seek ;
[0034] S5033. For each proposal found, find the corresponding neighbor in R and mark it as R neighbor ;
[0035] S5034, the R obtained in step S5032 seek and step S5033R neighbor Merge to get the positive sample box.
[0036] Specifically, in step S6, the total loss function L of the overall network is:
[0037]
[0038] Among them, L base is the loss function of the MIL detector, λ 1 is the loss weight of K instance classifiers, K is the number of instance classifiers, is the loss function of the k-th instance classifier, λ 2 is the loss weight of the bounding box regressor, L box is the loss function of the bounding box regressor.
[0039] Specifically, in step S7, the hyperparameters are adjusted as follows:
[0040] The VGG16 network is used in the feature extraction stage; the loss weight λ of the instance classifier detector 1 = 1, the number of instance classifiers K = 3, the loss weight of the bounding box regression network λ 2 =0.3; NMS threshold T nms =0.3,,,T conf =0.7, T iou=0.5 ; During network training, the initial learning rate is 0.001, the learning rate decay is 0.0005, and the total number of iterations is 200000.
[0041] Another technical solution of the present invention is a weakly supervised target detection system, comprising:
[0042] The reading module reads image data and image-level labels. The image-level labels are image-level classification labels that only contain the object categories in the image, and divides the image data into a training data set and a test data set.
[0043] The weighting module uses the selective search algorithm to generate candidate boxes in the training data set read by the reading module, and then generates high-quality object proposal boxes through the gradient-based weighted class activation mapping method;
[0044] The matrix module inputs the training data set read by the reading module into the VGG16 convolutional neural network for feature extraction, and generates a feature matrix of the same shape through the ROI Pooling layer for the extracted proposal box features;
[0045] The learning module matches the image-level labels read by the reading module with the feature matrix obtained in step S3 one by one, performs multi-instance learning, and constructs a MIL detector;
[0046] The iterative module adds low-level supervision information to the object proposal box obtained by the weighted module and establishes an instance classifier for optimization iteration; the proposal box with the highest score is used as a pseudo label to train the bounding box regression network;
[0047] A function module that determines the loss function of the MIL detector obtained by the learning module and the bounding box regression network obtained by the iteration module;
[0048] The adjustment module adjusts the hyperparameters of the bounding box regression network in the function module to obtain a weakly supervised detection model;
[0049] The training module uses the training data set obtained by the reading module to train the weak supervision detection model obtained in the adjustment module to obtain a trained weak supervision detection model;
[0050] The detection module uses the selective search algorithm to generate candidate boxes on the test data set obtained by the reading module, and then uses the weakly supervised detection model trained by the training module to classify and regress the candidate boxes to obtain the final target detection box.
[0051] Compared with the prior art, the present invention has at least the following beneficial effects:
[0052] The invention discloses a weakly supervised target detection method. The proposal frame is gradually screened at each stage to generate a proposal frame that can frame the entire part of an object as pseudo-supervision information, supervise the final bounding box regression network, and improve the accuracy of target detection. The proposal frame is generated by Grad-CAM technology before the MIL detector to obtain a series of high-quality (covering the target more completely) proposal frames, which are beneficial to the detection of subsequent detectors and improve the defect that the traditional method is prone to fall into the local optimum. In the iterative optimization process, the supervision of low-level features is added, the concept of object-likeness is introduced, the proposal frame containing the complete target is screened, and pseudo-labels are generated to supervise the subsequent iterative process, thereby improving the defect that the existing method only frames the local part of the target and improving the detection accuracy.
[0053] Furthermore, in step S2, the candidate boxes generated by the selective search algorithm are screened based on the gradient weighted class activation mapping method, and the screened proposal boxes are close to the object target, which reduces the noise and interference in the subsequent training of the MIL detector.
[0054] Furthermore, in step S4, a MIL classifier is constructed. In the case of only image-level labels but no instance-level labels, the proposal boxes in each image are correctly classified, which greatly reduces the manpower and material resources required to annotate instance-level information.
[0055] Furthermore, in step S402, the loss function of the MIL detector is set to a binary cross entropy loss. By back-propagating the loss during the training process, the direction of network optimization is better guided and the training process is accelerated.
[0056] Furthermore, in step S5, by introducing the concept of low-level supervision information, namely, image similarity, this method can well filter out the proposal box that frames the entire target from a series of proposal boxes, thus solving the problem that other weakly supervised target detection methods of the same period are prone to fall into local optimality.
[0057] Furthermore, step S501 iteratively trains K instance classifiers, and each iteration performs a screening, and the pseudo-supervisory information generated is of high quality. The loss function of the instance classifier is the cross entropy loss, which better guides the direction of network optimization and speeds up the training process.
[0058] Furthermore, in step S503, by setting a threshold, the proposal boxes are subjected to NMS processing, proposal boxes with large overlap are merged, pseudo-supervision information and positive samples are generated, and the training of the bounding box regression network is better guided.
[0059] Furthermore, in step S6, by integrating the loss functions of different parts of the network (MIL classifier, K instance classifiers, bounding box regressor) and setting the weights of the loss functions of different parts, the network can converge faster during back-propagation training.
[0060] Furthermore, by adjusting the hyperparameters in step S7 and setting the learning rate decay, the rapid convergence in the early stage of training and the convergence stability in the later stage of training are ensured.
[0061] In summary, the present invention combines the high-level information and low-level supervisory information extracted by the deep neural network, screens the proposal boxes layer by layer, generates high-quality proposal boxes as supervisory information for target detection, and improves the accuracy of target detection; at the same time, the network hyperparameters are reasonably set, and the training process is fast and stable.
[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 It is the overall design framework of the present invention;
[0064] Figure 2 This is the pseudo-label generation process. DETAILED DESCRIPTION
[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.
[0067] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0068] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0069] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.
[0070] The present invention provides a weakly supervised target detection method. In the case of only image category annotation, a target detector is trained to detect the target in the picture. The present invention can save a lot of manpower, material resources and financial resources consumed by the annotation information. In the generation part of the prior frame, the selective search (SS) algorithm and the gradient weighted class activation mapping (Grad-CAM) method are combined to generate better prior frames. The intersection and union ratios of these prior frames with the Ground Truth are higher than the intersection and union ratios obtained by the greedy search method, and the entire object can be better covered. At the same time, in the optimization iteration process of the detector, the supervision information of the low-level features is added, and the concept of object-likeness is introduced to measure the degree to which the target in the prior frame is a complete target. The pain point that the current weakly supervised target detection method is prone to fall into the local optimum is solved, so that the network is more inclined to select a prior frame that covers the entire target rather than a part of the target without the supervision of the target boundary box information. The network of the present invention improves the performance of weakly supervised target detection and can be used in related fields of image processing and detection such as face detection, pedestrian counting, vehicle detection, robot navigation, and security systems. Experimental results show that it has good competitive performance.
[0071] See also Figure 1 The present invention provides a weakly supervised target detection method, comprising the following steps:
[0072] S1. Read images and labels from the dataset. The labels are image-level classification labels of only the object categories in the images.
[0073] S2. Generate high-quality proposal boxes
[0074] Figure 1 Part 1 of the proposed box generation combines selective search (SS) and gradient-weighted class activation mapping (Grad-CAM) techniques to generate better proposed boxes. The intersection-over-union (IOU) of these proposed boxes with the Ground Truth is higher than that obtained by the greedy search method, and can better cover the entire object.
[0075] S201, using a selective search algorithm to generate a large number of candidate boxes;
[0076] S202: Generate high-quality object proposal boxes through Grad-CAM.
[0077] S2021. When only image-level labels are available, a coarse classifier is trained for multi-label image classification tasks. Its sigmoid cross entropy loss function is as follows:
[0078]
[0079] Where C is the total number of image categories, y i is the label of the i-th image, P i is the prediction result of the i-th sigmoid classifier.
[0080] S2022. For each image containing an object category c, a coarse classifier is used to perform a weighted combination of a set of convolutional feature maps to obtain an activation map M for a specific category. c , and then set the ReLU function for activation:
[0081]
[0082] Among them, A k is the kth convolutional feature map, It is the feature map A k The weight for category c is calculated by subtracting y c Relative to A k The gradient of is globally averaged pooled:
[0083]
[0084] Among them, yc is the score of the cth classifier before sigmoid.
[0085] For each class-specific activation map of a given input image, we first set 10 segmentation thresholds that are evenly distributed between the maximum grayscale value of the activation map and the average grayscale value of all pixels;
[0086] Then, for each segmentation threshold, a binary image is obtained from the activation map of a specific class;
[0087] Finally, the maximum connected area is used to obtain a set of bounding boxes, each of which tightly surrounds a maximum connected area. These bounding boxes are the filtered proposal boxes.
[0088] In this way, a large number of class-specific object proposals are obtained. However, although the high-response regions contain objects, they are still far from fully localizing the entire object.
[0089] S2023. To solve this problem, we further train a set of fine classifiers to better locate the entire object in a weakly supervised environment. For a given object category, we only select the proposal box generated in the first stage whose softmax response is the highest (or sigmoid score is 1) and use it as the input for the second stage fine classifier training. This is actually a proposal box classification task, and its loss function is as follows:
[0090]
[0091] By repeating the operations in the first stage, higher quality object proposal boxes are generated, which can better locate the entire object than stage 1.
[0092] In summary, we first use a one-stage image-level classifier to predict the n potential object categories with the highest scores in each test image, and then use a two-stage proposal box-level classifier to obtain the final proposal box for detection.
[0093] S3, input the entire image into the convolutional neural network for feature extraction, and generate a feature matrix of the same shape through the ROI Pooling layer for the extracted proposal box features;
[0094] Figure 1 Part 2 in is the feature extraction network.
[0095] S4, multiple example learning;
[0096] Figure 1 Part 3 is to construct the MIL detector.
[0097] S401, for a given image x and its corresponding label yi =[y 1 ,…, C ], a series of proposal boxes are obtained through step S1 where y C = 1 or 0 indicates the presence or absence of object category c in the image, and C is the number of object categories;
[0098] Then, its proposal box features (the output of FC7) are fed into two data streams, called the classification data stream and the detection data stream, and two data matrices x are obtained by passing through two fully connected layers FC8c and FC8d respectively. c , These two data matrices are passed through two softmax operators to generate the classification score and detection score of each proposal box, as shown below:
[0099]
[0100]
[0101] S402, the final proposal box score is obtained by the element-wise product of the two matrices x R =σ cls (x c )⊙σ det (x d ), which will be used to optimize the next stage of the target detector. At the same time, the scores of all the proposal boxes are added in the dimension of R to obtain the image-level prediction score of the cth category:
[0102]
[0103] The above score is the predicted score of category c in the image, and the loss function of the MIL detector is the binary cross entropy loss:
[0104]
[0105] S5. Add low-level supervision information to optimize iteration
[0106] In a convolutional neural network, the information obtained by the bottom convolution layer is low-level and belongs to appearance features, such as edges, colors, textures, etc.; the information obtained after the top convolution layer is high-level and belongs to semantic features, such as category information. If you only score it based on the score of the proposed box category, this only considers high-level (top-down) semantic information. However, whether a box contains a complete object is not determined by high-level semantic information, but by low-level appearance information. Object-likeness is a measure of the degree to which an image is an object.
[0107] An object is represented in an image as having a clear boundary and center. Therefore, it is expected that a proposal box with a complete object has a higher similarity score than a proposal box that only frames a part or the background. Therefore, low-level supervision information is introduced in this stage to optimize the iterative object detector.
[0108] S501, iterative training of K classifiers
[0109] Inspired by OICR, K instance classifiers are established based on the MIL detector. The output of the kth classifier is used as supervision for the (+1)th classifier, and the low-level object-likeness information is used to guide network training.
[0110] Each classifier is implemented by a fully connected layer and a softmax layer on C+1 category dimensions (background is class 0). For the k-th instance classifier, the training loss function is:
[0111]
[0112] in, Represents the classification probability of C+1 categories of the r-th proposal box, is its classification label.
[0113]
[0114] Weight It is generated based on the similarity score of the proposal box r.
[0115] Specifically, we first calculate the similarity O of the low-level features of the proposal box r bu (r), which is measured by measuring the degree of superpixels straddling (SS) and the category score obtained by the previous classifier Perform weighted addition:
[0116]
[0117]
[0118] S502, the last part of the network is the bounding box regressor. The goal of this regressor is to output a correction value for each box, respectively correcting the four parameters x, y, w, and h:
[0119]
[0120] The loss function for the bounding box regressor is:
[0121]
[0122] S503: Generation of pseudo-supervision information and positive sample frames.
[0123] S5031, category probability based on k-1 branches Perform NMS on the proposal set R, and the threshold is the predefined T nms , the set of frames after NMS is recorded as R keep ;
[0124] S5032. For each category c (c>0, i.e. not a background category), if That is to say, if the MIL detector determines that the image contains objects of type C, then keep Score higher than T in the search category conf proposals, and then assign them labels c. If no box satisfies the condition, the one with the highest score is assigned label c. All the boxes found are recorded as R seek ;
[0125] S5033. For each proposal found, find its neighbors in R, that is, those with an IOU threshold of T greater than the IOU threshold. iou The proposals of the neighboring nodes are labeled with the same category. neighbor , other boxes are considered as background. In this way, each proposal has its pseudo-category label;
[0126] S5034, the set of positive sample frames is R seek and R neighbor A collection of merged items.
[0127] S6. Determine the loss function of the overall network (including the MIL detector, K instance classifiers, and bounding box regressor);
[0128]
[0129] S7, hyperparameter adjustment;
[0130] S701, λ 1 =1,λ 2 =0.3;
[0131] S702, optimizing the number of iterative detectors K=3;
[0132] S703, feature extraction network uses VGG16;
[0133] S704、T nms =0.3,T conf =0.7,T iou=0.5 ;
[0134] S705, the initial learning rate is 0.001, the learning rate decay is 0.0005, the total number of iterations is 200000; the NMS threshold of the processing is 0.3.
[0135] S8, training the detection model using the training data set to obtain a trained detection model;
[0136] The sample pairs of the training data set (images and their corresponding image-level labels) are used as the input of the detection model. The category of the target in each image in the training data set, the position and category of each optimization iteration proposal are used as the output of the detection model. At the same time, pseudo labels are generated during the iteration process. The overall loss is calculated by solving the error between the prediction result and the label. Then, the error is minimized through back propagation, and the network parameters of the detection model are optimized to obtain a trained weakly supervised detection model, such as Figure 2 shown.
[0137] S9. Use the trained model to test the test data set.
[0138] The test dataset is used as the input of the weakly supervised detection model. The output of the weakly supervised detection model is the location and category of the target in the test dataset, which is compared with the label (instance-level label) in the test dataset to verify the performance of the model.
[0139] In another embodiment of the present invention, a weakly supervised target detection system is provided, which can be used to implement the above-mentioned weakly supervised target detection method. Specifically, the weakly supervised target detection system includes a reading module, a weighting module, a matrix module, a learning module, an iteration module, a function module, an adjustment module, a training module and a detection module.
[0140] The reading module reads image data and image-level labels, where the image-level labels are image-level classification labels that only contain object categories in the image, and divides the image data into a training data set and a test data set;
[0141] The weighting module uses the selective search algorithm to generate candidate boxes in the training data set read by the reading module, and then generates high-quality object proposal boxes through the gradient-based weighted class activation mapping method;
[0142] The matrix module inputs the training data set read by the reading module into the VGG16 convolutional neural network for feature extraction, and generates a feature matrix of the same shape through the ROI Pooling layer for the extracted proposal box features;
[0143] The learning module matches the image-level labels read by the reading module with the feature matrix obtained in step S3 one by one, performs multi-instance learning, and constructs a MIL detector;
[0144] The iterative module adds low-level supervision information to the object proposal box obtained by the weighted module and establishes an instance classifier for optimization iteration; the proposal box with the highest score is used as a pseudo label to train the bounding box regression network;
[0145] A function module that determines the loss function of the MIL detector obtained by the learning module and the bounding box regression network obtained by the iteration module;
[0146] Adjust the module, adjust the hyper parameters of the loss function in the function module, and obtain a weak supervision detection model;
[0147] The training module uses the training data set obtained by the reading module to train the weak supervision detection model obtained in the adjustment module to obtain a trained weak supervision detection model;
[0148] The detection module uses the selective search algorithm to generate candidate boxes on the test data set obtained by the reading module, and then uses the weakly supervised detection model trained by the training module to classify and regress the candidate boxes to obtain the final target detection box.
[0149] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0150] 1. Simulation conditions:
[0151] The hardware platform is: HP-Z840 workstation, TITAN-X-12GB-GPU, 64GB RAM.
[0152] The software platform is: Python, Pytorch deep learning framework.
[0153] 2. Simulation content and results:
[0154] The data sets of the simulation experiment of the present invention are PASCAL VOC 2007 and 2012 data sets, and MS COCO data sets. The instance-level labels in the data sets are removed, and only image-level labels are used. PASCAL VOC 2007 and 2012 data sets are composed of 9962 and 22531 images of 20 categories, respectively. The 2007 data set has 5011 training images, and the 2012 data set has 11540 training images; the MS COCO data set is composed of 123278 images of 80 categories, and 82783 images are selected as training images and 40504 images are selected as test images. Mean Average Precision (mAP) (IOU>0.5) is used as the evaluation criterion.
[0155] After training, the model test results are shown in Table 1.
[0156] Table 1 Test results of the present invention on various data sets
[0157]
[0158] The target detection accuracy of the present invention on the PASCAL VOC 2007 dataset is 54.2%, the target detection accuracy on the PASCAL VOC2012 dataset is 47.5%, and the target detection accuracy on the MS COCO dataset is 23.2%, reaching the advanced level of weakly supervised target detection networks at the same stage.
[0159] In summary, the weakly supervised target detection method and system of the present invention have the following characteristics:
[0160] 1. Only image-level supervision information is required
[0161] In the task of object detection, labeling instance-level information (object bounding boxes) requires a lot of manpower, material resources, and financial resources. The cost of labeling with only image categories is obviously lower, and we can also crawl a large number of images with category annotations from web search engines, social media, etc. A large amount of training data can improve the performance of object detection. These cheap and easily available images with only category annotations are conducive to the development and implementation of engineering in the field of object detection.
[0162] 2. Better suggestion boxes
[0163] Most of the existing weakly supervised object detection algorithms use the traditional selective search (SS) algorithm to generate proposal frames, and then perform classification problems based on them. Therefore, the typical defects of machine learning classification algorithms become typical defects of weakly supervised target detection, that is, it is easy to detect more prominent objects or parts of objects in the image, while losing small objects or complete objects. The present invention generates proposal frames through Grad-CAM technology before the MIL detector to obtain a series of high-quality (covering the target more completely) proposal frames, which is beneficial to the detection of subsequent detectors and improves the defect that traditional methods are prone to fall into local optimality;
[0164] 3. Low-level supervisory information
[0165] When existing methods iteratively screen proposal boxes, the evaluation criteria are only the classification scores of the proposal boxes, which only consider high-level semantic information and ignore the importance of low-level information (edges, textures, etc.) in measuring whether a target is an object. The present invention adds supervision of low-level features in the iterative optimization process, introduces the concept of object-likeness, screens out proposal boxes that contain complete targets, and generates pseudo labels to supervise subsequent iterative processes, which improves the shortcomings of existing methods that only frame local targets and improves the detection accuracy.
[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0170] The above contents are only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.
Claims
1. A weakly supervised target detection method, characterized in that: The following steps are involved: S1, read image data and image-level labels, where the image-level labels are image-level classification labels that only contain object categories in the image, and divide the image data into a training data set and a test data set; S2, using the selective search algorithm to generate candidate boxes in the training data set read in step S1, and then generating high-quality object proposal boxes based on the gradient weighted class activation mapping method; S3, input the training data set read in step S1 into the VGG16 convolutional neural network for feature extraction, and generate a feature matrix of the same shape through the ROIPooling layer for the extracted proposal box features; S4, the image-level label read in step S1 and the feature matrix obtained in step S3 are matched one by one, multiple-instance learning is performed, and a MIL detector is constructed; S5. Add low-level supervision information to the object proposal box obtained in step S2, and establish an instance classifier for optimization iteration; use the proposal box with the highest score as a pseudo label to train the bounding box regression network, specifically: S501: Establish K instance classifiers based on the MIL detector and calculate the similarity O of the low-level features of the proposal box r. bu (r), which is measured by measuring the degree of Superpixels Straddling, and the similarity O bu (r) and the category score obtained by the previous classifier Perform weighted addition, take the proposal box with the highest score after addition as the proposal box with the complete object target, and take the first n proposal boxes with the highest scores as the pseudo-supervisory information for the next iteration of training the instance classifier. Train K instance classifiers for a total of K iterations; S502, a bounding box regressor is connected behind the K instance classifiers. The goal of the bounding box regressor is to output a correction value for each box, respectively correcting the four parameters x, y, w, and h; S6, determining the loss function of the MIL detector obtained in step S4 and the bounding box regression network obtained in step S5; S7, adjusting the hyperparameters of the bounding box regression network in step S6 to obtain a weakly supervised detection model; S8, using the training data set obtained in step S1 to train the weak supervision detection model obtained in step S7, to obtain a trained weak supervision detection model; S9. The test data set obtained in step S1 uses a selective search algorithm to generate candidate boxes, and then uses the weakly supervised detection model trained in step S8 to classify and regress the candidate boxes to obtain the final target detection box.
2. The weakly supervised target detection method according to claim 1, characterized in that: In step S2, the first-stage image-level classifier is used to predict the n potential object categories with the highest scores in each test image, and then the second-stage proposal box level classifier is used to obtain the final proposal box for detection. The loss function of the first stage is as follows: Where C is the total number of image categories, y i is the label of the i-th image, and Pi is the prediction result of the i-th sigmoid classifier.
3. The weakly supervised target detection method according to claim 1, characterized in that: Step S4 is specifically as follows: S401, for a given image and its corresponding label, obtain a series of object proposal boxes through step S2; Then the object proposal box is sent to the classification data stream and the detection data stream, and two data matrices are obtained through two fully connected layers respectively; the two data matrices are respectively passed through two softmax operators to generate the classification score and detection score of each proposal box; S402: perform element-wise multiplication of the classification score and the detection score obtained in step S401 to obtain the final proposal box score, and add the scores of all proposal boxes in the dimension of R to obtain the image-level prediction score of each category.
4. The weakly supervised target detection method according to claim 3, characterized in that: In step S402, the loss function of the MIL detector is: Among them, P c is the image-level prediction score for the c-th category, y c The label of the image.
5. The weakly supervised target detection method according to claim 1, characterized in that: In step S501, the loss function of iteratively training K instance classifiers is for: Among them, R is the number of proposal boxes generated in step 3, is the loss weight for each proposal box, CE is the cross entropy loss function, Represents the classification probability of C+1 categories of the r-th proposal box, is its classification label.
6. The weakly supervised target detection method according to claim 1, characterized in that: In step S503, pseudo supervision information and positive sample frames are generated as follows: S5031, category probability based on k-1 branches Perform NMS on the proposal set R, and the threshold is the predefined T nms , the set of frames after NMS is recorded as R keep ; S5032. For each category c, if The set R obtained in step S5031 keep The search category scores higher than T conf The proposal is then assigned a label c. If no box satisfies the requirement, the one with the highest score is assigned a label c. All boxes found are recorded as R. seek ; S5033. For each proposal found, find the corresponding neighbor in R and mark it as R neighbor ; S5034, the R obtained in step S5032 seek and step S5033R neighbor Merge to get the positive sample box.
7. The weakly supervised target detection method according to claim 1, characterized in that: In step S6, the loss function L is: Among them, L base is the loss function of the MIL detector, λ1 is the loss weight of the K instance classifiers, K is the number of instance classifiers, is the loss function of the k-th instance classifier, λ2 is the loss weight of the bounding box regressor, and L box is the loss function of the bounding box regressor.
8. The weakly supervised target detection method according to claim 1, characterized in that: In step S7, the hyperparameters are adjusted as follows: The VGG16 network is used in the feature extraction stage in step S3; the loss weight of the instance classifier detector λ1=1, the number of instance classifiers K=3, the loss weight of the bounding box regression network λ2=0.3; the threshold T of NMS nms =0.3, T conf =0.7, T iou=0.5 ; During network training, the initial learning rate is 0.001, the learning rate decay is 0.0005, and the total number of iterations is 200000.
9. A weakly supervised target detection system, characterized in that: include: The reading module reads image data and image-level labels. The image-level labels are image-level classification labels that only contain the object categories in the image, and divides the image data into a training data set and a test data set. The weighting module uses the selective search algorithm to generate candidate boxes in the training data set read by the reading module, and then generates high-quality object proposal boxes through the gradient-based weighted class activation mapping method; The matrix module inputs the training data set read by the reading module into the VGG16 convolutional neural network for feature extraction, and generates a feature matrix of the same shape through the ROIPooling layer for the extracted proposal box features; The learning module matches the image-level labels read by the reading module with the feature matrix obtained in step S3 one by one, performs multi-instance learning, and constructs a MIL detector; The iterative module adds low-level supervision information to the object proposal box obtained by the weighted module and establishes an instance classifier for optimization iteration; the proposal box with the highest score is used as a pseudo label to train the bounding box regression network, specifically: Based on the MIL detector, K instance classifiers are established to calculate the similarity O of the low-level features of the proposal box r. bu (r), which is measured by measuring the degree of Superpixels Straddling, and the similarity O bu (r) and the category score obtained by the previous classifier Perform weighted addition, take the proposal box with the highest score after addition as the proposal box with the complete object target, and take the first n proposal boxes with the highest scores as the pseudo-supervisory information for the next iteration of training the instance classifier. Train K instance classifiers for a total of K iterations; Connect a bounding box regressor after the K instance classifiers. The goal of the bounding box regressor is to output a correction value for each box, correcting the four parameters x, y, w, and h respectively; A function module that determines the loss function of the MIL detector obtained by the learning module and the bounding box regression network obtained by the iteration module; The adjustment module adjusts the hyperparameters of the bounding box regression network in the function module to obtain a weakly supervised detection model; The training module uses the training data set obtained by the reading module to train the weak supervision detection model obtained in the adjustment module to obtain a trained weak supervision detection model; The detection module uses the selective search algorithm to generate candidate boxes on the test data set obtained by the reading module, and then uses the weakly supervised detection model trained by the training module to classify and regress the candidate boxes to obtain the final target detection box.
