Object detection

Abstract

An apparatus (401) for classifying regions in images, the apparatus comprising: an input for receiving a first set of images (512) and, for each image of the first set (512), data defining one or more regions of the image and a classification of the regions, and a second set of images (513) and, for each image of the second set (513), a classification of the image; a processor (404) for training a first machine learning algorithm (106, 514) to classify features in images by: processing the images of the first set (512) and the second set (513) using the first algorithm (106, 514) to estimate feature regions in the images and a classification of each feature region, and training the first algorithm (106, 514) in dependence on the classifications received for the images of the first set (512) and the second set (513). This can obtain a more stable model at a minimum annotation cost.

Description

Technical Field

[0001] This invention relates to object detection in images. Background Technology

[0002] In recent years, object detection has made significant progress relying on large-scale, manually annotated databases. However, in many real-world scenarios, building and annotating such databases is extremely time-consuming and impractical.

[0003] These drawbacks have spurred the development of weakly supervised and few-shot object detection methods. Both strategies show promise, but still fall short compared to strongly supervised strategies.

[0004] The crux of the problem can be attributed to the inability to learn stable object features by relying on weak training signals provided by image-level labels or only a few annotated examples, which often leads to incomplete or oversized detections.

[0005] Some existing work on weakly-supervised object detection (WSOD) relies on the multiple-instance learning (MIL) framework, which aims to classify individual instances from global group-level labels. In these scenarios, each image is represented as multiple bounding box proposals (usually extracted by conventional methods, such as Uijlings, JR, Van De Sande, KE, Gevers, T., Smeulders, AW: Selective search for object recognition, International Journal of Computer Vision 104(2), pp. 154-171 (2013) and Zitnick, CL, Dollar, P.: Edge boxes: Locating object proposals from edges, European Conference on Computer Vision, pp. 391-405, Springer (2014)). These proposals are labeled using image-level labels, effectively transforming weakly supervised object detection into a multi-label classification problem (as described in Bilen, H., Vedaldi, A.: Weakly supervised deep detection networks, IEEE Conference Proceedings on Computer Vision and Pattern Recognition, pp. 2846-2854 (2016)).

[0006] Previous work has focused on alternating pseudo-labeled images and training fully supervised models. Due to the computational limitations of these methods, recent work has concentrated on enhancing the MIL-based labeling step, which is typically followed by training a single fully supervised model on pseudo-labeled data, as described in Girshick, R.: Fast r-cnn, Proceedings of the IEEE International Conference on Computer Vision, pp. 1440-1448 (2015). The proposed approach involves cascaded refinement of MIL classification using saliency maps or modeling uncertainty.

[0007] While WSOD methods and current evaluations hold promise, they also have several limitations. First, the PASCAL VOC 2007 database has been the sole focus of experimental evaluation for most research. Because the dataset is designed for fully supervised detection, PASCAL VOC 2007 lacks the diversity required for accurate statistical information about object class appearance. Furthermore, the non-convex MIL loss function makes optimization and refinement strategies for WSOD models extremely sensitive to initialization anomalies. This has led to the development of ad hoc training strategies and meticulously crafted dataset-specific parameters, significantly reducing the generalizability of the dataset.

[0008] Few-shot learning has recently garnered significant attention and made progress. However, most previous work has focused on classification tasks, with limited attention paid to other computer vision tasks, including detection. Few-shot object detection (FSOD) considers a fully supervised training set, along with a new set of classes consisting of only K training images per class. Therefore, the goal is to achieve good performance on new classes. Recent FSOD methods have often adapted few-shot classification techniques to object detection settings, exploring metric learning strategies or meta-learning.

[0009] Compared to FSOD, mixed supervision for object detection (MSOD) enhances the WSOD training set with image-level labels, which contains a small subset of fully annotated images (e.g., K images for each class in an FSOD scene).

[0010] In an alternative MSOD setting, Pan, Tianxiang et al., “Low shot box correction for weakly supervised object detection,” Proceedings of the 28th International Joint Conference on Artificial Intelligence, AAAI Press, 2019, disclose a two-stage approach that learns to refine the output of a pre-trained WSOD model using a small set of fully annotated images. The definition of a small set explored in their work ranges from 10 samples to 20% of the entire dataset. This approach significantly improves performance compared to the WSOD method and achieves high performance on small datasets. However, performance is highly dependent on the WSOD method used as model input. BCNet learns to refine the proposed detections, so initially lost objects can never be recovered. Finally, this approach requires training two separate models, which makes adapting the WSOD parameters—i.e., training on new datasets—challenging.

[0011] EHSOD (e.g., Fang, L., Xu, H., Liu, Z., Parisot, S., Li, Z.: EHSOD: CAM-Guided End-to-End Hybrid-Supervised Object Detection with cascade refinement, AAAI Press 2020) and BAOD (e.g., Pardo, A., Xu, M., Thabet, A., Arbelaez, P., Ghanem, B.: Baod: Budget-aware object detection, Arxiv 2019) focus on larger datasets (e.g., 10% to 100%), aiming to reduce the amount of data required to achieve fully supervised performance using cascaded MIL models and separate teacher and student settings trained with strong and weak annotations. These methods often require significantly more data to achieve high performance.

[0012] There is a need to develop an object detection device and method to solve these problems. Summary of the Invention

[0013] According to a first aspect, an apparatus for classifying regions in an image is provided, the apparatus comprising: an input terminal for receiving a first image set and, for each image in the first set, data defining one or more regions of the image and classifications of the regions; a second image set and, for each image in the second set, classifications of the images; and a processor for training a first machine learning algorithm to classify features in the image by: processing the images in the first set and the second set using the first algorithm to estimate feature regions in the image and classifications of each feature region, and training the first algorithm based on classifications received for the images in the first set and the second set.

[0014] The processor can be used to train the algorithm to generate confidence estimates for the classification of at least some of the images in the second set. This facilitates the weighting of these images for subsequent use.

[0015] The processor can be used to train a second machine learning algorithm to classify regions in an image. The processor can also be used to use a subset of the images in the second set during the training of the second algorithm, such that the weight of each image in the subset depends on the estimated confidence level of the corresponding image during the training of the second algorithm, and the subset of images in the second set is classified by the first algorithm. This can improve the training of the second algorithm.

[0016] The first algorithm may include a first sub-part for estimating feature regions in an image and a second sub-part for estimating the classification of the feature regions. The processor may be used to train the first sub-part to estimate feature regions in a second set of images, the feature regions being classified by the second sub-part to match the received classification of the corresponding image. This helps improve the accuracy of the first sub-part.

[0017] The second algorithm can be used to train the second sub-part. This can improve the accuracy of the second sub-part.

[0018] The processor can be used to train the first algorithm by repeatedly using at least some of the first image set and the second image set as input. This can refine the accuracy of the first algorithm.

[0019] The first algorithm can be run based on a stored set of weights, and the processor can be used to train these weights based on the performance of the first algorithm when classifying images from the first set and the second set. This allows for easy training of the first algorithm.

[0020] The second algorithm can implement a different machine learning architecture than the first algorithm. Therefore, the first algorithm can be used to help train algorithms that are simpler or more complex than itself.

[0021] The second algorithm can achieve less internal feedback than the first algorithm. The second algorithm is more suitable for implementation in devices with simplified processing capabilities. The first algorithm can help reduce the need for manual image classification.

[0022] The first algorithm and the second algorithm may include a common feature encoder. The device can be used to simultaneously train the first algorithm and the second algorithm based on each other's performance. The first algorithm and the second algorithm can be trained end-to-end. These features can aid in algorithm training.

[0023] According to a second aspect, a method for classifying regions in an image is provided, the method comprising: receiving a first set of images, and for each image in the first set, data defining one or more regions of the image and classifications of the regions, and a second set of images, and for each image in the second set, classifications of the images; a processor training a first machine learning algorithm to classify features in the images by: processing the images in the first set and the second set using the first algorithm to estimate feature regions in the images and classifications of each feature region, and training the first algorithm based on classifications received for the images in the first set and the second set.

[0024] The method may include forming confidence estimates for the classification estimated for at least some of the images in the second set. This may allow for weighting of these images for subsequent use.

[0025] The method may include: training a second machine learning algorithm to classify regions in an image, thereby using a subset of the images from a second set when training the second algorithm, such that the weight of each image in the subset depends on the estimated confidence of the corresponding image when training the second algorithm, the subset of images from the second set being classified by the first algorithm. This can improve the accuracy of training.

[0026] The method may include: in a device for classifying features in an image, implementing the second algorithm after the second algorithm has been trained, without the first algorithm. In some embodiments, the second algorithm may be more suitable for implementation in a device with limited processing power. Attached Figure Description

[0027] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0028] Figure 1 An example of the proposed MSOD model architecture is illustrated schematically;

[0029] Figure 2 The diagram illustrates online supervised generation.

[0030] Figure 3 A flowchart illustrating an example of a method for classifying regions in an image is shown;

[0031] Figure 4 An example of an imaging device for implementing the apparatus and methods described herein is shown;

[0032] Figure 5 This illustration schematically compares the low-data-supervision strategy with the hybrid-supervision-based approach described in this paper. Detailed Implementation

[0033] This paper describes an object detection strategy that utilizes hybrid supervision, combining low-sample fully annotated examples with a large set of weakly labeled images.

[0034] In the following text, I refers to the set of training images annotated with image-level supervision. Under the hybrid supervision paradigm, this subset of images... Further annotations are provided using bounding box annotations. The images contained in S are called strongly trained images. Therefore, this first set of images defines one or more regions and a classification for each image. Images in W = I\S that contain only image-level annotations are called weakly trained images. Therefore, this second set of images defines a classification for each image.

[0035] Figure 1 An example of a network architecture 100 is shown. This model includes two branches with a shared encoder backbone.

[0036] The shared encoder backbone is indicated at 102. For each image 101, the shared encoder 102 processes bounding box proposal features from candidate proposals 103, which may be RPNs or pre-computed. This part of the network uses ROI pooling layers 104 to compute fixed-length feature representations for the bounding box proposals for each image.

[0037] First, unsupervised methods are used (e.g., selective search (as described in Uijlings, JR, Van De Sande, KE, Gevers, T., Smeulders, AW: Selective search for object recognition, International Journal of Computer Vision 104(2), pp. 154-171 (2013)) or bounding boxes (e.g., Zitnick, CL, Dollar, P.: Edge boxes: Locating object proposals from edges, European Conference on Computer Vision, pp. 391-405, Springer (2014)) or region proposal networks (RPNs) (e.g., Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal). Networks), Research Progress in Neural Information Processing Systems, pp. 91-99 (2015) and its related feature vectors Calculate the set of candidate proposals B These eigenvectors Obtained using a standard CNN backbone and ROI pooling layers 104, and provides common input to two branches: an online annotation module (OAM) 106 and a fully supervised branch (standard object detector 110).

[0038] The first branch of the first machine learning algorithm is trained using two sets of training images: a set of weak images and a set of strong training images, to learn an online annotation module (OAM) for the weak training images. The OAM generates bounding box annotations for the weak training images in real time, along with optional relevant confidence scores. If a weak image is already annotated with high confidence, it is added to a third set of images (i.e., this third set is a subset of the second 'weak' image set). Furthermore, if the confidence level of subsequent annotations decreases, these images are removed. The images included in P are referred to here as semi-strong (SS) training images. The second branch is designed as a standard fully supervised model, trained end-to-end using both strong and semi-strong images.

[0039] The online annotation module (OAM) 106 is designed to efficiently combine weak and strong supervision. It comprises three main components: a joint detection module, generally shown as 107, for predicting bounding boxes and their categories using weak and strong labels from a single common architecture; an online bounding box enhancement step 108 for generating refined bounding box proposals; and a supervision generator 109 for identifying trustworthy annotations to be used as supervision. These components are described in detail below.

[0040] A set of weakly trained images 'W' and a set of strongly trained images 'S' are used to train a joint detection module 107. This module uses the strongly trained images to better distinguish between partial and full object range proposals in the MIL latent space. Furthermore, the module learns to annotate the weakly trained images online and can assign annotation confidence scores to all computed bounding boxes in the weakly trained images, as well as assign global annotation scores to the images.

[0041] As described above, these annotated weak training images constitute a third image set, which is a strict subset of the weak images 'W' (the second image set), and is called semi-strong (SS) images. This SS image set can be expanded during training to include increasingly more annotated weak images provided by the proposed joint detection module 107. The resulting strong training image set and semi-strong training image set are then used to simultaneously train a fully supervised second branch (described in detail below) (an example is given here using the fast R-CNN branch).

[0042] The weakly supervised component employs a multiple instance learning (MIL) framework and includes two parallel, fully-connected (FC) layers (105), which output two separate feature matrices. Among them, consider category C.

[0043] In the joint detection module 107, the softmax function is used independently along two dimensions (category corresponding to γ). C The proposal corresponds to γ R This yields two feature vectors that provide complementary information. C (c, l) represents the probability that the l-th proposal belongs to category c, while γ R (c, l) represents the proportional contribution of proposal l to the image classified as category c. The proposal score φ is obtained for each category by combining the two scores using the Hadamard product ⊙. P =γ C ⊙γ R .

[0044] Sum the proposed fractions Image-level supervision can be used to learn how to identify relevant proposals using φP. The image-level loss function is defined using binary cross-entropy:

[0045]

[0046] Among them, y c It is a label indicating whether class c is present or absent in the image. Intuitively, in some implementations, this type of MIL model may have difficulty distinguishing between the full scope of object proposals and the scope of partial object proposals, and often achieves convergence by assigning high scores to proposals that contain highly discriminative object parts.

[0047] Compared to the MIL task, the strongly supervised component in the second branch of the joint detection module can determine accurate object representations, but is likely to overfit low-sample training examples. This strongly supervised component includes standard box-level two-branch classification and regression tasks (performed by the portions shown in 107a and 107b). Therefore, the MIL and the strongly supervised component are combined via a classification task to predict γ for the proposed class. C Provide direct supervision. Enhance the synergy between these two types of supervision, normalize the low-sample task, and constrain the MIL task and encoder to learn stronger discriminative features between full-range and partial-range object proposals.

[0048] The regression branch of the output box coordinate prediction (107b) of the joint prediction module 107 is identical to the bounding box regression operation performed in a conventional object detector. Each bounding box b is parameterized as a quadruple (x, y, hw), specifying its center coordinates (x, y) and its height and width (h, w). For each pre-computed proposal, the last branch predicts the offset t of these coordinates. k =(t x , t y , t h , t w ).

[0049] Finally, use an additional loss to train these two operations on strong images:

[0050] L p (γ, u, t, v) = L cls (γ, u) + λ[u≥1]L reg (t, v) (2)

[0051] in,

[0052]

[0053] Parameters γ and u constitute the predicted proposal class and the target proposal class, respectively, and t and v are the predicted bounding box offset and the target bounding box offset, respectively. L1 It is a smooth L1 loss function. Therefore, the loss function for strongly trained images is... The loss function for weakly trained images is

[0054] The online bounding box enhancement steps performed in OAM are now described, such as... Figure 1 As shown in 108.

[0055] Learning strong bounding box regression blocks from low-sample data can be challenging, especially when large updates (spatial offsets) are required. To fully utilize available annotations during training, a second forward pass of the proposed branch is performed using the new bounding boxes obtained from the regression branch as input. More specifically, the highest-scoring M-proposal is selected for each class corresponding to the image-level labels obtained after the first feedforward pass.

[0056] Offset t calculated using the bounding box regression branch k The coordinates of these proposals are modified accordingly, and these new proposals are fed into the ROI pooling layer to generate a new set of bounding box features. A loss function is calculated for these new boxes with only the new coordinates, and this loss function is added to the first pass loss. With this second pass, the overall loss function of the OAM branch is: In this context, the superscripts I and II represent the first pass and the second pass, respectively.

[0057] This second pass has a dual motivation. First, this approach provides an intrinsic bounding box augmentation strategy, generating a new set of proposal candidates for training regression and classification tasks. This is especially important in settings using pre-computed proposals (the main setting in WSOD), because without this augmentation strategy, during training there would only be a static set of input proposals (and associated update transformations). Modifying the pre-computed proposals increases the variability of the proposals, thereby enabling the regressor to learn from a more diverse range of inputs.

[0058] Secondly, this strategy allows for the utilization of additional signals from weak images and achieves more stable performance. In fact, any weak image proposal that receives modifications in a direction that hinders correct image-level label inference is penalized, thus regulating the training process of the regressor.

[0059] The key objective of OAM 106 is to generate reliable annotations for a large set of weakly labeled images to guide the training of the fully supervised second branch. Since the first model in OAM 106 is trained simultaneously with the second model in the second branch 110, it is desirable to identify only reliable annotations and add them to the training image pool. The rationale is that only these images should be used to train the final supervised detection network, while images that are difficult for the joint detection module to annotate with high confidence should not be used for model training (the second model), as these images may impair the training process and degrade detector performance.

[0060] In the early stages of training, the uncertainty in the regression refinement of the proposed bounding box categories and box coordinates may be high. As training progresses and the model's prediction quality improves, confidence, accuracy, and stability will also increase. This may enable the accurate annotation of increasingly difficult image sets.

[0061] This behavior can be utilized by introducing a supervised generator 109, which is capable of identifying what to add to a set called a semi-strong image set. A new set of images with reliable annotations is then used to train a second fully supervised branch.

[0062] Figure 2 The online supervised generation performed by module 109 is illustrated in more detail.

[0063] Intuitively, a semi-strong image set P can include simple images from the early stages of training (e.g., single instances, uniformly colored backgrounds), with the diversity likely to gradually increase as the model becomes more accurate. To construct the semi-strong image set P, the following mechanism can be used, leveraging bounding boxes and associated annotation confidence scores.

[0064] Given a weak image I, after performing non-maximum suppression (NMS) as shown at position 201 on the output of the joint detection module 107, N is obtained. I A set of bounding boxes Among them, c r and p r These correspond to the category label and coordinates of the bounding box r, respectively. For example... Figure 2 As shown, a strategy similar to the online augmentation steps was adopted, using D t-1 As input candidate proposals, N is iteratively predicted at each iteration t>1. t A new set of bounding boxes More specifically, the bounding box D obtained in the previous iteration t-1 It is fed back into the ROI pooling layer 104, providing a new set of image features that allows the computation of new proposed coordinates.

[0065] like Figure 2 As indicated by arrow 202, this process iterates until the bounding box prediction stabilizes and converges (as shown at 203). Preferably, when D t-1 =D t That is, the process stops when two subsequent passes yield the same bounding box. It is estimated that for each bounding box b... t ∈D t There exists a corresponding box b. t-1 ∈D t–1 , making b t And b t-1 Convergence can be achieved when there is an IoU ≥ 0.5 and a matching class prediction (i.e., a popular criterion for characterizing object equivalence in detection methods). A global confidence weight of 1 / T can be assigned to each image, where T steps are required to achieve convergence.

[0066] The set of proposals D1 obtained in iteration 1 constitutes the final bounding box annotations. Each box is weighted by the average overlap with the best-matching box over all subsequent iterations (box-level confidence). Boxes that do not exist in a given iteration (overlap < 50%) are assigned a zero overlap for that iteration.

[0067] Finally, in the preferred implementation, an image is added to the semi-strong image pool P only if the obtained annotation set contains all categories of image-level labels. Images that converge after a large number of iterations can be assigned low confidence scores by design, thus having a limited impact on the training process of the second branch. Images that fail to converge after K iterations, or for which no foreground proposals can be found, may not be annotated and may not be added to the semi-strong pool P. In one example, the maximum number of updates is set to K = 30 to prevent large sets of iterations that may occur in the early stages of training.

[0068] Because weak images are annotated by OAM during training, the semi-strong image pool P will expand, while the annotations and confidence levels will be refined as the model improves. In a given training step, weak images that were not successfully annotated and are present in the semi-strong image pool P can be removed.

[0069] Therefore, in a manner similar to fast R-CNN type methods (see Girshick, R.: Fast R-CNN, Proceedings of the IEEE International Conference on Computer Vision, pp. 1440-1448 (2015)), training is performed simultaneously with OAM training using the obtained strong training image set and semi-strong training image set. Figure 1 The second fully supervised branch 110 includes a bounding box classification module 111 and a regression module 112.

[0070] Therefore, in the second branch 110, the SS image is used together with the S image to learn a fully supervised object detection model.

[0071] Standard object detector 110 for proposed features ε rf Perform classification and bounding box regression, training on strong and half-strong images. Specifically, in each training iteration, use batches with the same number of strong and half-strong images. The loss function for this branch is:

[0072] L 2B (p, u, t, v) = L cls (p,u)+L reg (t, v) (4)

[0073] Where p is the predicted ROI class, t is the predicted offset between the ROI and the target, u is the class label, and v is the target offset. Only ROIs with foreground labels will result in regression loss L. reg L cls The loss consists of the weighted cross-entropy for each image:

[0074]

[0075] In each batch, the proposals that cause losses are indexed by i, and the GT proposals are indexed by u. i The confidence level is represented by w i The image-level annotation confidence score is denoted as α. Strong images are assigned image and proposal-level weights of 1. In summary, the collected components produce the overall loss function of the model: L tot =L 1B +L 2B .

[0076] During testing, only fully supervised object detection models were used.

[0077] Therefore, the online annotation module utilizes hybrid supervision and can be used in conjunction with two-stage object detection methods (such as the Fast-RCNN series) to generate accurate, high-confidence automatic annotations for concurrent, fully supervised training.

[0078] Overall, the model is trained on a small set of fully annotated images (images with bounding box annotations, i.e., images with one or more regions defined by a classification of one or more regions) and a large set of weakly labeled images (image-level labels, i.e., images with classifications).

[0079] The method described in this paper learns to annotate weakly labeled input images in real time and leverages prediction stability to additionally provide relevant annotation confidence scores. The newly annotated images with high confidence are then used as confidence-weighted supervision for a second fast-RCNN-like branch that shares encoded features with the OAM branch. This strategy can efficiently train standard detection algorithms with minimal annotation requirements, providing an intrinsic training curriculum since only simple images can be labeled with high confidence from the outset.

[0080] Figure 3 An example method for classifying features in an image is shown. In step 301, the method includes receiving a first set of images, and for each image in the first set, data defining one or more regions of the image and classifications of the regions, and a second set of images, and for each image in the second set, classifications of the images. In step 302, the method includes training a first machine learning algorithm via a processor to classify features in the images through the following steps: Step 302a includes processing the images in the first and second sets using the first algorithm to estimate feature regions in the images and classifications of each feature region. Step 302b includes training the first algorithm based on classifications received for the images in the first and second sets.

[0081] Figure 4 An example of a system 400 including device 401 is shown. Device 401 is used to process image data captured by at least one image sensor in the device using the methods described herein. Device 401 includes image sensors 402, 403. Such device 401 typically includes some onboard processing capabilities. This may be provided by processor 404. Processor 404 may also be used to perform basic functions of the device.

[0082] Transceiver 405 is capable of communicating with other entities 410, 411 via a network. These entities may be physically located away from device 401. The network may be a publicly accessible network, such as the Internet. Entities 410, 411 may be cloud-based. Entity 410 is a computing entity. Entity 411 is a command and control entity. These entities are logical entities. In practice, these entities may be provided by one or more physical devices (such as servers and data storage), and the functionality of two or more entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. These devices may also include transceivers for sending data to and receiving data from transceiver 405 of device 401. The memory stores code in a non-transitory manner, which can be executed by the processor to implement the corresponding entity in the manner described herein.

[0083] Command and control entity 411 can train the artificial intelligence model used in the device. This is typically a computationally intensive task, and even if the resulting model can be efficiently described, developing algorithms that execute in the cloud is likely to be efficient, with a significant amount of energy and computing resources expected to be available. It can be anticipated that this is more efficient than forming such a model in a typical imaging device.

[0084] In one implementation, once the algorithm is developed in the cloud, the command and control entities can automatically generate a corresponding model and transmit it to the relevant imaging device. In this example, the model is implemented by the processor 404 in device 401.

[0085] In another possible implementation, the image can be captured by one or both of sensors 402 and 403, and the image data can be sent to the cloud for processing by transceiver 405. The resulting image can then be sent back to device 401, such as... Figure 4 As shown in 412.

[0086] Therefore, the method can be deployed in various ways, such as in the cloud, on a device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or refine existing ones. Depending on the computing power near the data corpus, training can be performed near the source data or in the cloud, for example, using an inference engine. The method can also be implemented on a device, in dedicated hardware, or in the cloud.

[0087] Figure 5 A comparison between different low-data-supervision strategies and the hybrid-supervision-based setup described in this paper is shown. The first row at 501 shows known weakly supervised object detection (WSOD) models 502, which are typically deployed to annotate images using image-level annotations 503. The WSOD output is images with instance-level labels 504, which are then used to train a fully supervised model 505 to provide the output 506.

[0088] In the second row at 507, known few-shot object detection (FSOD) methods train the model on a small number of training examples 508. The model is then trained on low-sample images with instance-level labels 509 to provide the output 510.

[0089] As shown in the bottom row at 511, the mixed supervised object detection (MSOD) setup described in this paper combines a low-sample set of images 512 with object (instance-level) annotations with a large database 513 including image-level annotations. As described above, an online annotation module 514 is trained to generate a multi-sample set, which is simultaneously used to train a fully supervised model 515 using online labeled images 516 with instance-level labels and annotation confidence to provide output 517 (images with instance-level labels).

[0090] The MSOD method described in this paper has the following advantages.

[0091] Extensive experiments based on two popular benchmarks (PASCAL VOC 2007 and MS-COCO) show that, in some implementations, the model outperforms existing low-sample techniques and improves the performance of fully supervised models by 10% in low-sample systems.

[0092] The online annotation module learns to generate bounding box annotations in real time from hybrid supervision. This enables high-quality training of multi-sample detectors based on low-sample annotations. OAM can be used in conjunction with standard two-stage detectors and significantly improves their performance in settings with limited annotation examples.

[0093] The model is trained end-to-end so that the OAM and fully supervised branches share a common feature encoder, and these branches are trained in parallel so that the two tasks can be normalized to each other and learn a better latent space.

[0094] This approach constitutes a flexible end-to-end strategy that achieves high performance using a pre-computed network of proposals and region proposals.

[0095] As described above, in the first branch of the network, an online annotation module 106 is trained on weak and strong training images, which annotates weak images in real time. Weak images annotated with high confidence are added to the pool of semi-strong (SS) images, and images annotated with low confidence (W) images that have been previously added are removed.

[0096] Therefore, guided by a small set of fully annotated images, the online annotation module learns to annotate a large set of weakly labeled images in real time, while providing annotation confidence. Furthermore, the online data-driven strategy simultaneously trains the object detector in a fully supervised manner by utilizing supervision generated by a modified MIL network.

[0097] As described above, the WSOD-type image-level classification task is combined with a fully supervised joint classification and regression task. This allows for the use of low-sample fully annotated images to guide and regulate the ill-posed MIL training process, while leveraging the statistics provided by the weak images to strengthen and regulate the proposal-level classification and regression tasks. Compared to the WSOD method, including a small number of labeled samples generates a more stable model with minimal annotation cost.

[0098] The applicant hereby discloses individually each of the individual features described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein; and without limiting the scope of the claims. The applicant indicates that aspects of the invention can be constituted by any such individual features or combinations of features. In view of the foregoing description, various modifications that can be made within the scope of the invention will be apparent to those skilled in the art.

Claims

1. An apparatus (401) for classifying a region in an image, characterized by The device comprises: an input for receiving a first set of images (512) and, for each image of the first set of images (512), data defining one or more regions of the image and a classification of the regions, and a second set of images (513) and, for each image of the second set of images (513), a classification; a processor (404) for training a first machine learning algorithm (106, 514) to classify features in images by: processing the images of the first set of images (512) and the second set of images (513) using the first machine learning algorithm (106, 514) to estimate regions of features in the images and a classification of each region of features, and training the first machine learning algorithm (106, 514) from the classifications received for the images of the first set of images (512) and the second set of images (513); for training the first machine learning algorithm (106, 514) to form, for at least some of the images of the second set of images (513), a confidence estimate for the classification estimated for the image; for training a second machine learning algorithm (110, 515) to classify regions in images, the processor (404) being configured to use, in training the second machine learning algorithm (110, 515), a subset of the images of the second set of images, such that a weight for each image of the subset of the images is dependent on an estimated confidence of the respective image in training the second machine learning algorithm (110, 515), the subset of the images of the second set of images being classified by the first machine learning algorithm (106, 514).

2. The device (401) according to claim 1, characterized in that, the first machine learning algorithm (106, 514) comprises a first subpart for estimating regions of features in an image and a second subpart for estimating a classification of a region of features, and the processor (404) is configured to train the first subpart to estimate regions of features in images of the second set of images (513) that are classified by the second subpart to match a received classification of the respective image.

3. The device (401) according to claim 2, characterized in that, the second machine learning algorithm (110, 515) is for training the second subpart.

4. The device (401) according to claim 1 or 2, characterized in that, the processor (404) is configured to train the first machine learning algorithm (106, 514) by multiple times using at least some of the first set of images (512) and the second set of images (513) as input to the first machine learning algorithm (106, 514).

5. The device (401) according to claim 1 or 2, characterized in that, the first machine learning algorithm (106, 514) is run according to a stored set of weights, and the processor (404) is configured to train the weights from a performance of the first machine learning algorithm (106, 514) in classifying the images of the first set of images (512) and the second set of images (513).

6. The device (401) according to claim 1 or 2, characterized by The second machine learning algorithm (110, 515) implements a different machine learning architecture than the first machine learning algorithm (106, 514).

7. The device (401) according to claim 1 or 2, characterized by The second machine learning algorithm (110, 515) implements less internal feedback than the first machine learning algorithm (106, 514).

8. The device (401) according to claim 1 or 2, characterized by The first machine learning algorithm (106, 514) and the second machine learning algorithm (110, 515) comprise a common feature encoder (102).

9. The device (401) according to claim 1 or 2, characterized by The device (401) is configured to train the first machine learning algorithm (106, 514) and the second machine learning algorithm (110, 515) simultaneously from each other's performance.

10. The device (401) according to claim 1 or 2, characterized by The first machine learning algorithm (106, 514) and the second machine learning algorithm (110, 515) are trained end-to-end.

11. A method (300) for classifying a region in an image, characterized in that, The method comprises: receiving (301) a first set of images (512) and, for each image of the first set of images (512), data defining one or more regions of the image and a classification of the regions, and a second set of images (513) and, for each image of the second set of images (513), a classification; training (302), by a processor (404), a first machine learning algorithm (106, 514) to classify features in images by: processing (302a) the images of the first set of images (512) and the second set of images (513) using the first machine learning algorithm (106, 514) to estimate regions of features in the images and a classification of each region of features, and to train the first machine learning algorithm (106, 514) from the classifications received for the images of the first set of images (512) and the second set of images (513); training the first machine learning algorithm (106, 514) such that it forms, for at least some of the images of the second set of images (513), a confidence estimate of the classification estimated for the image; training a second machine learning algorithm (110, 515) to classify regions in images by selecting, for use in training the second machine learning algorithm (110, 515), a subset of the images of the second set of images such that a weight of each image of the subset in training the second machine learning algorithm (110, 515) depends on an estimated confidence of the respective image, the subset of the images of the second set of images being classified by the first machine learning algorithm (106, 514).

12. The method (300) according to claim 11, characterized by comprising, in a device for classifying features in images, after the second machine learning algorithm (110, 515) has been trained, implementing the second machine learning algorithm without the first machine learning algorithm (106, 514).

Images