Neural network model for semantic image segmentation

Abstract

The invention relates to a method of training a model for use in semantic image segmentation. The model comprises a first classifier neural network and a second classifier neural network. The method comprises training the first network by inputting a first training image containing a target object to the first network, using the first network to identify and erase pixels in the first training image that are discriminative of the target object, inputting the first training image to the second network, using the second network to determine a likelihood that the first modified training image contains the target object, and updating weights of the first network using a first loss function that is a monotonic increasing function of the determined likelihood.

Description

Background Technology

[0001] This invention relates to neural network models for semantic image segmentation, and to methods and apparatus for training such models.

[0002] By treating various computer vision problems as image segmentation problems, convolutional neural networks (CNNs) have been successfully applied to these issues. Examples include road scene understanding for autonomous driving and interpretation of medical imaging. For such applications, networks are typically trained using multi-class per-pixel labels, which together form an image-sized segmentation map or mask. The output of such networks is then another image-sized map representing the per-pixel class probability.

[0003] In likelihood-based semantic segmentation, a neural network is trained to perform semantic segmentation on training data consisting of a segmentation map of an image (e.g., a photograph) and its corresponding human-annotated "ground truth" values. The final segmentation map for a given input image is then obtained from the trained network by applying the argmax function along the category dimension of the predicted probabilities (i.e., selecting the most likely category for each pixel). For example, this segmentation map can be used to enable autonomous vehicles to determine whether an object in their field of vision is likely another vehicle or a pedestrian.

[0004] However, producing such pixel-level ground truth labels is time-consuming and expensive. Recent advances in weakly supervised training demonstrate that reasonable performance can be achieved using only image-level labels—that is, training data consisting of images and corresponding sets of labels indicating which of a predetermined set of objects (categories) is present in each image. For example, an image of a street scene might be labeled with {"bicycle", "road", "vegetation"}.

[0005] Weakly supervised image-level classifier neural networks can be used as surrogate tasks to perform pixel-level image segmentation by extracting attention maps (also referred to as category activation maps in this paper) from the network. These attention maps identify regions in the input image that the network considers discriminative for a particular target object. These regions represent the locations where the network focuses in order to identify the object. By combining the attention maps of each of the different possible categories that an image may contain, an image can be segmented into different objects. However, classification requires minimal evidence to make predictions, so such networks tend to focus on the most discriminative object regions, and therefore any resulting segmentation may miss important parts of the object. For example, a network trained to classify dogs and cats may primarily distinguish the animal's head, so any resulting segmentation map may fail to correctly classify the animal's body. This "discriminative localization" problem typically leads to poor segmentation performance. It is particularly prevalent in non-rigid object categories such as birds, cats, horses, and sheep, where the texture of fur or skin is much less discriminative than other body parts such as the head or feet.

[0006] A proposed approach addresses this issue by using a classifier network to identify the most discriminative object regions in an input image, and then training one or more further classifier networks on the same image with these regions removed, enabling the further networks to learn to find less discriminative regions belonging to the target object. Combining attention maps from multiple classifier networks yields a larger map with better target object coverage. See, for example, [Zhang, X., Wei, Y., Feng, J., Yang, Y., Huang, TS: "Adversarial complementary learning for weakly supervised object localization", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018) 1325–1334] and [Wei, Y., Feng, J., Liang, X., Cheng, MM, Zhao, Y., Yan, S.: "Object region mining with adversarial erasing: A simple classification to semantic segmentation approach", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017) 1568–1576].

[0007] It has also been proposed to train a single network to classify images using attention maps that cover as many target objects as possible. The network consists of two serial streams with shared weights. The first “classifier” stream generates the attention map, which is passed as input to the second “attention mining” stream. See [Li, K., Wu, Z., Peng, KC, Ernst, J., Fu, Y.: “Tell me where to look: Guided attention inference network”, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (2018) 9215–9223].

[0008] However, the applicant has recognized that the performance of these known methods can be improved. Therefore, this invention seeks to propose an improved model for use in semantic image segmentation. Summary of the Invention

[0009] In a first aspect, the present invention provides a method for training a model for use in semantic image segmentation, wherein the model comprises:

[0010] First classifier neural network; and

[0011] The second classifier neural network,

[0012] The method includes training the first network in the following manner:

[0013] The first training image containing the target object is input into the first network;

[0014] The first network is used to identify pixels in the first training image that are discriminative to the target object;

[0015] The first modified training image is generated by attenuating or erasing the identified pixels in the first training image;

[0016] The first modified training image is input into the second network;

[0017] The second network is used to determine the likelihood that the first modified training image contains the target object; and

[0018] The weights of the first network are updated using a first loss function, which is a monotonically increasing function of the determined likelihood.

[0019] The method includes training the second network in the following manner:

[0020] The second training image containing the target object is input into the first network;

[0021] The first network is used to identify pixels in the second training image that are discriminative to the target object;

[0022] A second modified training image is generated by attenuating or erasing the identified pixels in the second training image;

[0023] A training image set is input into the second network, the set including the second modified training images and one or more training images that do not contain the target object; and

[0024] For each training image in the set, the second network is used to determine the likelihood that the corresponding training image contains the target object, and the weights of the second network are updated using a second loss function different from the first loss function. When the training image contains the target object, the second loss function is a monotonically decreasing function of the determined likelihood, and when the training image does not contain the target object, the second loss function is a monotonically increasing function of the determined likelihood.

[0025] In a second aspect, the present invention provides a computer processing system for training a model for use in semantic image segmentation.

[0026] The model mentioned above includes:

[0027] First classifier neural network; and

[0028] The second classifier neural network,

[0029] The computer processing system is configured to train the first network in the following manner:

[0030] The first training image containing the target object is input into the first network;

[0031] The first network is used to identify pixels in the first training image that are discriminative to the target object;

[0032] The first modified training image is generated by attenuating or erasing the identified pixels in the first training image;

[0033] The first modified training image is input into the second network;

[0034] The second network is used to determine the likelihood that the first modified training image contains the target object; and

[0035] The weights of the first network are updated using a first loss function, which is a monotonically increasing function of the determined likelihood.

[0036] The computer processing system is configured to train the second network in the following manner:

[0037] The second training image containing the target object is input into the first network;

[0038] The first network is used to identify pixels in the second training image that are discriminative to the target object;

[0039] A second modified training image is generated by attenuating or erasing the identified pixels in the second training image;

[0040] A training image set is input into the second network, the set including the second modified training images and one or more training images that do not contain the target object; and

[0041] For each training image in the set, the second network is used to determine the likelihood that the corresponding training image contains the target object, and the weights of the second network are updated using a second loss function different from the first loss function. When the training image contains the target object, the second loss function is a monotonically decreasing function of the determined likelihood, and when the training image does not contain the target object, the second loss function is a monotonically increasing function of the determined likelihood.

[0042] In a third aspect, the present invention provides computer software including instructions that, when executed on a computer processing system, cause the computer processing system to train a model for use in semantic image segmentation, wherein the model includes:

[0043] First classifier neural network; and

[0044] The second classifier neural network,

[0045] The instructions therein instruct the computer processing system to train the first network in the following manner:

[0046] The first training image containing the target object is input into the first network;

[0047] The first network is used to identify pixels in the first training image that are discriminative to the target object;

[0048] The first modified training image is generated by attenuating or erasing the identified pixels in the first training image;

[0049] The first modified training image is input into the second network;

[0050] The second network is used to determine the likelihood that the first modified training image contains the target object; and

[0051] The weights of the first network are updated using a first loss function, which is a monotonically increasing function of the determined likelihood.

[0052] The instructions therein instruct the computer processing system to train the second network in the following manner:

[0053] The second training image containing the target object is input into the first network;

[0054] The first network is used to identify pixels in the second training image that are discriminative to the target object;

[0055] A second modified training image is generated by attenuating or erasing the identified pixels in the second training image;

[0056] A training image set is input into the second network, the set including the second modified training images and one or more training images that do not contain the target object; and

[0057] For each training image in the set, the second network is used to determine the likelihood that the corresponding training image contains the target object, and the weights of the second network are updated using a second loss function different from the first loss function. When the training image contains the target object, the second loss function is a monotonically decreasing function of the determined likelihood, and when the training image does not contain the target object, the second loss function is a monotonically increasing function of the determined likelihood.

[0058] In a fourth aspect, the present invention provides computer software including instructions that, when executed on a computer processing system, cause the computer processing system to perform semantic image segmentation using a first classifier neural network of a model trained by the methods disclosed herein.

[0059] In a fifth aspect, the present invention provides a computer processing system configured to implement a first classifier neural network of a model trained by the methods disclosed herein for performing semantic image segmentation.

[0060] Therefore, it can be seen that, according to the present invention, two classifier neural networks are trained independently in an adversarial manner using two different loss functions. The first classifier neural network is trained to identify discriminative regions that are large enough that the second classifier neural network cannot identify the target object once it is removed from the first training image. Simultaneously, the second network is trained separately to improve its classification performance on modified images generated by the first network, thereby forcing the first network to improve its performance. Thus, the model can be trained in an adversarial minimax manner, enabling the first network to learn to identify regions that accurately correspond to the target object.

[0061] Once trained, the first classifier neural network can be used in inference mode to perform accurate semantic image segmentation, for example by identifying corresponding sets of pixels in an input image that are discriminative of one or more target objects, and combining these sets of pixels (e.g., using the argmax operation) to produce a segmentation map of the input image.

[0062] For convenience, the first classifier neural network will be referred to as the "localizer" network in this paper, and the second classifier neural network will be referred to as the "adversarial" network.

[0063] As will be understood, this approach with two independently trained networks differs from models using a single network trained to minimize a single loss function containing both classification and attention mining loss terms, as described, for example, in [Li, K., Wu, Z., Peng, KC, Ernst, J., Fu, Y.: “Tell Me Where to Look: Guided Attention Inference Networks,” in: IEEE Transactions on Computer Vision and Pattern Recognition, (2018) 9215–9223]. As explained below, this method has been found to offer better classification and localization performance because it is better adapted to the different data distributions on which its two independently trained networks operate.

[0064] The localizer and adversarial network can be any convolutional neural network. They can be the same or different types of networks. However, the localizer and adversarial network are preferably two independent networks—that is, they do not share weights.

[0065] Training the second network may include using the first network, with updated weights determined through training the first network, to identify pixels in training images that are discriminative of the target object. Training the first network may include using the second network, with updated weights determined through training the second network, to determine the likelihood that a modified training image contains the target object.

[0066] Adversarial training can be performed iteratively—that is, using an iterative, alternating training process. The localizer network can be trained on one or more training images, where the classifier network operates in inference mode, for example, during a first time period. The classifier network can also be trained on one or more training images, where the localizer network operates in inference mode, for example, during a second time period different from the first time period. The weights of the first network can be updated independently of the weights of the second network—that is, no weights of the adversarial network are changed when updating the weights of the localizer network, and no weights of the localizer network are changed when updating the weights of the adversarial network. Training the model can include training the first network and the second network alternately. This alternating training can be repeated many times—for example, tens, hundreds, or thousands of cycles.

[0067] The second training image may be different from the first training image, or it may be the same. More generally, the network may be trained on a common set of training images, or it may be trained on different (e.g., non-overlapping) sets of training images.

[0068] The localizer and adversarial network can be the only two classifier neural networks in the model. This allows the model to be trained faster compared to models that require training three or more classifier models.

[0069] The pixels identified by the localizer network may correspond to one or more regions that are discriminative to the target object. For the localizer network, the pixel may be a pixel that is relatively more discriminative than any other pixel in the image. The pixel may be determined from the attention map (i.e., class activation map) of the localizer network. The pixel may be determined from the global average pooling layer and fully connected layers of the localizer network. The pixel can be identified by applying a hard or soft threshold to the attention map data obtained from the localizer network. The attention map data may include values ​​representing the relative discriminative power of the corresponding pixel in the training image to the target object.

[0070] Attenuating or erasing identified pixels may include reducing the pixel value of each identified pixel. This may include reducing each pixel value to zero (i.e., erasing the pixel). However, in some embodiments, it includes using differentiable operations to attenuate the pixels—for example, by applying a soft thresholding function to the attention map data. This allows gradients to still propagate back.

[0071] The localizer network is preferably trained to classify the target object. Therefore, training the localizer network may include inputting a training image set into the localizer network, the set including one or more training images containing the target object and one or more training images not containing the target object. This may include, for each training image in the set, using the localizer network to determine the likelihood that the corresponding training image contains the target object, and using a loss function to update the localizer network, wherein the loss function is a monotonically decreasing function of the determined likelihood when the training image contains the target object, and a monotonically increasing function of the determined likelihood when the training image does not contain the target object. This loss function may be the same as that disclosed above. It may include a first loss term associated with the likelihood determined by the adversarial network and a second loss term for correctly classifying the target object. The first network may be trained using the first training images to classify the target object.

[0072] The localizer and adversarial network are preferably image-level classifier networks. The model is preferably trained using training data including training images and associated image-level labels. The model is preferably an end-to-end model. The adversarial network is preferably trained using backpropagation independently of the localizer network.

[0073] The localizer network is preferably trained to support the recognition of a small set of discriminative pixels. This helps prevent the localizer from supporting a trivial solution where the entire image is recognized as a discriminative one. Therefore, training the localizer network may include updating the weights of the localizer network using a loss function that is a function of the number of pixels recognized in the training image—for example, a function of the number of pixels recognized. This loss function may monotonically increase with the number of pixels recognized. When the pixels are recognized via thresholded attention map data, this loss function may be a function of the attention map data output by the localizer network. The loss function may penalize attention maps that have a high average or total value on some or all of the image. It may be a monotonically increasing function of the sum of pixel values ​​on all (or part of) the attention map of the training image.

[0074] The localizer network can be trained to classify the target object and / or support the identification of smaller distinguishing pixel sets and / or reduce the success of the adversarial network during a single weight update operation using a single localizer loss function. The same first training image containing the target object can be used in the update operation.

[0075] Therefore, the localizer network can be trained using a localizer loss function, which is a monotonically increasing function of the likelihood that the first modified training image contains the target object, determined by the adversarial network.

[0076] The monotonically decreasing function of the likelihood that the first trained image contains the target object, determined by the localizer network; and / or

[0077] A monotonically increasing function of the number of pixels identified by the localizer network, or a monotonically increasing function of the sum of all or part of the values ​​in the attention map generated by the localizer network from the first training image.

[0078] The first and / or second training images may contain more than one target object, which may fall into different categories. The first and second classifier networks may be trained to classify multiple target object types (i.e., categories). Each target object may be a solid object (e.g., "bicycle"), or it may be any other entity (e.g., "sky" or "purple"). The image may contain a target object even if it contains only a portion of a real-world object (e.g., half of a person).

[0079] The training can be implemented by training logic in software or a computer processing system. Training of any of the neural networks may include a gradient descent process. The model can be trained on a plurality of training images, which may include 100, 1000, 10000 or more images.

[0080] In some embodiments, each training image may be a photographic image from a camera—for example, a frame from a video stream. It may be an image of a street view.

[0081] The model may have a training mode and a trained (inference) mode. Training may be performed during a training phase. After the training phase, the model may be configured to receive an input image (or an uncalibrated predicted label distribution of the input image) and segment the input image. In some embodiments, weights may be extracted from a trained first localizer neural network and used to create a model for semantic image segmentation. This may be an independent inference model that includes the first classifier neural network but does not include a second classifier neural network or any training logic. The trained model may be configured to output a segmentation map of the input image. It can generate the segmentation map by operating the first network to identify corresponding pixels (e.g., from activation maps) in the input image that are discriminative for each of the target object sets. It may combine multiple activation maps or thresholded pixel masks to generate the segmentation map.

[0082] The computer processing system may include input for receiving image data from the camera. It may be an onboard computer processing system. It may be configured to output segmented data—for example, to an autonomous driving system.

[0083] The training images, modified training images, inference input images, and activation maps can be represented and encoded in any suitable manner. They can be rescaled between operations. Data (including training data) can be stored in and accessed from a database or other data retrieval system. The weights of the neural network can be stored as values ​​in digital memory.

[0084] The methods disclosed herein may be fully or entirely computer-implemented. The computer software disclosed herein may be on transient or non-transitory computer-readable media. The model may be implemented on one or more additional computer processing systems, different from the computer processing system configured to train the model—for example, wherein the systems are arranged to communicate via physical interfaces. However, in other embodiments, the computer processing system is also configured to implement the first network and / or the second network.

[0085] The first and second classifier neural networks may each include any number of convolutional layers, dense blocks, and other processing layers. The model logic and / or training logic may include software instructions for one or more processors, or may include dedicated hardware logic, or may be implemented using a combination of software and dedicated hardware. The software may include instructions stored in the memory of a computer processing system. The computer processing system may include one or more of the following: CPU, DSP, GPU, FPGA, ASIC, volatile memory, non-volatile memory, inputs, outputs, a display, network connectivity, power supply, radio, clock, and any other suitable components. It may include one or more servers or supercomputers. It may include a microcontroller or system-on-a-chip (e.g., when implementing a trained model for inference operations). It may be configured to store, display, or output segmentation maps or other segmentation data.

[0086] Features of any aspect or embodiment described herein may be applied, where appropriate, to any other aspect or embodiment described herein. When referring to different embodiments or groups of embodiments, it should be understood that these embodiments are not necessarily different, but may overlap. Attached Figure Description

[0087] Some preferred embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0088] Figure 1 This is a high-level schematic diagram illustrating the convolutional neural network (CNN) model of this invention;

[0089] Figure 2 This is a schematic diagram of a computer system embodying the implementation model of the present invention;

[0090] Figure 3 This is a more detailed schematic diagram of the model;

[0091] Figure 4 It is an attention map generated by the model embodying the present invention for two instance input images, targeting four different adversarial loss terms; and

[0092] Figure 5 It is a table of six input images, each with a corresponding ground truth label, a segmentation map generated by conventional methods, and a segmentation map generated by the model embodying the present invention. Detailed Implementation

[0093] Semantic segmentation involves predicting the height H, width W, and color space C of an input image. pixel-wise category label y∈{1,...,K} H″x″W The task. In some use cases, such as in autonomous vehicles, image x can be a red-green-blue (RGB) image or video frame of a street scene, and category K can include "road", "sidewalk", "building", "wall", "person", "vehicle", etc.

[0094] This document describes an exemplary end-to-end adversarial erasure model embodying the present invention. For convenience, it will be abbreviated as "EADER".

[0095] In Weakly Supervised Semantic Segmentation (WSSS), the model is not trained on images annotated by humans with pixel-level category labels, but rather on training data labeled with bounding boxes, doodles, dots, or image-level labels. The method described below trains the model using image-level labels, which minimizes the annotation workload. This is typically the most difficult form of WSSS training. The method disclosed in this paper can be adapted to support other, simpler types of WSSS training in variant embodiments.

[0096] First, we propose a novel adversarial model for weakly supervised semantic segmentation. Then, we demonstrate the effectiveness of the proposed method by showing how end-to-end adversarial erasure can be integrated into a conventional weakly supervised semantic segmentation framework and providing experimental test results.

[0097] End-to-end anti-erase ("EADER")

[0098] Figure 1 Presents a high-level overview of the exemplary end-to-end adversarial model 10.

[0099] Model 10 uses two image classifiers: a first image classifier network 12 and a second adversarial image classifier network 16. Each can be instantiated by any suitable convolutional neural network (CNN).

[0100] A first image classifier network 12 receives an input image 11 and is trained to localize a target object (a bird in this example) using an attention map 13. Therefore, this paper will refer to it as the "localizer" network. The attention map 13 is used to create new modified images 15, in which the most discriminative object regions in the input image 11 are erased by a pixel decay operation 14. These modified images 15 are then forwarded through a second network 16, which will be referred to as the "adversarial" classifier network. Its goal is to correctly classify the image 15 even when the target category has been erased.

[0101] The image classifier of the localizer 12 and the adversarial network 16 is optimized using binary cross-entropy loss, but different optimizers are used alternately. To force the localizer network 12 not only to correctly classify image 11, but also to divert its attention to less discriminative object regions, the localizer 12 is trained using a target function that includes an adversarial loss term. This captures the ability of the adversarial network 16 to still classify erased objects. The localizer 12 is regularized using an additional regularization loss term to prevent it from finding a trivial solution where the localizer 12 identifies the entire image 11 as discriminative, causing the entire image to subsequently hide to the adversarial network 16. This regularization limits the attention of the localizer 12 and forces it to erase only regions belonging to the target class.

[0102] Figure 2 Demonstrating a demonstrative computer processing system 20 on which model 10 can be implemented is an example computer processing system 20. It includes a processor 21 (e.g., Intel). TM The processor 21 is arranged to execute software stored in memory 22. The processor 21 also uses memory 22 to read and write data, such as input data, intermediate computation results, and output data. The software can control the processor 21 to perform any of the methods or steps disclosed herein. Specifically, memory 22 may store training software for training model 10, which may include instructions for providing input to and receiving output from localizers and adversarial networks 12, 16, and for performing pixel attenuation operation 14. Computer 20 has input / output peripherals 23—for example, for receiving training data and / or for deriving data encoded by the trained network.

[0103] Once model 10 has been trained, the localizer network 12 can be used in inference mode for image classification. It can be used without the adversarial network 16. The localizer network 12 can be replicated and installed on another computer system similar to system 20. It can be installed in devices such as control or alarm systems of autonomous vehicles. It can be trained to classify objects in street scenes and can be used in inference mode to classify objects in images captured by one or more cameras installed in a vehicle. However, it is not limited to such applications.

[0104] Figure 3 A more detailed demonstration of the end-to-end adversarial erasure framework 10.

[0105] Figure 2 The example training input image x11 is a photograph containing both a dog and a cat, and is therefore labeled with the image-level category label y = {cat, dog}. Other training images may contain only a dog or only a cat (or any other object) and are labeled accordingly. During training, the input image x is forwarded through the localizer network. 12. Extract the corresponding attention map A for each category c of the target object. c 13. Using soft thresholding, convert each attention map 13 into a corresponding binary mask M. c 17, where the pixels of the most discriminative object portion of the corresponding target object (i.e., category) are "1", and all other pixels are "0". The image subtraction module 14 creates a corresponding modified image from each mask 17 by erasing or attenuating the most discriminative pixels from the input image x. 15. Forward the modified image 15 through the adversarial network F. θ 16. Localizer 12 uses classification loss. and countermeasures against losses Both are optimized. These two terms force the localizer 12 to focus its attention on less discriminative parts of the object, while further regularizing it. The loss term encourages the localizer model 12 to bind activations to the minimum necessary region.

[0106] Using classification loss Optimize the adversarial network F θ 16. Adversarial Networks F θ 16 and localizer networks Network 12 can be trained iteratively in alternating steps. A common dataset is used here to train both networks 12 and 16.

[0107] Consider dataset Where N is the number of images, x i It is the input image, and yi Let x be a multi-hot vector of length C, where C is the number of classes, and x is a vector whose class C exists if x is a multi-hot vector. i In the middle, then y i,c =1, and where category C does not exist in x i In the middle, then y i,c =0. Note that in a multi-label setting, multiple categories can appear in the input image, and therefore ∑ c y i，c ≥1.

[0108] Localizer network. Localizer 12 can be instantiated from any convolutional neural network from which class activation maps (CAMs) or gradient-weighted class activation maps (Grad-CAMs) can be extracted. Class activation maps highlight relevant regions of the image and can be obtained by classification using global average pooling layers and fully connected layers. For simplicity, the following explanation assumes the use of CAM, but the same approach can be extended to more advanced attention extraction methods, such as Grad-CAM.

[0109] The localizer G12 has trainable parameters Furthermore, it is trained as a multi-label classifier, which uses binary cross-entropy loss on each label class c for C different categories to minimize the loss term:

[0110]

[0111] Attention graph. Given a trained or partially trained localizer network. 12. The final convolutional layer can be used. Feature maps and classification weights w c Obtain attention graph A for category c c as follows:

[0112]

[0113] Then, A c Normalization makes the maximum activation equal to 1.

[0114] Soft mask. Attention maps for only the true value categories (i.e., categories that actually exist in the input image 11) are retained. These attention maps 13 are then resized to have the same pixel size as the input image 11, and soft thresholding is applied to the input image x. i Generate category-specific masks 17:

[0115] M c (x i )=σ(ω(A c (x i)-ψ)), (3)

[0116] Where σ is an sigmoid nonlinearity, ψ is a threshold, and ω is a scaling parameter that ensures values ​​above the threshold are close to 1 and values ​​below the threshold are close to 0. Unlike conventional thresholding operations, this soft thresholding is differentiable, allowing gradients from any further computation to backpropagate to the localizer. Adjustable parameters (e.g., ψ) can be set by performing a hyperparameter optimization process on the validation set.

[0117] Erase. The pixel attenuation module 14 creates the input image for the adversarial network 16 as follows. 15:

[0118]

[0119] Where ⊙ represents element-wise multiplication.

[0120] Note that in any given modified image 15, only the attention map 13 of a specific class is erased, thus creating multiple images 15 in the case of more than one target class c.

[0121] Adversarial networks. The adversarial network F16 has trainable parameters θ and is trained as a multi-label classifier using the binary cross-entropy loss function.

[0122]

[0123] This is consistent with the classifier loss term in equation (1) used to train the localizer network 12. The same applies, although it is applied to modified images.

[0124] Therefore, the goal of this adversarial classifier network 16 is to classify the same target as before, even though the evidence has been erased.

[0125] Attention mining loss. To encourage the localizer network 12 to completely erase evidence of the target object, we make the localizer network 12 participate in an adversarial game against the adversarial model 16.

[0126] Therefore, the loss function of the localizer 12 also includes an adversarial loss term, which encourages it to learn to force the adversarial network 16 to misclassify the modified image. As the adversarial loss term, this embodiment utilizes an attention mining loss, which is the average of the logarithms of the erased categories:

[0127]

[0128] Regularization loss.To regularize the localizer 12 and prevent it from learning a trivial solution where it erases the entire image to globally minimize the attention mining loss, we impose an additional regularization loss term:

[0129]

[0130] Where W and H represent the width and height of the active area.

[0131] Incorporating this regularization loss term during the optimization process encourages the localizer 12 to find the minimum attention graph that covers the target category.

[0132] Total loss. Therefore, the total loss function used when training localizer 12 is:

[0133]

[0134] Here, α and β are hyperparameters that can be adjusted to tune the importance of adversarial and regularization losses, respectively—for example, based on the results of empirical tests.

[0135] Localizer 12 is trained to minimize the adversarial loss term. The adversarial model 16 attempts to minimize the loss function in equation (5) by being trained. Maximize the resistance loss term

[0136] Segmentation image. After computer system 20 trains model 10 with the described loss term, localizer network 12 can be used in inference mode by transforming attention map 13 into segmentation map. This can be done using the same computer system 20, or by replicating localizer 12 and implementing it on one or more inference systems. The attention map is upsampled to image resolution and stacked together using C+1 channels. Since classification model 10 is not trained against the background category, the first channel is set to a threshold ρ. Segmentation masks can then be obtained by taking argmax in the class dimension. These segmentation masks can be further processed as needed—for example, when used in a vehicle, to detect potential collisions with pedestrians and issue warnings and / or take evasive driving actions in response.

[0137] Experimental results

[0138] set up

[0139] Model.The method described herein can be integrated into other systems. This is demonstrated by integrating the end-to-end adversarial erasure (EADER) scheme described above into an existing WSSS method called pixel-level semantic similarity (PSA), as described, for example, in Ahn et al. [Ahn, J., Kwak, S.: Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (2018) 4981–4990].

[0140] PSA consists of three stages. The first stage trains a classification network to generate a CAM (Classification Image). However, this stage suffers from the problem of "discriminative localization," where its learning focuses only on the most discriminative object regions. No specific method exists to improve the segmentation mask in this stage, but it is augmented using a large amount of training and testing-time data. The segmentation mask generated from the CAM is used to generate two sets of labels for the second stage. The first set consists of labels with high confidence for background regions, while the second set consists of labels with high confidence for object regions. In the second stage, AffinityNet is used to learn semantic similarity, which propagates local responses in the graph to nearby regions belonging to the same semantic entity. These two distinct sets allow for reliable training of AffinityNet because low-confidence regions are discarded. The AffinityNet output predictions are used as pseudo-labels in the final stage, where they are used to train a fully supervised semantic segmentation network.

[0141] The CAM generation stage of PSA is suitable for adversarial training using the method disclosed in this paper because it suffers from discriminative localization, and because the classification network is suitable as a localizer network. Specifically, it is suitable because the CAM is generated from the final convolutional layer without any post-processing or other gradient-breaking computations.

[0142] In experimental testing, the PSA implementation was used as localizer network 12, and as described above, a soft threshold was applied to the attention map 13 generated using PSA to create a mask 17, which was used to erase the most discriminative object regions from the input image 11. The resulting image 15 was then forwarded through adversarial network 16, where an attention mining loss was applied. It is applied as an adversarial loss to the PSA localizer network 12.

[0143] Dataset.We evaluate the performance of the method described in this paper on the Pascal VOC 2012 segmentation dataset, a widely used benchmark for weakly supervised semantic segmentation. The dataset consists of twenty object categories and one background category, containing 1464, 1449, and 1456 images in the training, validation, and test sets, respectively. We augment the dataset with annotations from Hariharan et al. [Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S., Malik, J.: "Semantic contours from inverse detectors," in: International Conference on Computer Vision (ICCV), (2011)], generating a total of 10582 training images. We determine the mean intersection-union ratio (mIoU) for the validation and test sets. Test set results are obtained using the official Pascal VOC evaluation server.

[0144] No post-processing was used, and the method did not employ techniques such as test time augmentation, post-processing, or saliency hints for our integrated method.

[0145] Network architecture details. We test the adversarial training method using a ResNet-101 localizer network and a ResNet-18 adversarial network. We utilize ImageNet pre-trained weights for both networks. When integrating with PSA, to ensure a fair comparison, we do not modify any existing networks, meaning the localizer is a WideResNet with 38 convolutional layers, while the adversarial model remains ResNet-18. In the final stage, we train a fully supervised semantic segmentation network with agent labels. We utilize the segmentation model DeepLabV3+ with a ResNet-101 and Xception-65 backbone and the default training policy.

[0146] Training standards.We train the localizer with a batch size of 16, while the batch size of the adversarial model is dynamic because it depends on the number of objects in each image. For example, when each image in batch 16 has two object categories, the two objects are erased from each image separately, and the batch size of the adversarial network is 32. We randomly resize and crop the input images to 448×448 for both the localizer and the adversarial model. Both networks are optimized for ten epochs with stochastic gradient descent and a learning rate of 0.01. We train the localizer and the adversarial model alternately every 200 training steps.

[0147] Throughout the experiment, unless otherwise instructed, the values ​​of α and β in equation (8) shall be set to α = 0.05 and β = 10. -5 The other hyperparameter values ​​in equation (3) are ω = 100, Ψ = 0.5, and ρ = 0.3 as the first channel threshold. We followed the training settings of Ahn et al. when training the PSA and used an initial learning rate of 0.01 for the adversarial network. In order to generate attention maps, we utilized Grad-CAM in our experiments [Selvaraju, RR, Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Gradcam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE International Conference on Computer Vision] Vision, (2017) 618–626], while when integrating into PSA we used CAM [Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization, see: IEEE Transactions on Computer Vision and Pattern Recognition, (2016) 2921–2929], to maintain a fair comparison.

[0148] Ablation Research

[0149] Our first experiment was to validate the ablation study in which the adversarial network forces the localizer network to divert its attention to less discriminative object regions.

[0150] Recalling Equation (8), α controls the strength of the adversarial loss term. The table below shows the results of varying the α parameter on the Pascal VOC 2012 validation set, and reports the mIoU, precision, and recall of the segmentation mask. We report the mean and standard deviation of six runs.

[0151]

[0152] Figure 4 Examples of attention maps generated for input images are shown, obtained using Grad-CAM from the novel end-to-end adversarial erasure (EADER) method described in this paper, with different values ​​applied to the adversarial loss term α. The top row shows the attention map for images of people riding horses in the "horse" category. The bottom row shows the attention map for images of people petting cats in the "person" category. As the α value increases, attention can be seen diverted to less discriminative object regions.

[0153] Comparison with PSA

[0154] We now compare the raw PSA results with those from the integrated end-to-end anti-erase (“EADER”) that we have integrated as disclosed in this paper.

[0155] The table below shows the improvement in mIoU compared to our baseline (pixel-level semantic similarity (PSA) method on the Pascal VOC 2012 validation set). To achieve a fair comparison, we reproduce the PSA numbers and train agent labels from AffinityNet on DeepLabV3+. The numbers represent the results of our reproduction.

[0156]

[0157] In addition, we report the accuracy and recall rates after the CAM generation stage.

[0158] End-to-end adversarial erasure (EADER) improves performance across all metrics in this stage. In other words, the combination of adversarial and regularized loss terms forces the attention map to be distributed across less discriminative object regions rather than background regions. Furthermore, the mIoU score in this stage is higher than the score reported in the first table above, which is due to the significant test time augmentation used by PSA.

[0159] In the next stage, training AffinityNet with the improved output from the first stage again yields a better mIoU score. Finally, we report the results for training a fully supervised semantic segmentation model with proxy labels generated by AffinityNet. Since we utilize DeepLabV3+ instead of ResNet-38 as the fully supervised semantic segmentation model, we also report the results for DeepLabV3+ trained with proxy labels without end-to-end adversarial erasure. Again, this leads to improvements in mIoU, demonstrating the ensembleability of end-to-end adversarial erasure into existing WSSS methods.

[0160] In the two tables below, we perform per-class comparisons on the validation and test sets.

[0161] This table shows a per-class comparison with pixel-level semantic similarity (PSA) on the Pascal VOC 2012 validation set with only image-level supervision:

[0162]

[0163] The table below shows a per-category comparison with pixel-level semantic similarity (PSA) on the Pascal VOC 2012 test set with only image-level supervision:

[0164]

[0165] Note that in this last table, the PSA method uses the more powerful WideResNet-38 as the final fully supervised segmentation model instead of DeepLabV3+.

[0166] Discriminative localization is particularly prevalent in non-rigid object categories. We found that end-to-end adversarial erasure (EADER) significantly improves results in many non-rigid object categories, such as birds, cats, cows, and horses. Typically, in these categories, the most discriminative object regions are the head or feet, causing the attention map to cover only a small portion of these object categories. With end-to-end adversarial erasure, the localizer is forced to capture the entire object region because, while the fur or skin of these object categories is less discriminative, it is still identifiable. For outdoor object categories, the results are generally similar to PSA, while for indoor object categories, performance typically degrades. Overall, end-to-end adversarial erasure improves performance, despite utilizing a less robust fully supervised segmentation model.

[0167] Figure 5This section presents some qualitative results on the Pascal VOC 2012 validation set. The white edges in the ground truth mask represent pixels ignored during evaluation. These images demonstrate improvements in precision, recall, and mIoU. For input images containing a pair of cats, a bird, a horse, and a train, the first four rows show that End-to-End Adversarial Eraser (EADER) segments objects better by capturing less discriminative object regions (including those for non-rigid object categories) (i.e., more pixels correctly identifying the target object category compared to regular PSA). For images containing a person and a train, the last two rows show higher specificity (i.e., fewer pixels misclassified as belonging to the target category outside the target object), which can be attributed to regularization terms that force attention to be focused only on regions where the localizer is certain they are object regions.

[0168] Comparison with anti-erasure methods

[0169] The table below compares our results with known WSSS methods that follow an adversarial erasure strategy on the Pascal VOC 2012 dataset. For supervised learning, Let S denote the image-level label, and S denote the saliency mask. The results show that EADER outperforms all existing adversarial erasure methods, even though most of them use stronger supervisory signals in the form of saliency masks. When compared to AcoL (the only other adversarial erasure method without a saliency mask), we significantly outperform it on the validation set.

[0170]

[0171] AE-PSL is described in [Wei, Y., Feng, J., Liang, X., Cheng, MM, Zhao, Y., Yan, S.: “Object region mining using adversarial erasure: a simple classification of semantic segmentation methods”, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017) 1568–1576].

[0172] GAIN is described in [Li, K., Wu, Z., Peng, KC, Ernst, J., Fu, Y., “Tell me where to look: Guided attention inference networks”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018) 9215–9223].

[0173] SeeNet is described in [Hou, Q., Jiang, P., Wei, Y., Cheng, MM: "Self-erasing network for integral object attention", in Advances in Neural Information Processing Systems, (2018) 549–559]. (The last part is a typo and can be left as is.) The results are also obtained from here.

[0174] ACoL is described in [Zhang, X., Wei, Y., Feng, J., Yang, Y., Huang, TS: Adversarial Complementary Learning for Weakly Supervised Object Localization, in: IEEE Transactions on Computer Vision and Pattern Recognition, (2018) 1325–1334].

[0175] Comparison with the latest technologies

[0176] The table below compares EADER with known WSSS methods on the Pascal VOC 2012 dataset. It shows the feature extractor for generating object locations, the fully supervised model trained with respect to proxy labels (where applicable), and the supervised form. The method is represented by the supervision signal F, which indicates the upper bound of segmentation performance. Note that our method outperforms a fully supervised FCN network, and we achieve approximately 73% of the upper bound set by the fully supervised DeepLabV3+ model. Furthermore, it demonstrates that EADER achieves performance comparable to methods using only image-level labels. Its performance competes with other models.

[0177] Regarding supervision, Let S represent the image-level label, S represent the saliency mask, and F represent the pixel-level label, which is the upper bound of fully supervised semantic segmentation. For the feature extractor, the architecture mentioned is the architecture used to generate initial object locations (e.g., by using CAM):

[0178]

[0179] FCN is described in [Long, J., Shelhamer, E., Darrell, T., “Fully convolutional networks for semantic segmentation”, in: IEEE Transactions on Computer Vision and Pattern Recognition, (2015) 3431–3440].

[0180] WideResNet-38 is described in [Wu, Z., Shen, C., Van Den Hengel, A.: "Wider or deeper: Revisiting the ResNet model for visual recognition," *Pattern Recognition*, 90(2019), 119–133].

[0181] DeepLabV3+ is described in [Chen, LC, Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic imagesegmentation, in Proceedings of the European Conference on Computer Vision (ECCV), (2018) 801–818].

[0182] FickleNet is described in [Lee, J., Kim, E., Lee, S., Lee, J., Yoon, S.: "FickleNet: Weakly and semi-supervised semantic image segmentation using stochastic inference", in IEEE Transactions on Computer Vision and Pattern Recognition, (2019) 5267–5276].

[0183] Fan et al. [Fan, R., Hou, Q., Cheng, MM, Yu, G., Martin, RR, Hu, SM: "Associating inter-image salient instances forweakly supervised semantic segmentation", Proceedings of the European Conference on Computer Vision (ECCV), (2018) 367–383] describe

[0184] OAA+ is described in [Jiang, PT, Hou, Q., Cao, Y., Cheng, MM, Wei, Y., Xiong, HK: "Integral object mining via online attention accumulation", in: IEEE International Conference on Computer Vision, (2019) 2070–2079].

[0185] EM Adapt is described in [Papandreou, G., Chen, LC, Murphy, KP, Yuille, AL: "Weakly-and semi-supervised learning of a deep convolutional network for semantic imagesegmentation", Proceedings of the IEEE International Conference on Computer Vision, (2015) 1742–1750].

[0186] SEC describes this in [Kolesnikov, A., and Lampert, CH: Seed, expand and constrain: Three principles for weakly-supervised image segmentation, in: European Conference on Computer Vision, Springer, (2016) 695–711].

[0187] MMEF is described in [Ge, W., Yang, S., Yu, Y.: "Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning", in: IEEE Transactions on Computer Vision and Pattern Recognition, (2018) 1277–1286].

[0188] RRM and SSDD are described in [Zhang, B., Xiao, J., Wei, Y., Sun, M., Huang, K.: "Reliability does matter: An end-to-end weakly supervised semantic segmentation approach", in: 34th AAAI Conference on Artificial Intelligence, AAAI 2020, 32nd Innovative Application of the Conference on Artificial Intelligence, IAAI 2020, 10th AAAI Workshop on Advances in Artificial Intelligence Education, EAAI 2020, New York, NY, USA, February 7-12, 2020, AAAI Press, (2020) 12765–12772].

[0189] Aralsanov and Roth describe [Aaraslanov, N., Roth, S.: Single-stage semantic segmentation from image labels, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, (2020) 4253–4262]

[0190] SSDD is described in [Shimoda, W., Yanai, K., "Self-supervised difference detection for weakly-supervised semanticsegmentation", Proceedings of IEEE International Conference on Computer Vision, (2019) 5208–5217].

[0191] Therefore, it can be seen that the novel end-to-end adversarial erasure (EADER) method helps to address the "discriminative localization" problem, an inherent issue in weakly supervised semantic segmentation methods. Unlike many previous adversarial erasure methods, this method can be integrated with existing methods without requiring iterative classifiers, post-processing, weight sharing, or saliency masks. It has been demonstrated to improve performance on the PascalVOC 2012 dataset, particularly on non-rigid categories, which are most susceptible to the discriminative localization problem.

[0192] Those skilled in the art will appreciate that the invention has been described by way of one or more specific embodiments, but is not limited to these embodiments; many changes and modifications are possible within the scope of the appended claims.

Claims

1. A method for training a model for use in semantic image segmentation, wherein the model comprises: First classifier neural network; and The second classifier neural network, The method includes training the first classifier neural network in the following manner: The first training image containing the target object is input into the first classifier neural network; The first classifier neural network is used to identify pixels in the first training image that are discriminative to the target object; The first modified training image is generated by attenuating or erasing the identified pixels in the first training image; The first modified training image is input into the second classifier neural network; The second classifier neural network is used to determine the likelihood that the first modified training image contains the target object; and The weights of the first classifier neural network are updated using a first loss function, which is a monotonically increasing function of the determined likelihood. The method includes training the second classifier neural network in the following manner: The second training image containing the target object is input into the first classifier neural network; The first classifier neural network is used to identify pixels in the second training image that are discriminative to the target object; A second modified training image is generated by attenuating or erasing the identified pixels in the second training image; The training image set is input into the second classifier neural network, the set including the second modified training image and one or more training images that do not contain the target object; and For each training image in the set, the second classifier neural network is used to determine the likelihood that the corresponding training image contains the target object, and the weights of the second classifier neural network are updated using a second loss function different from the first loss function. When the training image contains the target object, the second loss function is a monotonically decreasing function of the determined likelihood, and when the training image does not contain the target object, the second loss function is a monotonically increasing function of the determined likelihood.

2. The method according to claim 1, wherein the first classifier neural network and the second classifier neural network are independent convolutional neural networks with independent weight sets.

3. The method according to claim 1 or 2, wherein the first classifier neural network and the second classifier neural network are trained alternately in a plurality of training cycles.

4. The method according to claim 1 or 2, comprising: The first classifier neural network is trained while the second classifier neural network operates in inference mode; Update the weights of the first classifier neural network without changing any weights of the second classifier neural network; The second classifier neural network is trained while the first classifier neural network is trained in inference mode. and The weights of the second classifier neural network are updated without changing any weights of the first classifier neural network.

5. The method according to claim 1 or 2, comprising training the first classifier neural network and the second classifier neural network with respect to a common plurality of training images.

6. The method according to claim 1 or 2, wherein the first classifier neural network and the second classifier neural network are the only classifier neural networks in the model.

7. The method of claim 1 or 2, wherein the pixels of the first training image identified by the first classifier neural network are pixels that are relatively more discriminative to the first classifier neural network than any other pixel of the image.

8. The method of claim 1 or 2, comprising identifying the pixels of the first training image by applying a hard or soft threshold to attention map data obtained using the first classifier neural network.

9. The method according to claim 1 or 2, further comprising training the first classifier neural network to classify the target object.

10. The method of claim 1 or 2, wherein the first classifier neural network and the second classifier neural network are image-level classifier networks, and wherein the model is trained using training data including training images and associated image-level labels.

11. The method of claim 1 or 2, further comprising training the first classifier neural network to support the identification of a small set of discriminative pixels by means of the first loss function, wherein the first loss function is either a monotonically increasing function of the number of pixels identified by the first classifier neural network or a monotonically increasing function of the sum of all or part of the values ​​in the attention map of the first training image generated using the first classifier neural network.

12. The method according to claim 1 or 2, comprising training the first classifier neural network and the second classifier neural network to classify a plurality of target object types.

13. A computer-readable medium including instructions that, when executed on a computer processing system, cause the computer processing system to perform semantic image segmentation using a first classifier neural network of a model trained by the method of any of the preceding claims.

14. A computer processing system configured to implement a first classifier neural network of a model trained by the method of any one of claims 1 to 12 for performing semantic image segmentation.

15. The computer processing system of claim 14, wherein the computer processing system is a computer processing system for a vehicle, and includes input for receiving image data from a camera and output for outputting segmented data to an autonomous driving system of the vehicle.

16. A computer processing system for training a model for use in semantic image segmentation, The model mentioned above includes: First classifier neural network; and The second classifier neural network, The computer processing system is configured to train the first classifier neural network in the following manner: The first training image containing the target object is input into the first classifier neural network; The first classifier neural network is used to identify pixels in the first training image that are discriminative to the target object; The first modified training image is generated by attenuating or erasing the identified pixels in the first training image; The first modified training image is input into the second classifier neural network; The second classifier neural network is used to determine the likelihood that the first modified training image contains the target object; and The weights of the first classifier neural network are updated using a first loss function, which is a monotonically increasing function of the determined likelihood. The computer processing system is configured to train the second classifier neural network in the following manner: The second training image containing the target object is input into the first classifier neural network; The first classifier neural network is used to identify pixels in the second training image that are discriminative to the target object; A second modified training image is generated by attenuating or erasing the identified pixels in the second training image; The training image set is input into the second classifier neural network, the set including the second modified training image and one or more training images that do not contain the target object; and For each training image in the set, the second classifier neural network is used to determine the likelihood that the corresponding training image contains the target object, and the weights of the second classifier neural network are updated using a second loss function different from the first loss function. When the training image contains the target object, the second loss function is a monotonically decreasing function of the determined likelihood, and when the training image does not contain the target object, the second loss function is a monotonically increasing function of the determined likelihood.

17. The computer processing system of claim 16, wherein the computer processing system is configured to implement the first classifier neural network and the second classifier neural network.

18. The computer processing system according to claim 16 or 17, wherein the first classifier neural network and the second classifier neural network are independent convolutional neural networks with independent weight sets.

19. The computer processing system of claim 16 or 17, configured to alternately train the first classifier neural network and the second classifier neural network in a plurality of training cycles.

20. The computer processing system according to claim 16 or 17, configured to: The first classifier neural network is trained while the second classifier neural network operates in inference mode; Update the weights of the first classifier neural network without changing any weights of the second classifier neural network; The second classifier neural network is trained while the first classifier neural network operates in inference mode; and The weights of the second classifier neural network are updated without changing any weights of the first classifier neural network.

21. The computer processing system according to claim 16 or 17, wherein the first classifier neural network and the second classifier neural network are the only classifier neural networks in the model.

22. The computer processing system of claim 16 or 17, wherein the pixels of the first training image identified by the first classifier neural network are pixels that are relatively more discriminative to the first classifier neural network than any other pixel of the image.

23. The computer processing system of claim 16 or 17, configured to identify the pixels of the first training image by applying a hard or soft threshold to attention map data obtained using the first classifier neural network.

24. The computer processing system of claim 16 or 17, configured to train the first classifier neural network to classify the target object.

25. The computer processing system of claim 16 or 17, wherein the first classifier neural network and the second classifier neural network are image-level classifier networks, and wherein the computer processing system is configured to train the model using training data including training images and associated image-level labels.

26. The computer processing system of claim 16 or 17, configured to train the first classifier neural network to support the identification of a small set of discriminative pixels by means of the first loss function, wherein the first loss function is either a monotonically increasing function of the number of pixels identified by the first classifier neural network or a monotonically increasing function of the sum of all or part of the values ​​in the attention map of the first training image generated using the first classifier neural network.

27. The computer processing system of claim 16 or 17, configured to train the first classifier neural network and the second classifier neural network to classify a plurality of target object types.

28. A computer-readable medium including instructions that, when executed on a computer processing system, cause the computer processing system to train a model for use in semantic image segmentation, wherein the model includes: First classifier neural network; and The second classifier neural network, The instructions therein instruct the computer processing system to train the first classifier neural network in the following manner: The first training image containing the target object is input into the first classifier neural network; The first classifier neural network is used to identify pixels in the first training image that are discriminative to the target object; The first modified training image is generated by attenuating or erasing the identified pixels in the first training image; The first modified training image is input into the second classifier neural network; The second classifier neural network is used to determine the likelihood that the first modified training image contains the target object; and The weights of the first classifier neural network are updated using a first loss function, which is a monotonically increasing function of the determined likelihood. The instructions therein instruct the computer processing system to train the second classifier neural network in the following manner: The second training image containing the target object is input into the first classifier neural network; The first classifier neural network is used to identify pixels in the second training image that are discriminative to the target object; A second modified training image is generated by attenuating or erasing the identified pixels in the second training image; The training image set is input into the second classifier neural network, the set including the second modified training image and one or more training images that do not contain the target object; and For each training image in the set, the second classifier neural network is used to determine the likelihood that the corresponding training image contains the target object, and the weights of the second classifier neural network are updated using a second loss function different from the first loss function. When the training image contains the target object, the second loss function is a monotonically decreasing function of the determined likelihood, and when the training image does not contain the target object, the second loss function is a monotonically increasing function of the determined likelihood.

Images