Generative Adversarial Networks for Image Segmentation

Abstract

The present invention relates to a method for training a generative adversarial network for performing image semantic segmentation. The generative adversarial network includes a generator neural network and a critic neural network. The method includes: using the generator neural network to generate a predicted image segmentation map from an input image, wherein each predicted image segmentation map includes a classification prediction for each of a plurality of pixels of the corresponding input image; providing the predicted image segmentation map generated by the generator neural network to the critic neural network; training the critic neural network to determine the weights of the corresponding pixels of the predicted image segmentation map generated by the generator neural network, wherein the weight of each pixel is used to weight the pixel-level cross entropy term in the objective function of the critic neural network; and using the weight determined by the critic neural network to train the generator neural network.

Description

Background Art

[0001] The present invention relates to the training of generative adversarial networks for image segmentation.

[0002] Certain convolutional neural networks (CNNs) have been successfully applied to various computer vision problems by formulating them as image segmentation problems. Examples include road scene understanding for autonomous driving, and interpreting medical imaging. For such applications, the network is usually trained with multi-class per-pixel labels that together form an image-sized segmentation map. The output of such a network is again an image-sized map representing the probability of each pixel class.

[0003] However, since the output of the semantic segmentation network does not necessarily maintain quality, additional post-processing steps may be required, such as applying a conditional random field (CRF). The generated segmentation map is probabilistic and has a different distribution compared to the corresponding labels. The fundamental reason behind this is that the training loss is formulated in such a way (e.g., per-pixel cross entropy) that each output pixel in the segmentation map is considered independently of all other output pixels, i.e., no explicit pixel-to-pixel consistency is enforced.

[0004] One approach to maintaining quality in semantic segmentation problems is to use a Generative Adversarial Network (GAN) to "learn" the expected loss function for the segmenter network. GANs typically work by training two networks in an alternating fashion in a minimax game: the generator (e.g., segmenter) is trained to produce results, while the discriminator is trained to distinguish between the predicted segmentation map data (i.e., "fake" data) and the human-labeled true data labels (i.e., "real" data).

[0005] GANs have been applied to the semantic segmentation problem in an attempt to address the per-pixel loss problem. During training, the generator produces semantic segmentation maps, while the discriminator alternately observes the true data labels and the predicted segmentation maps.

[0006] However, applicants have recognized that known GAN discriminators tend to learn to distinguish between real and fake data based on simple differences in the value distribution between fuzzy predictions of fake data (i.e., fractional values ​​between 0 and 1) and hard classifications of real data (i.e., 0 or 1), which can lead to suboptimal classification results.

[0007] The present invention seeks to provide a novel approach to training generative adversarial networks that may lead to better image segmentation performance, at least for certain types of inputs. Summary of the invention

[0008] From a first aspect, the present invention provides a method for training a generative adversarial network for performing image semantic segmentation, wherein the generative adversarial network comprises:

[0009] Generator neural network; and

[0010] The judge neural network,

[0011] The method comprises:

[0012] using the generator neural network to generate predicted image segmentation maps from input images, wherein each predicted image segmentation map comprises a classification prediction for each of a plurality of pixels of a corresponding input image;

[0013] providing one or more predicted image segmentation maps generated by the generator neural network to the judge neural network;

[0014] training the critic neural network to determine weights for corresponding pixels of a predicted image segmentation map generated by the generator neural network, wherein the weight for each pixel is used to weight a pixel-wise cross entropy term in an objective function of the critic neural network; and

[0015] The generator neural network is trained using the weights determined by the critic neural network.

[0016] Viewed from a second aspect, the present invention provides a computer processing system implementing a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises:

[0017] Generator neural network;

[0018] The judge neural network; and

[0019] Training logic,

[0020] And wherein the training logic is configured to:

[0021] operating the generator neural network to generate predicted image segmentation maps from the input images, wherein each predicted image segmentation map comprises a classification prediction for each of a plurality of pixels of the corresponding input image;

[0022] providing one or more predicted image segmentation maps generated by the generator neural network to the judge neural network;

[0023] training the critic neural network to determine weights for corresponding pixels of a predicted image segmentation map generated by the generator neural network, wherein the weight for each pixel is used to weight a pixel-wise cross entropy term in an objective function of the critic neural network; and

[0024] The generator neural network is trained using the weights determined by the critic neural network.

[0025] Viewed from a third aspect, the present invention provides computer software comprising instructions which, when executed on a computer processing system, cause the computer processing system to train a generative adversarial network for performing semantic image segmentation, wherein the generative adversarial network comprises:

[0026] Generator neural network; and

[0027] The judge neural network,

[0028] and wherein the instructions cause the computer processing system to:

[0029] operating the generator neural network to generate predicted image segmentation maps from the input images, wherein each predicted image segmentation map comprises a classification prediction for each of a plurality of pixels of the corresponding input image;

[0030] providing one or more predicted image segmentation maps generated by the generator neural network to the judge neural network;

[0031] training the critic neural network to determine weights for corresponding pixels of a predicted image segmentation map generated by the generator neural network, wherein the weight for each pixel is used to weight a pixel-wise cross entropy term in an objective function of the critic neural network; and

[0032] The generator neural network is trained using the weights determined by the critic neural network.

[0033] The computer software may further include instructions for implementing the generator neural network and / or implementing the discriminator neural network. The computer software may be stored on a non-transitory computer readable medium (e.g., a magnetic or solid-state memory), or may be carried by a transient signal (e.g., an electrical or electromagnetic signal).

[0034] Thus, it can be seen that, according to the present invention, instead of a conventional discriminator network that is trained to distinguish between true and false segmentation maps, a judge network is provided that learns to assess the relative reliability of different pixel classifications on an image. This can be used to improve the predictions of the generator (segmenter) network. In particular, this approach allows the judge neural network to use pixel-to-pixel consistency clues to find incorrect predictions, and thus can help train the generator to focus more on high-level structure.

[0035] This helps overcome the shortcomings of conventional GAN ​​methods, which can learn to discriminate primarily based on pixel values ​​rather than properly considering higher-level structure. The use of a critic network as disclosed herein can avoid the generator (segmenter) network being forced to push its predictions toward zeros and ones in order to mimic the low-level nature of human annotations. This enables the network to better express uncertainty. It also avoids the softmax probability vector from being too close to exact zeros and ones, which translate to infinite logits, and thus can force the critic to learn more complex geometric differences rather than just carefully examining the small but still detectable value gaps between true and false distributions. This can allow the generator of the adversarial network to properly learn to preserve scene structure, such as learning the structural continuity of objects commonly seen in street scenes.

[0036] The weights may represent the judges' relative confidence in the relative accuracy of the pixel classifications for the entire image, eg, the judges place lower weights on pixels for which they have greater confidence in the correct classification.

[0037] The training of the generator neural network and the critic neural network may include performing a minimax training process. As shown below, the minimax game between such a critic network and the generator can lead to significant improvements in the generator segmentation, both for structural metrics and pixel-level metrics.

[0038] By using the weights in the training of the generator, the generator can be trained to leave as few structural clues as possible in the segmented image, which will enable the judge to identify where to place high weights that increase its objective function.

[0039] For a particular segmentation map, evaluating the objective function of the critic neural network may include calculating the sum of the product of the weight for each pixel and the cross entropy loss term for the corresponding pixel over all pixels of the segmentation map. This sum may then be scaled by a scaling factor (e.g., depending on the size of the image), such as divided by the number of pixels in the image. The critic objective function may decrease as this sum or scaled sum increases, such as being equal to the negative of the sum or scaled sum. The critic neural network may be trained to maximize this critic objective function.

[0040] The generator neural network may be trained to minimize a generator objective function (i.e., a loss function). The generator objective function may be configured to decrease as the judge objective function increases. It may include an inverse term (e.g., a negative term) of the judge objective function. It may also include a pixel-level cross entropy term. It may be equal to the pixel-level cross entropy term minus the judge objective function.

[0041] According to embodiments of the invention, the judge neural network may learn to identify and weight more heavily those areas where the network's predictions are more likely to be wrong. (Of course, in other embodiments, the objective function may instead be formulated so that higher weight values ​​are assigned to pixel classifications that the judge considers to be relatively more reliable relative to the classifications of other pixels in the image.)

[0042] The judge network can be limited to only assigning limited weight values ​​to pixels. This prevents it from maximizing its objective function by assigning unlimited weights. The sum of all weights on the image can be limited to a threshold amount (an amount that can or cannot be reached in each weight assignment). The judge can be configured to assign weights to each pixel, wherein each weight is in the range of zero to one. In some embodiments, the weights can be determined by applying a sigmoid function to the intermediate weight values ​​generated in the judge network. Determining the weights can include applying a spatial smoothing function to the intermediate weight values ​​generated in the judge network. Smoothing the weight map can adjust the model so that the weights are distributed over multiple pixels, rather than concentrated on a single location or pixel.

[0043] In some embodiments, the critic neural network receives only the input image and the image segmentation map as forward input. In particular, it does not receive any real data segmentation map. This is in stark contrast to conventional methods, which usually train alternately on predicted (fake) and real data (true) segmentation maps.

[0044] In one set of embodiments, the image segmentation map provided to the judge includes score values ​​representing the likelihood of a pixel belonging to each class.

[0045] However, in another set of embodiments, providing the critic neural network with one or more predicted image segmentation maps generated by the generator neural network may include providing the critic neural network with the argmax of the image segmentation map, i.e., providing a binary classification value. This ensures that the critic network cannot exploit differences in the distribution of values ​​between the true data segmentation map and the generated segmentation map. However, since the argmax is not differentiable, an approximation of the argmax may be used to update the segmenter.

[0046] The computer processing system may include an input for receiving image data from a camera. It may be an on-board computer processing system or a medical image processing system. It may be configured to output segmentation data, for example to an autonomous driving system.

[0047] It should be understood that the pixels or picture elements referred to herein do not necessarily correspond one-to-one to the pixels in the source image. For example, downsampling or upsampling may be performed. The input image may be a two-dimensional image. However, it may be a three-dimensional or n-dimensional image. It may include color information, such as red-green-blue data, or other data layers.

[0048] The generator neural network and / or the judge neural network may include any number of convolutional layers, dense blocks and other conventional layers. The generator neural network and / or the judge neural network and / or the training logic may include software instructions for a processor, or may include dedicated hardware logic, or may include a combination of both. The computer processing system may include one or more of the following: a CPU, a DSP, a GPU, an FPGA, an ASIC, volatile memory, non-volatile memory, input, output, display, network connection, power supply, radio, clock and any other suitable components. It may be configured to store or display or output predicted segmentation maps or other segmentation data.

[0049] Some embodiments may achieve segmentation predictions that are sufficiently similar to the training labels that no additional problem-specific loss terms and / or post-processing steps are required.

[0050] Features of any aspect or embodiment described herein may be applied to any other aspect or embodiment described herein, as appropriate.Where reference is made to different embodiments or groups of embodiments, it should be understood that these are not necessarily different, but may overlap. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Certain preferred implementations will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0052] Figure 1 is a first instance RGB input image of a street scene;

[0053] Figure 2 is a segmentation map of the first instance input image generated using a conventional method;

[0054] Figure 3 is a weighted map of a first instance input image generated using a method embodying the invention;

[0055] Figure 4 is a segmentation map of a first example input image generated using a method embodying the present invention;

[0056] Figure 5 is a schematic diagram of a generative adversarial network embodying the present invention;

[0057] Figure 6 is a performance comparison diagram of a network embodying the present invention and a conventional network;

[0058] Figure 7is a set of images comprising a second example RGB input image of a street scene and various segmentation maps generated using conventional methods and methods embodying the present invention;

[0059] Figure 8 is a third instance RGB input image of a street scene;

[0060] Fig. 9 is the real data segmentation map of the third instance input image;

[0061] Fig.10 is a segmentation map of the third instance input image generated using a conventional method;

[0062] Fig.11 is a segmentation map of a third example input image generated using a method embodying the present invention;

[0063] Fig.12 is a schematic diagram of an alternative generative adversarial network embodying the present invention; and

[0064] Fig.13 is a schematic diagram of a computer system embodying the present invention. DETAILED DESCRIPTION

[0065] The following describes certain example neural networks embodying the present invention that use adversarial training to perform semantic image segmentation. These novel types of artificial neural networks are sometimes referred to herein as "gambling adversarial networks."

[0066] The technology disclosed in this article has been tested on the segmentation of street scenes acquired by cameras on cars, and has been found to be particularly effective in this task. However, it should be understood that these networks can also be applied to other image segmentation problems in different fields.

[0067] Context

[0068] To provide context for the description and terminology used below, conventional approaches to semantic segmentation using generative adversarial networks (GANs) will first be described.

[0069] In a typical adversarial training formulation, the discriminator learns to distinguish between human-annotated real data segmentations (i.e., "true" labels) and predictions provided by the network (i.e., "fake" labels). By introducing the generator (which can be called a segmenter) and the discriminator into a minimax game, the generator faces the challenge of improving its predictions to provide realistic segmentation predictions to fool the discriminator, ideally in a way that guides the generator to produce desired high-level structural qualities such as connectivity, pixel-to-pixel consistency, and smoothness.

[0070] The minimax game can be formulated by forming the following loss term for the discriminator:

[0071]

[0072] in:

[0073] x is the input image (e.g. a set of 2D red-green-blue pixel values);

[0074] y is the corresponding classification label map (i.e. real data);

[0075] s(x;θ s ) is a segmenter that maps the input image x to a dense segmentation map, parameterized by θ s ;

[0076] d(x,y;θ d ) represents the discriminator operating on the segmentation y, parameterized by θ d , conditioned on the input image x; and

[0077] The binary cross entropy is defined as

[0078] The loss function of the segmenter can be a low-level (pixel-level) loss term and high-level (adversarial) loss terms A combination of these to maintain a higher level of consistency:

[0079]

[0080] The parameter λ determines the relative importance of the adversarial loss, which can be a non-saturating reformulation of the original minimax loss term to prevent gradient vanishing.

[0081] Pixel-level cross entropy All pixels are optimized independently of each other by minimizing

[0082]

[0083] where w and h are the width and height of the image x, and c is the number of classes in the dataset (i.e. the number of classes each pixel can belong to).

[0084] Ideally, the discriminator would make decisions based purely on the structural differences between true and false segmentation predictions. However, in conventional settings, the discriminator can often completely distinguish labels from predictions based solely on the predicted values. This is because the output of the segmenter is typically a softmax vector for each pixel, which assigns a probability between zero and one to the pixel belonging to each class. In contrast, when using a typical one-hot representation, the values ​​in real data segmentations generated by humans are binary, i.e., zero or one. Therefore, the discriminator can learn to recognize the generated labels based on the fact that they contain non-binary values.

[0085] Such value-based differentiation produces unsatisfactory gradient feedback because it forces the segmenter to simply mimic a one-hot representation of the true data labels rather than identifying global structure. It can also become a never-ending training problem because achieving exact ones and zeros translates to an infinite number of logits, which always leaves small value-based gaps that can be exploited by the discriminator.

[0086] Another undesirable consequence may be that the segmenter loses the ability to express uncertainty since all of its predictions will converge to the one-hot representation.

[0087] Gambling Adversarial Network

[0088] Compared to conventional methods, the embodiments described in this invention differ from conventional "true-false" adversarial training methods. Instead of conventional discriminator networks, a novel critic network is used that learns to distinguish between correct and incorrect pixel classification predictions by considering the entire prediction map. These critic networks, also referred to herein as gaming adversarial networks, are prevented from exploiting value-based differences, so that they instead improve structural inconsistencies in segmenter predictions.

[0089] The judge network (also referred to herein as the bettor network) does not observe the true data labels classified by humans, but only observes the red-green-blue (RGB) image and the associated predictions of the segmentation network (generator neural network) through the judge. Given a limited investment budget, the bettor predicts a betting map of the size of the image (i.e., with the same number of pixels as the RGB image), where high bets (i.e., weights as described above) indicate pixels that may be incorrectly classified given the contextual prediction clues around the pixels. As disclosed herein, the betting map embodies the set of weights for the image. Since the bettor receives the entire prediction map, those structurally poorly constructed predictions, such as non-smooth, disjointed, and abnormally shaped, provide visual clues for the bettor's profitable investment.

[0090] Figures 1 to 4 Examples of improvements that can be produced using this approach are provided. Figure 1 An image showing a street scene. Although it is presented in grayscale in this article, the original image is a full-color RGB image. Figure 2 and 4 is a grayscale representation of a multi-class segmentation map. The actual segmentation map labels each image pixel with the probability that the pixel belongs to each of the 19 classes, such as "road", "person", "bicycle", "building", "pole", "traffic sign", "vegetation", etc. For the purposes of these visual representations, only the argmax of the probabilities (i.e. the maximum likelihood class for each pixel) is shown. Figure 2 is classified by the conventional U-Net convolutional neural network, while Figure 4It is classified by the gambling adversarial network embodying the present invention. Figure 3 is an image of a betting map showing the confidence values ​​assigned to each pixel by the betting adversarial network, where lighter shades represent lower confidence in the correctness of the classification and darker shades represent higher confidence. Figure 4 The predicted ratio Figure 2 The predictions in are better, for example there are fewer false classifications of pixels on the wall of the office building, around the parked cars, and in the tree on the far right of the scene.

[0091] Figure 5 Provide an overview of Gambling Adversarial Networks, where x is the input image and y is the label map for human classification. is the predicted image segmentation map from the segmenter network, and is the corresponding betting graph generated by the bettor network. The horizontal arrows from left to right through the layers represent the forward pass of the input image and segmentation map; the horizontal arrows from right to left through the layers represent the forward pass weights; and the other arrows represent the gradient flow.

[0092] Similar to regular adversarial training, the gambler and segmenter are trained by playing a minimax game. Here, the gambler is trained by maximizing the expected weighted pixel-wise cross entropy, where the weights are determined by its betting graph:

[0093]

[0094] Given an input image x and a prediction map is the budget reward (i.e. weight) that the gambler invests in position (i, j).

[0095] The segmentation network is trained by minimizing the opposite term, namely:

[0096]

[0097] The segmentation network is optimizing a combination of loss terms: a per-pixel cross entropy loss and an adversarial loss between pixels. Note that a bettor can easily maximize this loss by betting an infinite amount on all pixels. Therefore, the budget a bettor can spend should be limited. This can be accommodated by converting the betting map into a smooth probability distribution:

[0098]

[0099] where β is the smoothing factor and Represents the sigmoid (i.e. between zero and one) output of the bettor network for the pixel with index i,j. Smoothing the betting graph regularizes the model to spread its funds over multiple pixels instead of concentrating them on a single pixel.

[0100] The adversarial loss leads to two different gradient flows in the segmentation network, such as Figure 5 , where the solid diagonal arrows indicate the forward gradient flow, and the dashed arrows indicate the backward flow. The gradient flow A provides pixel-level feedback independent of other pixel predictions. At the same time, the gradient flow B reflects the consistency between pixels and structures through the gambler network.

[0101] While the adversarial loss in the Gaming Adversarial Network of the present invention has some passing similarities to the focal loss approach (see Lin et al., “Focal Loss for Dense Object Detection,” Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988, 2017), since both approaches increase the weights of harder samples that contain more useful information, the adversarial loss in the Gaming Adversarial Network of the present invention learns a weight map unlike the focal loss approach. Furthermore, in addition to providing the segmentation network with gradient flow ( Figure 5 In addition to the flow A in the method, the adversarial loss in this method also generates an additional gradient flow ( Figure 5 This is also very different from the focal loss method.

[0102] Experimental Results

[0103] In this section, we describe an experimental evaluation of an embodiment of the Gambling Adversarial Network architecture disclosed herein. We first describe the dataset and metrics, then describe different network architectures for the segmenter and gambler networks, including training details, and finally describe various experimental results.

[0104] Dataset

[0105] Experiments are conducted on two different urban road-scene semantic segmentation datasets: Cityscapes and Cambridge Driving Labelled Video Database (CamVid). However, it is expected that the same approach can be effectively applied to any segmentation dataset.

[0106] The Cityscapes dataset contains 2975 training images, 500 validation images, and 1525 test images with a resolution of 2048x1024, consisting of 19 different classes such as cars, people, and road signs. To preprocess the data, the images are scaled to 1024x512, randomly flipped, and random crops of 512x512 are taken for training. In addition, intensity jittering is performed on the RGB images.

[0107] The urban scene CamVid dataset consists of 429 training images, 101 validation images, and 171 test images with a resolution of 960x720. The same preprocessing is also applied to this dataset, except that no downscaling is performed.

[0108] measure

[0109] In addition to the average intersection over union (IoU), the structural consistency of the segmentation map is also quantified. First, the BF score is calculated, which measures whether the object contour in the prediction matches the contour of the real data. If the distance between the real data and the prediction is within the allowed distance τ (which is set to 0.75% of the image diagonal), the point is a matching point. In addition, the modified Hausdorff distance is used to quantitatively measure the correctness of the structure. The original Hausdorff distance is slightly modified to prevent it from being overwhelmed by outliers:

[0110]

[0111] where x and y are the profiles of the prediction and label from a particular class, and d(x,y) is the Euclidean distance. The score is the average score of all classes that appear in the predictions and true data.

[0112] Network Architecture

[0113] The performance is compared using two well-known baseline segmentation network architectures:

[0114] - a U-net based architecture (see Ronneberger et al., "Convolutional Networks for Biomedical Image Segmentation", International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241, Springer-Verlag, 2015) as implemented in pix2pix (see Isola et al., "Image-to-Image Translation with Conditional Adversarial Networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134, 2017), which is an encoder-decoder structure with skip connections. The encoder consists of nine downsampling blocks containing convolutions, batch normalization, and linear rectification functions. The decoder blocks are identical, except that the convolutions are replaced by transposed convolutions; and

[0115] -PSP-net, which utilizes a pyramid pooling module to capture more contextual information (see Zhao et al., "Pyramid Scene Parsing Networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890, 2017). ImageNet pre-trained ResNet-101 is used as the backbone.

[0116] For the gambler network, the same network is used for the segmentation network. When trained using the U-net based architecture, the gambler network is identical except that it contains only six downsampling blocks. For the PSP-net architecture, the architectures of the gambler and the segmenter are identical. For the baseline adversarial approach, the PatchGAN discriminator from pix2pix is ​​used.

[0117] train

[0118] To train the model, an Adam optimizer with a learning rate that decays linearly over time is used, similar to regular adversarial training. The gambler and segmenter are trained in an alternating manner, where the gambler is frozen while the segmenter is updated and vice versa. Furthermore, the best results for the gambler adversarial network are obtained without pre-training the segmentation network before starting adversarial training.

[0119] Result-Confidence Expression

[0120] As discussed above, value-based discrimination encourages the segmentation network to mimic the one-hot vectors of real data and results in a loss of the ability to express uncertainty. To verify that embodiments of the present invention mitigate these problems, the mean and standard deviation of the maximum similar probability value in each softmax vector for each pixel on the validation set were tracked over different training cycles. The results are shown in Figure 2. Figure 6 Depicted.

[0121] Figure 6 Showing the average maximum likelihood (average confidence) over time on the Cityscapes validation set using four different models:

[0122] - Cross Entropy (“CE”),

[0123] - Standard cross entropy with adversarial training (“CE+adv”) (see Luke et al., “Semantic Segmentation using Adversarial Networks”, ARXIV:1611.08408, 2016),

[0124] - Embedding Loss Generative Adversarial Networks (“EL-GAN”) (see Galfolian et al., “EL-GAN: Embedding Loss Driven Generative Adversarial Networks for Lane Detection”, Proceedings of the European Conference on Computer Vision, 2018), and

[0125] - The gaming network disclosed herein ("Gaming Network").

[0126] The solid central curves 60, 61, 62, 63 represent the average values ​​using EL-GAN 60, CE+adv 61, betting network 62 and CE 63, respectively. The shaded areas 60a, 61a, 62a, 63a around the average values ​​represent the corresponding standard deviations.

[0127] In both standard CE+adv adversarial training and EL-GAN for distinguishing true and false predictions, Figure 6 It can be seen that predictions 60 and 61 converge towards unity with almost no standard deviation. For the Gambling Adversarial Network, the uncertainty of prediction 62 is well preserved.

[0128] The following table shows the average maximum value in each softmax vector, averaged over the past ten epochs. This confirms that the Gambling Adversarial Network maintains uncertainty similar to the cross-entropy model, while known adversarial methods try to converge to a one-hot vector:

[0129]

[0130] The U-net based City Landscape architecture was used in these experiments, but similar results were observed using other segmentation networks and on other datasets.

[0131] Results - U-net based segmenter

[0132] The baseline is compared to Gaming Adversarial Network on the validation set of Cityscapes based on the U-net architecture. The results in the table below show that Gaming Adversarial Network performs better than CE, CE+adv, and EL-GAN not only on pixel-level metrics (IoU) but also on structural metrics. The “focal loss” method adopts the technique described in Lin et al., “Focal Loss for Dense Object Detection”, Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988, 2017:

[0133]

[0134] In the table below, the same experiment provides the IoU for each class. The Gambling Adversarial Network performs better on most classes. It performs especially well on classes with finer structures, such as traffic lights and people:

[0135]

[0136]

[0137] Figure 7 Some qualitative results are shown. The input image (top left) is represented here in grayscale, but is processed as an RGB image. The real data (top middle) is a human-labeled classification. Various segmentation outputs are generated using the methods already described above. The betting adversarial network correctly resolves some artifacts, such as the pattern on the side of the truck in the left foreground visible in the CE segmentation. In addition, the boundaries of pedestrians on hard surfaces (sidewalks) are more precise and closer to the boundaries of the real data labeled data. The example betting graph represents the assignment of the bettor's confidence values ​​(weights) on the image given the prediction of the baseline model trained using cross entropy combined with the RGB image. Note that the bettor bets on the poorly shaped building and also reacts to the artifacts in the truck on the left.

[0138] Results - PSP-net splitter

[0139] Experiments were performed using the PSP-net segmenter on the CamVid and Cityscapes datasets.

[0140] The following table shows the results from the Cityscapes validation set:

[0141]

[0142] Again, Gambling Adversarial Networks outperform existing methods on both the structure-based score and the average IoU.

[0143] Figures 8 to 11 Some qualitative results are presented. Figure 8 Display the input RGB image (here in grayscale). Fig. 9 Show real data labels. Fig.10 shows the cross entropy output, while Fig.11 Showing the Gaming Network segmentation output. For example, it can be seen that the Gaming Adversarial Network provides more details about the traffic lights. In addition, the structure of the hard road surface (sidewalk) shows significant improvement compared to the predictions from the model trained with the standard segmentation loss.

[0144] The quantitative results of the Camvid dataset are shown in the following table:

[0145]

[0146] Here, Gambling Adversarial Network achieves the highest scores on average IoU and BF score, while standard adversarial training (CE+adv) also achieves reasonable performance. Gambling Adversarial Network significantly outperforms other methods on BF score, while Hausdorff distance outperforms other methods in standard adversarial training, albeit by a smaller margin.

[0147] analyze

[0148] In a betting adversarial network, the critic network is trained to learn to spot predictions that are likely to be incorrect. Figure 6As shown, since all softmax vectors converge to a one-hot vector, true and false discrimination causes undesirable gradient feedback. This behavior is caused by the value-based discrimination of the adversarial network. In addition, replacing the conventional adversarial task with the correct / incorrect discrimination of a single pixel solves several problems. First, the application of adversarial training to semantic segmentation is to improve high-level structures. However, conventional value-based discriminators not only provide feedback based on the visual difference between predictions and labels, but also provide undesirable value-based feedback. In addition, the constraint for updating weights in the network is that the output must be a one-hot vector, which makes training unnecessarily complicated. Finally, the value-based discriminator prevents the network from properly exposing uncertainty. Both structured predictions and expressing uncertainty are of great value for semantic segmentation in, for example, autonomous driving and medical imaging. However, changing the adversarial task to distinguish between correct and incorrect predictions solves the above problems. The segmentation network is not forced to imitate a one-hot vector, allowing it to retain uncertainty in the prediction and simplifying training. Although gamblers can sometimes still take advantage of predicted values ​​by betting on pixels that the segmenter is uncertain about, these experimental results demonstrate that structure-based metrics can achieve improvements compared to existing adversarial methods. In particular, experimental results on two road scene semantic segmentation tasks show that the proposed method is not only able to express uncertainty, but also improves both pixel-based and structure-based metrics and further stabilizes the training process.

[0149] Variant Embodiment

[0150] To completely eliminate the value-based differences, in an alternative set of embodiments, the bettor (judge neural network) is trained on the argmax of the segmenter (generator neural network). The only way to distinguish the predicted argmax from the real data is to analyze the visual differences. Since argmax is not differentiable, the segmenter is updated with an approximation of argmax.

[0151] In contrast to the betting network disclosed above, the bettor here receives as input (i) the argmax of the one-hot encoding combined with the corresponding RGB image, or (ii) the real data of the one-hot encoding combined with the corresponding RGB image. Because there is no longer a value-based difference between the real data and the prediction, the real data can also be used for training.

[0152] The output of the bettor is an image of the size of the input, with each pixel assigned a probability of being correct or incorrect. Since the argmax operation results in values ​​of 0 or 1 in the input, the prediction is binary, i.e. either "correct" or "incorrect".

[0153] Therefore, the bettor’s adversarial loss function is converted to a binary cross entropy loss:

[0154]

[0155] Where s is the segmentation network, s arg Returns an estimate of argmax, where 0 indicates the pixel is correct and z i,j Indicates whether the pixel (i, j) is correctly predicted by the segmenter. As before, given the input x and the predicted segmentation map is the reward (i.e. weight) that the bettor invests in position (i,j) (similarly for the true data split y). Note that for the true data, all predictions are correct. The bettor's training data is severely imbalanced, as the true data does not contain any errors and the accuracy of the predictions also increases with training time. To mitigate this, the BCE loss is weighted and the true data is fed to the bettor less frequently than the predictions.

[0156] Meanwhile, the segmenter is pushing its predictions closer to the bettor’s idea of ​​what is correct:

[0157]

[0158] Where 0 indicates a good prediction and 1 indicates an incorrect prediction. Essentially, the segmenter requires bettors to provide feedback to push its predictions closer to perfection.

[0159] In order to pass the gradient from the bettor to the segmenter, the argmax function needs to be estimated. It is not possible to perform an argmax operation on the output of the segmenter while the segmenter is being trained, as it is not differentiable. However, the bettor is trained on the argmax input. In order to minimize the shift in the data distribution of the bettor, an operation close to one-hot prediction can be performed, such as Fig.12 As described.

[0160] Fig.12 Shows the approximation of argmax. Forward pass is indicated by rightward curving arrows, and backward pass is indicated by leftward curving arrows. The RGB input image is x. The segmentation network s logits Output the logits in this graph. The argmax operation returns the argmax of the one-hot encoding. τ is a hyperparameter that affects the degree of prediction sharpness; is a differentiable argmax approximation; b is a real data betting graph; and It is a prediction betting chart.

[0161] A softmax operation is performed immediately after the output of the final convolutional layer (logitization) of the segmentation network. The argmax function returns the one-hot encoding of the argmax, rather than the variable argument of the maximum. The hyperparameter τ controls the steepness of the softmax function. The argmax approximation is input to the bettor network along with the RGB input image x. The bettor network outputs a betting map, (i.e. a set of weights). After that, the last step in the progression is the calculation of the loss.

[0162] In the backward pass, there is no backpropagation on the argmax operation, simply because it is impossible. Essentially, the whole operation adds an extra amount to the maximum value among all softmax vectors, moving it away from other predictions. One advantage over using a warm softmax is that the gradients do not explode or vanish in the backward pass. Instead, the steepness of the softmax, and therefore the magnitude of the gradient, remains unchanged. However, in the backward pass, the inputs appear shifted. For example, if the maximum value of the softmax is initially 0.6, and this value is scaled up to 0.9 to make it closer to the one-hot representation, then our method obtains the gradient for the latter and further backpropagates it on the former.

[0163] Therefore, the complete objective of the segmenter is again a combination of the adversarial loss and the cross entropy loss:

[0164]

[0165] Fig.13 An exemplary computer processing system is shown on which embodiments may be implemented. Computer 130 includes processor 131 (e.g., Intel TM The computer 130 may include a processor 131 that is arranged to execute software stored in the memory 132. The processor 131 also uses the memory 132 to read and write data, such as input data, intermediate calculation results, and output data. The software may control the processor 131 to implement any of the methods disclosed herein. The computer 130 may have input / output peripherals, such as for receiving training data and / or for deriving data encoding the trained network.

[0166] The network may be trained centrally and the trained network replicated and installed on other devices such as an automobile guidance or warning system, or the training may be performed in the field, such as as continuous learning within the control system of an automated vehicle.

[0167] It will be appreciated by those skilled in the art that the present invention has been illustrated by describing one or more specific embodiments thereof, but is not limited to these embodiments; many variations and modifications are possible within the scope of the appended claims.

Claims

1. A method for training a generative adversarial network for performing image semantic segmentation, wherein the generative adversarial network comprises: Generator neural network; and The judge neural network, The method comprises: using the generator neural network to generate predicted image segmentation maps from input images, wherein each predicted image segmentation map comprises a classification prediction for each of a plurality of pixels of a corresponding input image; providing one or more predicted image segmentation maps generated by the generator neural network to the judge neural network; training the critic neural network to determine a betting map comprising weights for corresponding pixels of a predicted image segmentation map generated by the generator neural network, wherein in said training of the critic neural network, said weight for each pixel is used to weight a pixel-wise cross entropy term in an objective function of the critic neural network; and The generator neural network is trained using the weights of the betting map determined by the judge neural network.

2. A method according to any preceding claim, comprising performing a minimax training procedure to train the critic neural network and the generator neural network.

3. The method according to claim 1 or 2, comprising evaluating the objective function of the critic neural network by calculating the sum of the products of the weight of each pixel and the cross entropy loss term of the corresponding pixel over all pixels of the predicted image segmentation map.

4. The method of claim 1 or 2, comprising training the generator neural network using a generator objective function that decreases as the judge objective function increases.

5. The method according to claim 1 or 2, wherein the sum of all said weights on the predicted image segmentation map is limited to a threshold amount.

6. The method of claim 1 or 2, comprising determining the weights by applying a sigmoid function to a set of intermediate weight values ​​generated within the judge neural network.

7. The method of claim 1 or 2, wherein determining the weights comprises applying a spatial smoothing function to a set of intermediate weight values ​​generated within the judge neural network.

8. The method according to claim 1 or 2, wherein the critic neural network only receives the input image and the image segmentation map as forward input.

9. The method of claim 1 or 2, wherein the input image is a photographic image of a street scene, and wherein the method comprises training the generative adversarial network to classify objects in the street scene.

10. A computer processing system implementing a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises: Generator neural network; Judger neural network; and Training logic, And wherein the training logic is configured to: operating the generator neural network to generate predicted image segmentation maps from the input images, wherein each predicted image segmentation map comprises a classification prediction for each of a plurality of pixels of the corresponding input image; providing one or more predicted image segmentation maps generated by the generator neural network to the judge neural network; training the critic neural network to determine a betting map comprising weights for corresponding pixels of a predicted image segmentation map generated by the generator neural network, wherein in said training of the critic neural network, said weight for each pixel is used to weight a pixel-wise cross entropy term in an objective function of the critic neural network; and The generator neural network is trained using the weights of the betting map determined by the judge neural network.

11. The computer processing system of claim 10, wherein the training logic is configured to evaluate the objective function of the critic neural network by computing, for a predicted image segmentation map, a sum of products of the weight for each pixel and a cross entropy loss term for the corresponding pixel over all pixels of the predicted image segmentation map.

12. The computer processing system of claim 10 or 11, wherein the training logic is configured to train the generator neural network using a generator objective function that decreases as the judge objective function increases.

13. The computer processing system of claim 10 or 11, wherein the training logic is configured to determine the weights of the betting map such that the sum of all of the weights over a predicted image segmentation map is limited to a threshold amount.

14. The computer processing system of claim 10 or 11, wherein the training logic is configured to provide only an input image and a predicted image segmentation map as forward input to the critic neural network.

15. Computer software comprising instructions that, when executed on a computer processing system, cause the computer processing system to train a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises: Generator neural network; and The judge neural network, and wherein the instructions cause the computer processing system to: operating the generator neural network to generate predicted image segmentation maps from the input images, wherein each predicted image segmentation map comprises a classification prediction for each of a plurality of pixels of the corresponding input image; providing one or more predicted image segmentation maps generated by the generator neural network to the judge neural network; training the critic neural network to determine a betting map comprising weights for corresponding pixels of a predicted image segmentation map generated by the generator neural network, wherein in said training of the critic neural network, said weight for each pixel is used to weight a pixel-wise cross entropy term in an objective function of the critic neural network; and The generator neural network is trained using the weights of the betting map determined by the judge neural network.

16. Computer software according to claim 15, wherein the instructions cause the computer processing system to determine the weights of the betting map such that the sum of all of the weights over a predicted image segmentation map is limited to a threshold amount.

Images