Generative Adversarial Networks for Image Segmentation

Abstract

A method of training a generative adversarial network for performing semantic segmentation of an image is provided. The generative adversarial network includes a generator neural network and a discriminator neural network. The method includes: providing an image as an input to the generator neural network; receiving a predicted segmentation map for the image from the generator neural network; providing i) the image, ii) the predicted segmentation map, and iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network; in response to the training input; determining a set of one or more outputs from the discriminator neural network; and training the generator neural network using a loss function, the loss function being a function of the set of outputs from the discriminator neural network.

Description

Background Art

[0001] The present invention relates to neural networks for image segmentation, and in particular to generative adversarial networks.

[0002] Certain convolutional neural networks (CNNs) have been successfully applied to various computer vision problems by treating these problems as image segmentation problems. Examples include road scene understanding for autonomous driving and interpreting medical imaging. For such applications, the network is usually trained with multiple classes of per-pixel labels that together form an image-sized segmentation map. The output of this network is again an image-sized map that represents per-pixel class probabilities.

[0003] However, additional post-processing steps may be needed afterwards, because the output of the semantic segmentation network does not necessarily maintain quality. The output segmentation map is probabilistic and has a different distribution than the corresponding labels. The fundamental reason behind this is the way the training loss is formulated (e.g., per-pixel cross entropy), so that each output pixel in the segmentation map is considered independent of all other pixels, i.e., no explicit pixel-to-pixel consistency is enforced.

[0004] Examples of post-processing steps include applying a conditional random field (CRF), applying a second separately trained network or a non-learning problem-specific algorithm. The disadvantages of such methods are that they require effort to construct, may have many hyperparameters, are problem-specific, and may still not capture the ultimate goal. For example, CRFs only capture local consistency, or impose high computational costs to ensure long-range dependencies, and require separate training.

[0005] Another approach to ensure certain prediction quality is to add an additional loss term that represents the degree of quality preservation. However, it is often challenging to come up with an effective and distinguishable loss term for each target quality. A potential solution to address the lack of quality preservation in the semantic segmentation problem is to use a generative adversarial network (GAN) to 'learn' the required loss function. GANs work by training two networks in an alternating manner in a minimax game: the generator is trained to produce results, while the discriminator is trained to distinguish the generated (predicted) segmentation map data ('fake') from the ground truth labels ('true').

[0006] GANs have been applied to semantic segmentation problems to try to solve the above problem of per-pixel loss. During training, the generator produces semantic segmentation maps, while the discriminator alternately observes the ground truth labels and the predicted segmentation maps.

[0007] Xue et al., in their paper “SegAN: Adversarial Network with Muti-scale L1 Loss for Medical Image Segmentation” published in ArXiv e-prints (June 2017), disclose a generative adversarial network (GAN) for brain tumor segmentation. It uses a common multi-scale L1 loss function to train both the generator and the discriminator, rather than having separate losses for the generator network and the discriminator network.

[0008] However, the applicant has recognized that GANs (such as the GAN disclosed in the SegAN paper) may be effective for segmenting large spherical structures (such as brain tumors), but are not always effective for other types of image data. In particular, GANs have difficulty accurately identifying thin structures, such as road divider markings.

[0009] The present invention seeks to provide a novel method for training generative adversarial networks that may lead to better image segmentation performance, at least for certain types of inputs. Summary of the invention

[0010] From a first aspect, the present invention provides a method for training a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises:

[0011] Generator neural network; and

[0012] Discriminator neural network,

[0013] The method comprises:

[0014] providing an image as input to the generator neural network;

[0015] receiving a predicted segmentation map for the image from the generator neural network;

[0016] providing i) the image, ii) the predicted segmentation map, and iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network;

[0017] determining a set of one or more outputs from the discriminator neural network in response to the training input; and

[0018] The generator neural network is trained using a loss function that is a function of the set of outputs from the discriminator neural network.

[0019] From a second aspect, the present invention provides a computer processing system implementing a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises:

[0020] Generator neural network;

[0021] Discriminator neural network; and

[0022] Training logic,

[0023] and wherein the training logic is configured to:

[0024] providing an image as input to the generator neural network;

[0025] receiving a predicted segmentation map for the image from the generator neural network;

[0026] providing i) the image, ii) the predicted segmentation map, and iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network;

[0027] determining a set of one or more outputs from the discriminator neural network in response to the training input; and

[0028] The generator neural network is trained using a loss function that is a function of the set of outputs from the discriminator neural network.

[0029] From a third aspect, the present invention provides computer software comprising instructions which, when executed on a computer processing system, cause the computer processing system to train a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises:

[0030] Generator neural network; and

[0031] Discriminator neural network,

[0032] and wherein the instructions cause the computer processing system to:

[0033] providing an image as input to the generator neural network;

[0034] receiving a predicted segmentation map for the image from the generator neural network;

[0035] providing (i) the image, (ii) the predicted segmentation map, and (iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network;

[0036] determining a set of one or more outputs from the discriminator neural network in response to the training input; and

[0037] The generator neural network is trained using a loss function that is a function of the set of outputs from the discriminator neural network.

[0038] The image may be a photographic image of a road including one or more lane markings. The method may include training the generative adversarial network to recognize lane markings in the image.

[0039] The computer processing system may include a memory that stores the image and stores the ground-truth label data corresponding to the image, wherein the image is a photographic image of a road including one or more lane markings, and wherein the ground-truth label data includes data representative of the one or more lane markings.

[0040] The computer software may further include instructions to implement a generator neural network and / or to implement a discriminator neural network. The computer software may be stored on a non-transitory storage medium such as a magnetic or solid-state memory, or may be carried on a transient signal such as an electrical or electromagnetic signal.

[0041] It will therefore be seen that in accordance with the present invention, the generator neural network is trained by providing three separate inputs to the discriminator: the image, the predicted segmentation map for the image generator, and the ground truth label for the image.

[0042] This is in contrast to existing methods, in which the discriminator receives as input vector either an image and its corresponding segmentation map or an image and its corresponding ground truth label, rather than simultaneously receiving as input vector an image, its corresponding segmentation map, and its corresponding ground truth label. In embodiments of the present invention, by feeding the discriminator an input vector including predictions and labels for common images, the discriminator is likely to obtain more useful feedback to guide the training of the adversarial segmentation network in the direction of more realistic labels.

[0043] This approach also contrasts with the SegAN network mentioned above, in which the generator is trained by providing the discriminator with the pixel-by-pixel product of the image and the predicted segmentation map, and the generator does not receive the full input image separate from the predicted segmentation map. Applicants have appreciated that providing the full image, segmentation map, and label data as separate inputs to the discriminator gives the discriminator greater flexibility in deciding how to fuse the input image and predicted representations, rather than obtaining a fixed multi-way mixture. This enables networks embodying the present invention to provide better performance on at least some data types - particularly (but not exclusively) when semantically segmenting images containing thin structures such as road lane markings.

[0044] The set of one or more outputs from the discriminator neural network is not necessarily the final output of the discriminator network (e.g., a probability between 0 and 1). Instead, it may include one or more feature map (i.e., embedding) outputs obtained from one or more predetermined layers within the discriminator neural network.

[0045] The loss function may include an embedding term representing the difference between: i) the embedding (i.e., feature map) at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network; and ii) the embedding at the predetermined layer of the discriminator neural network when the ground truth label data is input to the discriminator neural network. The difference may be a distance according to an appropriate norm. While it may be an L1 distance, in a preferred set of embodiments, the difference is an L2 distance. Applicants have determined that, at least for some types of input images, minimizing the L2 loss may give better results than the L1 loss because the L2 loss generally prefers smaller average differences, which makes it well suited for matching embeddings.

[0046] The predetermined layer may be after (e.g., immediately after) the first dense block, which contains one or more shared convolutional layers with both the image and the predicted segmentation map (or ground truth label data). It may be after (e.g., immediately after) the final dense block before the classifier of the network. Alternatively, it may be a layer between these two layers.

[0047] The generator neural network can be trained to minimize the loss function. The training can include applying a gradient descent method to the network.

[0048] The loss function may include a fitness term in addition to the embedding term. The fitness term may be a pixel-level loss term. It may represent a relatively low-level fitness of the prediction / label. The embedding loss term may represent a relatively high-level consistency. The loss function may include a weighting parameter λ for weighting the embedding term relative to the fitness term. The system may include an input for receiving a value for the weighting parameter, for example from a user. This advantageously allows embodiments of the present invention to be configured for a desired pixel-level loss importance.

[0049] The discriminator neural network may also be trained. It may be trained to minimize the loss on its distinction between the predicted segmentation map and the ground truth label data.

[0050] The training logic may be configured to train the generator alternately with the discriminator.

[0051] The generative adversarial network may be trained on a plurality of images, which may include one hundred, one thousand, ten thousand, or more images.

[0052] In some embodiments, the image may be a photographic image from a camera. It may be an image of a road (e.g., a multi-lane road or a carriageway), which may contain one or more lane markings. The ground truth label data may include data representing one or more lane markings in the image (e.g., including coordinates, vectors, and / or polylines). The generative adversarial network may be trained to recognize linear structures in the image. It may be trained to recognize road markings, such as lane markings.

[0053] The generative adversarial network may be configured to have a training mode and a run mode. Training may be performed during a training phase. After the training phase, the network may be configured to receive an input image and segment the input image. It may be configured to output a predicted segmentation map for the image.

[0054] The computer processing system may include an input for receiving image data from a camera. It may be an onboard computer processing system. It may be configured to output segmentation data to, for example, an autonomous driving system.

[0055] The generator neural network and / or the discriminator neural network may include any number of convolutional layers, dense blocks, and other conventional layers. The generator neural network and / or the discriminator neural network and / or the training logic may include software instructions for a processor, or may include dedicated hardware logic, or may include a combination of both. The computer processing system may include one or more of the following: a CPU, a DSP, a GPU, an FPGA, an ASIC, volatile memory, non-volatile memory, input, output, display, network connection, power supply, radio, clock, and any other suitable components. It may be configured to store or display or output predicted segmentation maps or other segmentation data.

[0056] Some embodiments may achieve segmentation predictions that are sufficiently similar to the training labels such that no additional problem-specific loss terms and / or post-processing steps are required.

[0057] Where appropriate, features of any aspect or embodiment described herein may be applied to any other aspect or embodiment described herein. Where reference is made to different embodiments or groups of embodiments, it should be understood that these features are not necessarily distinct but may overlap. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Certain preferred embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0059] Figure 1 This is a schematic diagram of a conventional generative adversarial network;

[0060] Figure 2 is a schematic diagram of an adversarial network embodying the present invention;

[0061] Figure 3is a schematic diagram of a generative adversarial network trained for lane marking segmentation;

[0062] Figure 4 is an example ground truth labeling for lane marking segmentation overlaid on the corresponding photographic image;

[0063] Figure 5 are the corresponding raw predictions from a conventional segmentation network overlaid on a photographic image;

[0064] Figure 6 are the corresponding raw predictions from a generative adversarial network embodying the invention overlaid on a photographic image;

[0065] Figure 7 is a table having six rows corresponding to six different example photographic images, and columns showing ground truth labels, outputs from a generative adversarial network embodying the present invention, and outputs from a conventional segmentation network;

[0066] Figure 8 is a plot of the validation f-score versus training iterations for different adversarial loss terms;

[0067] Fig. 9 is a table having three rows corresponding to three different example photographic images, and columns showing ground truth labels, and outputs from a generative adversarial network embodying the present invention after different corresponding dense blocks within the network; and

[0068] Fig.10 is a table showing: first, a close-up portion of an input image; second, the segmentation output of a conventional segmentation network; and third, the overlapping segmentation outputs of three different generative adversarial networks embodying the present invention. DETAILED DESCRIPTION

[0069] Described below is a set of neural networks that embody the present invention. This new type of artificial neural network will be referred to herein as "El-GAN" - an abbreviation for "Embedding Loss Generative Adversarial Network". It uses adversarial training with embedding loss for semantic segmentation. The method can be applied to many different segmentation problems. However, it has been extensively tested on lane segmentation within images acquired by cameras on cars, and has been found to be particularly effective at this task. It will be demonstrated with reference to this application area, but it will be understood that it can also be applied to other image segmentation problems.

[0070] As background, we first describe typical current approaches to semantic segmentation with generative adversarial networks (GANs).

[0071] Adversarial training can be used to ensure a higher level of similar label qualities, such as smoothness, maintaining neighborhood consistency, etc. This is often done by using a discriminator network that learns a loss function for these desired properties over time, rather than formulating these properties explicitly.

[0072] A typical approach to benefit from adversarial training for semantic segmentation involves constructing a loss function for the segmentation network (generator) that consists of two terms: one term involves the low-level pixel-wise prediction / label fitness And another (adversarial) loss term is used to maintain a higher level of consistency quality It is conditioned on the input image:

[0073]

[0074] where x and y are the input image and the corresponding label map (i.e., ground truth label data), respectively, and θ gen and θ disc is the set of parameters for the generator and discriminator networks, f(x;θ) represents the transformation of the input image x imposed by the network parameterized by θ, and λ indicates the relative importance of the adversarial loss term.

[0075] Figure 1 A typical training setup for a traditional GAN ​​is illustrated. It shows the generator G and discriminator D trained using the loss function of equation (1) above.

[0076] Loss Item It is usually formulated using pixel-wise classification cross entropy loss,

[0077]

[0078] in

[0079]

[0080] w and h are the width and height of the image, and c is the number of classes to which each pixel can belong.

[0081] Adversarial Loss Term Indicates how successful the discriminator is in rejecting the (fake) dense prediction maps produced by the generator, and is usually formulated as a binary cross-entropy loss between zero and the discriminator’s binary predictions for the generated prediction maps:

[0082]

[0083] in

[0084]

[0085] While the generator is trained to minimize its adversarial loss term, the discriminator tries to maximize it by minimizing its loss, which is defined as:

[0086]

[0087] By first training the discriminator, and then training the generator, and then further training the discriminator, and so on in an alternating pattern, the discriminator learns the difference between the label and the predicted distribution, while the generator tries to change the quality of its predictions, similar to the predictions of the labels, so that the two distributions are indistinguishable. In practice, it is often observed that training adversarial networks tends to be more tricky and unstable than training normal networks. This can be attributed to the mutual training of the two networks involved in the minimax game, so that each affects the training of the other. The discriminator gives feedback to the generator based on the plausibility of the generator images.

[0088] There are two important problems with this traditional adversarial training and its application to semantic segmentation, which are mitigated in embodiments of the present invention:

[0089] 1. The idea of ​​how plausible and fake these prediction maps are comes from the discriminator’s imagination of these concepts and how its weights encode these qualities. This encoding may be far from perfect, leading to gradients in directions that may not improve the generator.

[0090] 2. The adversarial loss computation does not exploit valuable information about image / label pairs, which is commonly available in many supervised semantic segmentation tasks.

[0091] Adversarial Training with Embedding Loss

[0092] In contrast to conventional approaches, the current El-GAN embodiment exploits image-label pairing to base plausibility / fakeness decisions not only on the discriminator's understanding of these concepts, but also on true plausible label maps.

[0093] One way to leverage this idea is to use a discriminator to incorporate the prediction / label map into a higher-level description and define the adversarial loss as their difference on the embedding space:

[0094]

[0095] in

[0096]

[0097] Set equal to the embedding loss

[0098]

[0099] Defined as the L2 distance on the embedding:

[0100]

[0101] in

[0102] f e (p; x, θ)

[0103] represents the embedding extracted from a given layer in a network parameterized by θ, given p and x as its input. “Embedding” in this context refers to the set of feature maps obtained at a layer in the discriminator.

[0104] Figure 2 This illustrates the training of the generator G and the discriminator D. Figure 1 In contrast to the conventional approach of , the discriminator D not only receives the image x and its corresponding generated segmentation map f(x; θ) from the generator G, but is also trained on the ground truth labeled data y corresponding to the image x. For clarity, the discriminator D is Figure 2 In Figure 2, two separate discriminator examples with shared weights are shown to highlight the different sets of inputs and outputs: [image, fake label] and [image, true label]. Both examples share the same weights and can therefore be conceptually considered a single discriminator network D.

[0105] Therefore, the El-GAN architecture computes adversarial losses and corresponding gradients based on the differences in high-level descriptions (embeddings) of labels and predictions. While the discriminator learns to minimize the loss on its distinction between true and fake distributions and potentially learns a set of discriminative embeddings, the generator tries to minimize such embedding differences.

[0106] In some embodiments, Equation 2 for the discriminator update may optionally be rewritten as:

[0107]

[0108] However, in empirical studies, the applicants have found that using the cross-entropy loss to update the discriminator parameters gives better results in at least some cases.

[0109] Experimental setup

[0110] We will now describe the evaluation of the El-GAN approach, followed by the details of the network architecture and training method. The words "we", "us" and "our" hereinafter refer to the applicant.

[0111] Experimental Setup - Evaluation Dataset and Metrics

[0112] The El-GAN method has been evaluated in the application domain of autonomous driving. Specifically, the El-GAN method is evaluated using data from lane marking detection data, which is available from TuSimple. TMIt is publicly available via its website http: / / benchmark.tusimple.ai / # / t / 1. However, it will be appreciated that the method is general and can also be applied to other semantic segmentation problems.

[0113] One of the motivations for these embodiments is to be able to produce predictions that are as similar as possible to the ground truth labels. This is particularly true for TuSimple with a thin structure. TM Lane marking detection datasets are particularly useful, reducing the need for complex post-processing. TM The lane marking detection dataset1 consists of 3,626 annotated 1280×720 frontal road images of US highways in the San Diego area divided into four sequences, and a similar set of 2782 test images. The annotations are given in the form of polylines of lane markings: markings for the ego lane and lanes to the left and right of the car. The polylines are given at fixed height intervals of every 20 pixels. To generate labels for semantic segmentation, we convert these polylines into segmentation maps by discretizing the lines using a Gaussian with smooth interpolation and a sigma of 1 pixel width.

[0114] Figure 4 An example of this labeling is shown. For clarity, the labeling is shown here as overlaid on the original image, but it will be understood that the labeling data is distinct from the image data.

[0115] The result evaluation dataset is in the same format as the label (i.e., multiple polylines). For our evaluation, we use the official indicators defined in the challenge, i.e., accuracy, false positive rate, and false negative rate. We report the results of the official test set as well as the validation set, which is one of the labeled sequences with 409 images ('0601'). We note that the performance on this validation set may not be fully represented due to its small size. Different validation sequences also have their disadvantages, because the other three are much larger and will significantly reduce the size of the already small dataset. Since our network still outputs a segmentation map instead of the required polylines, we do apply post-processing, but keep it as simple as possible: after binarization, we transform each connected component into a separate polyline by obtaining the average x index of the non-zero value sequence at each y index. We call this method 'basic'. We also evaluate the "basic++" version, which also splits the connected components if the "basic++" version detects multiple non-zero value sequences at a sampled position.

[0116] Experimental setup - network architecture and training

[0117] In this section, we discuss the networks and training settings used for our experiments.

[0118] Figure 3Shows an overview of the high-level network architecture with example data. This shows different loss terms used for generator training or discriminator training or both.

[0119] For the generator, we use a fully convolutional U-Net style network with downward and upward paths and skip connections. Specifically, we use the Tiramisu DenseNet architecture for lane marking detection, configured with seven upward / downward stages for a total of sixty-four 3×3 convolutional layers. See Jégou et al., "The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation," CVPRW (July 2017), 1175–1183.

[0120] For the discriminator, we use a DenseNet architecture with seven blocks and a total of thirty-two 3×3 convolutional layers (see Huang et al., “Densely Connected Convolutional Networks”), followed by a fully convolutional patch GAN classifier (see Li and Wand, “Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks”, ECCV (2016), 702–716).

[0121] For the first two dense blocks, we use a two-head network to process the input image from labels or predictions separately, and then we concatenate the feature maps. We take the embedding after the final convolutional layer. However, other options are possible as described in the ablation study below.

[0122] We first pre-trained the generator model until convergence, and we also use the generator model as our baseline non-GAN model in the results section below. We then pre-trained the discriminator for 10,000 iterations using a batch size of eight, after which the generator and discriminator were trained for between 300 and 200 iterations respectively. The generator was trained with the Adam optimizer (see Kingma and Ba, Adam: A method for Stochastic Optimization, ICLR (2014)), and the discriminator training was found to be more stable with SGD. We trained the discriminator using a regular cross-entropy loss (Eq. 2), while we trained the generator with an adversarial embedding loss of λ=1 (Eqs. 3 and 4). We did not perform any data augmentation and did not pre-train the model on other data.

[0123] Details of the specific network architecture setup used for our experiments are as follows.

[0124] The generator characteristics are:

[0125] Architecture = Tiramisu DenseNet;

[0126] Number of dense blocks in down / up sampling paths = 7;

[0127] Number of 3×3 conv layers in each dense block = [1, 2, 3, 4, 6, 8, 8];

[0128] Growth rate = 18;

[0129] Nonlinearity = ReLU;

[0130] Initialization = He [see He et al., “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, ICCV (2015), 1026–1034]; and

[0131] Discard rate = 0.1.

[0132] The discriminator features are:

[0133] Architecture = Two-end DenseNet;

[0134] Join two heads = concatenate after the second dense block;

[0135] Number of dense blocks = 7;

[0136] Number of 3×3 conv layers in each dense block = [1, 2, 3, 4, 6, 8, 8];

[0137] Growth rate = 8;

[0138] Nonlinearity = ELU [see Clevert et al., “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELU)”, ICLR (2015)];

[0139] No discarding; and

[0140] Embeddings taken from layer = after the seventh dense block.

[0141] The general training hyperparameters are:

[0142] Number of iterations = 150K;

[0143] batch size = 8; and

[0144] Training schedule = (300: disc, 200: gen).

[0145] The generator training hyperparameters are:

[0146] optimizer = (Adam, Momentum:0.9);

[0147] Learning rate = (exponential, initial value: 5e-4, decay capacity: 0.99, decay rate: 200)

[0148] L2 regularization ratio = 1e-4; and

[0149] Pre-training = 100K iterations.

[0150] The discriminator training hyperparameters are:

[0151] optimizer = vanilla SGD;

[0152] learning rate = (exponential, initial value: 1e-5, decay capacity: 0.99, decay rate: 800);

[0153] Pre-training = 10K iterations;

[0154] L2 regularization ratio = 1e-5; and

[0155] Adversarial loss λ=1.

[0156] result

[0157] In this section, we report TuSimple using the experimental setup described above. TM Results on the dataset. Additionally, we perform three ablation studies: evaluating training stability, exploring options for the training loss, and varying the choice of embedding loss layer.

[0158] Results - TuSimple TM Lane marking challenge

[0159] In this section, we report TuSimple TM Results on the lane marking detection challenge and compare them with our baseline and the state-of-the-art.

[0160] As already mentioned, Figure 4 Showing example ground truth labels. Figure 5 Shown are the corresponding raw predictions for the same image by a conventional segmentation network. Figure 6 The corresponding original predictions through El-GAN are shown. Figure 5 Compared with the conventional output, Figure 6 The predictions in Figure 4 In all three cases, the data are overlaid on the input image for convenience.

[0161] We first evaluate El-GAN and our baseline on the validation set using two post-processing methods. The results in Table 1 show that the basic post-processing method does not work for the baseline model, while the improved basic++ method performs much better. El-GAN still performs better than the baseline, especially when using the most basic post-processing method.

[0162] Table 1. Results on the TuSimple lane marking validation set

[0163]

[0164]

[0165] Figure 7 Some results on the validation set are shown. They compare the two methods in terms of raw prediction maps and post-processing results using the basic++ method. Compared to the conventional baseline method, it can be seen that El-GAN produces considerably finer and more label-like outputs, and has less noise, which makes post-processing generally easier.

[0166] In addition, we train El-GAN and baselines on the entire labeled dataset and TMEvaluation is performed on the official test set of the challenge using basic++ post-processing. Table 2 shows the results, which contains all the methods in the top six and their ranking on the leaderboard as of March 14, 2018. El-GAN ranks 4th based on accuracy, less than half a percentage point away from the best, and achieves the lowest false positive rate. Compared to our baseline (without the discriminator), we show improvements consistent with our experiments in Table 1 on the validation set.

[0167] Table 2. TuSimple Lane Marking Challenge leaderboard as of March 14, 2018 (test set)

[0168]

[0169] Results - Ablation Study

[0170] Table 3 compares different choices of using embedding / cross entropy as the adversarial loss term for training the generator and discriminator networks. To compare the stability of training, statistics on validation accuracy are reported.

[0171] Table 3. TuSimple validation set accuracy statistics for different training iterations (every 10K), comparing the stability of different choices for adversarial loss

[0172]

[0173]

[0174] Figure 8 Shows a plot illustrating the validation F-score during training. These results show that using the embedding loss for the generator stabilizes GAN training. We observe similar behavior when training with other hyperparameters.

[0175] Features used for the embedding loss can be used at different locations in the discriminator. In this section, we explore three options: using features after the 3rd, 5th, or 7th dense block. We note that the 3rd block contains the first shared convolutional layer with the image input and the prediction or label, and the 7th block contains the last set of convolutions before the classifier of the network. Table 4 and Fig. 9 Given in TuSimple TM Results on the validation set for lane marking detection. From these results, we conclude that, at least for this dataset, the later we make the embeddings, the better the scores are and the more similar the predictions are to the labels.

[0176] Table 4. Selecting embedding loss layers

[0177]

[0178] Evaluate

[0179] In this section, we evaluate the described embodiments and the results.

[0180] Evaluation - Comparison with other lane marking detection methods

[0181] In Table 2, we show TuSimple TM Results on the lane marking dataset, where El-GAN ranks 4th on the leaderboard. In this section, we compare our method in more detail with two other public methods: Pan et al. [see Pan et al., “Spatial As Deep: Spatial CNN for Traffic Scene Understanding”, AAAI (February 2018)] (ranked 2nd) and Nevin et al. [see Nevin et al., “Towards End-to-End Lane Detection: an Instance Segmentation Approach”, Electronic Preprint Library (February 2018)] (ranked 5th).

[0182] Both works use a multi-class approach to lane marking detection, where each lane marking is a separate class. While this simplifies post-processing, it requires more complexity in label creation and makes the network's task more difficult: it should now also learn which lane is which, requiring a larger field of view and introducing ambiguity when lanes change. In contrast, with our GAN approach, we learn a simpler single-class problem without the need for complex post-processing to separate individual markings.

[0183] Nevin et al. also believe in their work that post-processing techniques such as curve fitting are preferably not performed on the output of the network, but from a bird's-eye view. To this extent, they train a separate network to learn homography to find a perspective transformation that is easier to curve fit.

[0184] In our work, we show that it is possible to achieve comparable accuracy results without having to perform curve fitting at all, thus eliminating the need to train and evaluate a separate network for this purpose.

[0185] Pan et al. argue that problems such as lane marking detection can benefit from spatial consistency and message passing before making a final prediction. For this reason, they propose to feed the output of the regular segmentation network into a problem-specific 'spatial CNN' where message passing convolutions are performed in different directions. This does lead to TuSimple being more efficient than El-GAN. TMThe better accuracy on the TuSimple dataset, however, is unclear how much of this is due to its spatial CNN and how much is due to the fact that it was trained on a non-public dataset that is less accurate than the regular TuSimple dataset. TM The data set is twenty times larger.

[0186] Evaluation - Analysis of Ablation Studies

[0187] As we can see in the comparison table 3 and Figure 8 Using the embedding loss for the generator makes the training more stable and prevents collapse, as observed when comparing different adversarial loss terms presented in . Compared to the usual formulation using the cross-entropy loss, the embedding loss provides a stronger signal because it exploits the existing ground truth instead of just basing it on the discriminator’s internal representation of fakeness and plausibility.

[0188] Therefore, using the normal cross entropy loss can lead to a crash, where the generator starts exploring samples in feature space where the discriminator's fake / real understanding is not well formed. In contrast, using the embedding loss, such noise generation leads to a high degree of variance in the embedding space and is severely penalized by the embedding loss. Furthermore, having an overwhelming discriminator that can perfectly distinguish between fake and real distributions leads to training crashes and instabilities. Therefore, using an embedding loss with better gradients flowing back to the generator can produce a more capable generator. Similarly, it is not surprising that using the embedding loss for the discriminator instead of the generator leads to severely divergent behavior, since the discriminator is much more dominant and the generator is not penalized too much for generating noise.

[0189] In the second ablation study, as shown in Table 4 and Fig. 9 As presented in , we observe that using deeper representations to extract embeddings leads to better performance. This is likely due to the larger receptive field of the embeddings, which better enables the generator to improve higher-level quality and consistency.

[0190] Evaluation - GAN for Semantic Segmentation

[0191] Take a closer look at the comparison between regular CNN and El-GAN (see Figure 7 ), we see a clear difference in the nature of their outputs. While the non-GAN network produces probabilistic outputs with a probability per class per pixel, the output of the El-GAN resembles possible labels without expressing any uncertainty. The lack of ability to express uncertainty may be simply seen as a hindrance to further post-processing. However, the first step of commonly applied post-processing schemes is to remove probabilities by thresholding or applying argmax. Additionally, the independent per-pixel probabilistic outputs of regular CNNs may hide inter-pixel correlations that are required for correct post-processing. The cross-entropy loss pushes the network to output a segmentation distribution that does not lie on the manifold of possible labels.

[0192] In El-GAN and other GANs for semantic segmentation, the network is trained to output samples from a distribution of possible labels conditioned on the input image.

[0193] Fig.10 Shown are examples of sample selection once lane markings are occluded and the network becomes more uncertain. The left view is a close-up of the input data. The center view is a regular semantic segmentation output. The right view shows the superimposed output of three different El-GAN models trained with the same settings. Although this sacrifices the possibility of expressing uncertainty, the fact that it lies on or close to the manifold of possible labels can make post-processing easier and more accurate. For the task of lane marking detection, we have indeed shown that semantic segmentation does not require output probabilities. However, this may not be the case for other applications. A straightforward way to reintroduce expressing uncertainty through GANs is to simply run the GAN multiple times conditioned on additional random inputs, or to use an ensemble of El-GANs. The resulting samples, which model the probabilities on the manifold of possible labels, are then used as input to post-processing.

[0194] In the above, we have revealed, studied, and compared El-GAN as a method to preserve label-like quality in network predictions. We showed that using El-GAN leads to a more stable adversarial training process. In addition, we compared it to other competing methods in TuSimple TM We achieve state-of-the-art results on the challenge without using any additional data or complex hand-designed post-processing pipelines.

[0195] Those skilled in the art will recognize that the invention has been illustrated by describing one or more particular embodiments thereof, but that the invention is not limited to these embodiments; many variations and modifications are possible within the scope of the appended claims.

Claims

1. A method for training a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises: Generator neural network; and a discriminator neural network, which includes one or more layers before the classifier, The method comprises: providing an image as input to the generator neural network; receiving a predicted segmentation map for the image from the generator neural network; providing i) the image, ii) the predicted segmentation map, and iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network; determining, in response to the training input, a set of one or more outputs from the discriminator neural network, wherein the one or more outputs include one or more embedding outputs obtained from at least one of the layers within the discriminator neural network; and The generator neural network is trained using a loss function that is a function of the set of outputs from the discriminator neural network.

2. The method of claim 1, further comprising using the trained generator neural network to segment a received image of a road and identify one or more lane markings in the image.

3. A method according to claim 1 or 2, wherein the loss function includes an embedding term representing the difference between: i) the embedding at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network; and ii) the embedding at the predetermined layer of the discriminator neural network when the ground truth label data is input to the discriminator neural network. The method of claim 3 , wherein the difference is an L2 distance.

5. The method of claim 3, wherein the predetermined layer is immediately after a first dense block, the first dense block containing one or more shared convolutional layers with both the image and the predicted segmentation map.

6. The method of claim 3, wherein the predetermined layer is immediately before the classifier of the discriminator neural network and after a final dense block.

7. The method of claim 3, wherein the loss function additionally includes a pixel-level fitness term, and wherein the loss function includes a weighting parameter λ for weighting the embedding term relative to the fitness term.

8. The method of claim 1 or 2, further comprising training the discriminator neural network to minimize a loss on distinguishing the discriminator neural network between predicted segmentation maps and ground truth label data.

9. A computer processing system implementing a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises: Generator neural network; a discriminator neural network, which includes one or more layers before the classifier; and Training logic, and wherein the training logic is configured to: providing an image as input to the generator neural network; receiving a predicted segmentation map for the image from the generator neural network; providing i) the image, ii) the predicted segmentation map, and iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network; determining, in response to the training input, a set of one or more outputs from the discriminator neural network, wherein the one or more outputs include one or more embedding outputs obtained from at least one of the layers within the discriminator neural network; and The generator neural network is trained using a loss function that is a function of the set of outputs from the discriminator neural network.

10. The computer processing system of claim 9, wherein the generative adversarial network, after being trained, is configured to segment a received image of a road and identify one or more lane markings in the image.

11. A computer processing system according to claim 9 or 10, wherein the loss function includes an embedding term representing the difference between: i) the embedding at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network; and ii) the embedding at the predetermined layer of the discriminator neural network when the ground truth label data is input to the discriminator neural network.

12. The computer processing system of claim 11, wherein the difference is an L2 distance.

13. The computer processing system of claim 11, wherein the predetermined layer immediately follows a first dense block, the first dense block containing one or more shared convolutional layers with both the image and the predicted segmentation map.

14. The computer processing system of claim 11, wherein the predetermined layer is immediately after a final dense block before the classifier of the discriminator neural network.

15. The computer processing system of claim 11, wherein the loss function additionally includes a pixel-level fitness term, and wherein the loss function includes a weighting parameter λ for weighting the embedding term relative to the fitness term.

16. The computer processing system of claim 11, wherein the training logic is further configured to train the discriminator neural network to minimize a loss with respect to distinguishing the discriminator neural network between predicted segmentation maps and ground truth label data.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processing system, cause the computer processing system to perform a method of training a generative adversarial network for performing semantic segmentation of an image, wherein the generative adversarial network comprises: Generator neural network; and a discriminator neural network, which includes one or more layers before the classifier, The method comprises: providing an image as input to the generator neural network; receiving a predicted segmentation map for the image from the generator neural network; providing (i) the image, (ii) the predicted segmentation map, and (iii) ground truth label data corresponding to the image as distinct training inputs to the discriminator neural network; determining, in response to the training input, a set of one or more outputs from the discriminator neural network, wherein the one or more outputs include one or more embedding outputs obtained from at least one of the layers within the discriminator neural network; and The generator neural network is trained using a loss function that is a function of the set of outputs from the discriminator neural network.

18. The non-transitory computer-readable storage medium of claim 17, wherein the loss function comprises an embedding term representing a difference between: (i) an embedding at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network, and (ii) an embedding at the predetermined layer of the discriminator neural network when the ground truth label data is input to the discriminator neural network.

19. The non-transitory computer-readable storage medium of claim 18, wherein the predetermined layer is immediately after a first dense block, the first dense block containing one or more shared convolutional layers with both the image and the predicted segmentation map.

20. The non-transitory computer-readable storage medium of claim 18, wherein the predetermined layer is immediately after a final dense block before the classifier of the discriminator neural network.

21. The non-transitory computer-readable storage medium of claim 18, wherein the loss function additionally includes a pixel-level fitness term, and wherein the loss function includes a weighting parameter λ for weighting the embedding term relative to the fitness term.

Images