Training generative adversarial networks

Abstract

Training generative adversarial networks. Training systems and computer-implemented training methods for training generative adversarial networks (GANs) are described. In training, interpretive information can be provided to a generative component (G) that identifies on what basis a discriminative component arrives at its classification. In particular, the interpretive information can be provided in the form of an attention mask (s) that can be generated by the discriminative component (D) and that can identify portions of a respective input instance that contribute to the classification of the respective input instance by the discriminative component. Further, data synthesis systems and computer-implemented methods for synthesizing data using a generative component of a trained generative adversarial network are described.

Description

TECHNICAL FIELD

[0001] The present invention relates to a training system and a computer-implemented training method for training a generative adversarial network. The present invention further relates to a data synthesis system and a computer-implemented data synthesis method for synthesizing data using a generative part of a trained generative adversarial network. The present invention further relates to a computer-readable medium comprising data of a trained generative adversarial network, and a computer-readable medium comprising data representing instructions arranged to cause a processor system to perform at least one computer-implemented method. BACKGROUND

[0002] Generative adversarial networks (GANs) have been introduced by Ian Goodfellow et al. in 2014 [1]. In their paper, a framework for estimating generative models via adversarial networks is presented, in which two models are trained simultaneously: a generative model G that captures the data distribution and a discriminative model D that estimates the probability that an input instance is obtained from the training data (input is "real") rather than from the generative model G (input is "fake").

[0003] Recent research has shown that the generative model of such trained generative adversarial networks, in particular when using deep generative models such as deep convolutional neural networks, are able to synthesize images that look natural with high resolution and with sufficient quality to even fool human observers.

[0004] There are also many other real-world applications of trained GANs, and in particular of the generative model of trained GANs, ranging from anomaly detection, synthetic data generation for discriminative model training, to so-called "explainability" tools for data sets and models. For example, in autonomous driving, various types of sensor data can be fused to establish a 360-degree view of the surroundings of a vehicle. In case the field of view of different types of sensors is different, a generative model can be trained and used to synthesize missing sensor data outside the sensor field of view based on sensor data of another sensor.

[0005] Another example in the field of autonomous driving is that trained GANs can be used to generate "edge case" scenarios for autonomous driving (e.g. synthetic images representing near-collision), which can be used to test and verify the performance of autonomous driving algorithms and systems in such scenarios.

[0006] In the following, the generative model is also referred to as generative part of a GAN or simply as "generator", and the discriminative model is also referred to as discriminative part of a GAN or simply as "discriminator".

[0007] Training of GANs typically involves the following. The generative part G can be configured to generate synthetic output instances from noisy samples drawn from a latent space. The discriminative part D can be trained to discriminate between input instances originating from the generative part G and training data. The generative part G can be trained to generate synthetic output instances from noisy samples that maximize a discrimination error when the discriminative part D is applied to the synthetic output instances. Such training of the discriminative part D and the generative part G can be iterative, where in each iteration the weights of the discriminative part D and the generative part G can be updated, e.g., by computing gradients via backpropagation.

[0008] The benefits of GANs can come at a cost. That is, GANs are difficult to train because they comprise not one but two main components that can work adversarially in a zero-sum game and can be trained to find a Nash equilibrium. This can already be a difficult task conceptually. However, from a practical perspective, there are several difficult and often unsolved problems to deal with, which include mode collapse, high-variance gradients, training instability, vanishing generator gradients, etc. While some problems are being addressed, there is currently no consensus on how to train GANs in an efficient way, let alone on how to understand their development throughout the training process.

[0009] The latter point is crucial in the context of explainable artificial intelligence (AI). That is, when deploying a trained model in real-life applications, the output of the trained model should be explainable, e.g., to ensure trust in the trained model. Such explainability is particularly crucial for safety-critical applications such as autonomous driving and is increasingly subject to regulations by authorities.

[0010] By definition, explainable models are also predictable. However, in the context of GAN training, the training process itself is often mysterious because generator and discriminator gradients are not only a function of the current model state or input from the training data, but also of each other. This makes it difficult to explain any apparent patterns in the learning history of these components. Additionally, the gradient curves can be almost impossible to relate to the quality of the synthetic output of the generative model.

[0011] References

[0012] [1] Ian J. Goodfellow et al., “Generative Adversarial Nets,” in NIPS 2014, pp. 2672-2680, Generative Adversarial Networks . https: / / arxiv.org / abs / 1406.2661 . SUMMARY

[0013] It can be desirable to be able to improve the training of a generative adversarial network, for example to be able to obtain from the training information that supports the explainability of the trained generative adversarial network.

[0014] According to a first aspect of the present application, there is provided a training system and a computer-implemented training method for training a generative adversarial network, as defined in claims 1 and 13, respectively. According to a further aspect of the present application, there is provided a data synthesis system and a computer-implemented data synthesis method for controlling or monitoring a physical system using a trained model, as defined in claims 12 and 14, respectively. According to a further aspect of the present application, there is provided a computer-readable medium comprising a trained generative adversarial network, as defined in claim 11. According to a further aspect of the present application, there is provided a computer-readable medium, as defined in claim 15.

[0015] The above measures provide for the training of a generative adversarial network that builds on known training techniques (e.g. as described in [1] and in the background section of the present specification), wherein the generative part and the discriminative part of the GAN are jointly trained, such that the generative part learns from the gradients of the discriminative part. In essence, the discriminative part can classify input instances as real (originating from the training data) or fake (originating from the synthetic output of the generative part), and the generative part can learn from this classification in order to better “fool” the discriminative part in the future, e.g. to maximize the discriminative error of the discriminative part.

[0016] Adversely, known training techniques for the generative part do not use any insight or details about what specific criteria the discriminative part uses to derive its classification of a respective input instance. This can slow down the training of the generative part and can lead to problems such as the discriminative part outpacing the generative part at the beginning of the training process.

[0017] The above measures provide the generative part with explanation information that identifies on which basis the discriminative part derives its classification. This explanation information is provided in the form of an attention mask that identifies the parts of a respective input instance that contribute to the classification of the respective input instance by the discriminative part, and can be understood as a mask that indicates the underlying reasons for the classification by the discriminative part. The attention mask can be generated by the discriminative part from the state of the discriminative part.

[0018] Such attention masks are known per se and are also referred to as “saliency maps” in case the input instances are images (see also chapter 2 of the reference [2] cited under “Further References” below), but can also take any other suitable form.

[0019] Such attention masks can represent more specific feedback cues than the softmax scores that are typically provided by the discriminative part and used for the generative part training. That is, the individual softmax scores can cause the generative part to entirely adapt its synthetic output in order to reduce the discriminative part's confidence that the synthetic output is fake. While such a "destructive" change can be desirable in the initial stages of training, as the generative part's training progresses, it can be desirable to use more insightful analysis to control the way the generative part synthesizes output instances.

[0020] The above measures provide and use attention masks as a means of conveying this nuanced information. Thus, the generative part can generate new synthetic output instances not only in accordance with how the discriminative part reacts to fake input instances, but also in accordance with where the discriminative part focuses (e.g., on which part of the input instance) to derive its classification.

[0021] The above measures can thereby establish an interpretable training process, as the attention masks indicate on which part(s) the discriminative part focuses during each training step, and thus on which part(s) the generative part has the potential to focus during the training step in order to better fool the discriminative part. Moreover, for example, the speed and robustness of the training can be improved when using attention masks as compared to training using only softmax scores.

[0022] In particular, as described in the background section, the training of a GAN can include an outer iteration loop. The discriminative part of the GAN can be modified with respect to the known discriminative part to output an attention mask that identifies a part of a respective input instance that contributes to the classification of the respective input instance by the discriminative part. Such attention masks can take various forms depending on the type of input instances. For example, in the case that the GAN is trained to generate images, the attention mask can represent an image overlay that highlights the part of the image that contributes to the classification.

[0023] The generative part of the GAN can be modified with respect to the known generative part to receive the attention mask as input and to generate a synthetic output instance not only in accordance with a noisy sample drawn from a latent space (which is also referred to as a "latent vector" or simply "sampling noise" and denoted by "z"), but also in accordance with the content of the attention mask.

[0024] Further, an inner loop is established in the iterative training in which the parameters of the generative part are trained, e.g. the model parameters of the generative model. The training is based on a loss function, which can be based on a known loss function by which the generative part is trained to generate synthetic outputs that "fool" the discriminative part. This loss function can be modified to additionally include a mutual information term. The mutual information term can express the mutual information between the current attention mask of the discriminative part and the changes in the current synthetic output instance that are caused by using the current attention mask compared to the previous attention mask. More specifically, in the inner iterative loop, the current attention mask can be obtained from the discriminative part by classifying the current synthetic output. The latter can be an output instance of the generative part and can be generated from the latent vector z and the previously computed attention mask. If the mask has not been computed yet, e.g. at the beginning of the iteration, a default mask can be used. In the inner iterative loop, the mutual information term can be maximized as part of minimizing the loss function. Thus, when minimizing the loss function by optimization or brute force search, the mutual information term with the higher value is preferred, all else being equal. Thus, the inner iterative loop can seek to maximize the mutual information between the attention mask resulting from the discriminative part and the changes proposed by the generative part. Thus, the inner iterative loop can train the generative part such that for a given iteration, the changes proposed by the generative part have a high mutual information with the attention mask resulting from the discriminative part. In essence, the generative part can determine which part(s) of the input instance contribute to the classification by the discriminative part and try to "fool" it there.

[0025] Optionally, the processor subsystem is configured to keep the noisy sample used as input to the generative part constant in the inner iterative loop of the iterative training. The latent vector (the noisy sample z) can be kept constant in the inner iterative loop to account for the fact that the attention mask is generated from the latent vector and thus only applies to a specific latent vector. It is noted that the inner iterative loop can then be performed several times, each time using a different sampled latent vector, e.g. as part of the outer iterative loop in the overall training of the generative adversarial network.

[0026] Optionally, the processor subsystem is configured to use, at the beginning of the inner iteration loop, an initial attention mask as the previous attention mask, where the initial attention mask equally identifies all portions of the respective input instance. For example, an attention mask consisting of all ones ("1") can be used as the initial attention mask, where the value "1" represents maximum attention. Thus, the attention mask can indicate attention of the discriminative portion that is uniformly distributed across the entire input instance, e.g., labeling each portion of the input instance that has attracted equal attention of the discriminative portion. Thus, in the initialization of the inner iteration loop, the generative portion can be discouraged from being biased to try to fool the discriminative portion in any particular portion of the input instance, but rather still equally consider all portions. Note that the same type of initial attention mask, e.g., an attention mask consisting of all ones ("1"), can be used as input to the generative portion when using the generative portion of the trained GAN to synthesize data for real-world applications.

[0027] Optionally, the mutual information term is an approximation of the true mutual information between the current attention mask (s) and the change in the synthesized output instance. Optionally, the mutual information term is lower-bounded by a variational information maximization. For example, the mutual information between the current attention mask and the change in the synthesized output instance can be approximated in the same or similar manner as described in [3] for the mutual information between a noise variable and an observation.

[0028] Optionally, the training data comprises training images, the generative portion is configured to generate a synthesized output image, and the discriminative portion is configured to discriminate between input images originating from the generative portion and the training data. Note that the GAN can also be configured and trained for use with any other suitable data type, including but not limited to text, structured data such as graphs, and audio. In the particular example of audio, the term "portion" of the input instance can refer to, e.g., a spectral (frequency) portion within a given time window. In some embodiments, the GAN can be configured and trained for use with non-image data that can be represented in a spatial format, such as image data, such as measurement data obtained from a non-image sensor.

[0029] Optionally, the discriminative portion is configured to generate an attention mask as a spatial overlay on the synthesized output image that is used as input to the discriminative portion. For example, the attention mask can be an image overlay that highlights portions of the image that contribute to the classification.

[0030] Optionally, the training system further comprises an output interface configured to output log information at least partially characterizing the training of the generative adversarial network. Optionally, the processor subsystem is configured to output one or more attention masks generated in an inner iteration loop of said training as the log information or as part of the log information. By outputting the attention mask(s), log information can be obtained which can be used to support the explainability of the trained generative adversarial network. Such log information can be stored in a persistent manner, e.g. in order to comply with regulatory restrictions or to enable the training process to be "audited".

[0031] Optionally, the processor subsystem is configured to generate the log information as metadata to the generative model data defining the trained generative adversarial model. The attention mask(s) can thereby be stored in a manner in which they are associated with the trained generative adversarial model.

[0032] The person skilled in the art will appreciate that two or more of the above-mentioned embodiments, implementations and / or optional aspects of the application can be combined in any manner deemed useful.

[0033] Modifications and variations of any of the systems, any of the computer-implemented methods or any of the computer-readable media can be performed by a person skilled in the art on the basis of the present description, which correspond to the described modifications and variations of the other of said entities.

[0034] Further references

[0035] [2] Julius Adebayo et al., "Synthesizing Robust Adversarial Examples", arXiv: 1712.06600v1, 2017. Sanity Checks for Saliency Maps https: / / arxiv.org / abs / 1810.03292

[0036] [3] Xi Chen et al., "Show Your Work: Improved Reporting Metrics for Neural Machine Translation", arXiv: 1811.10071v1, 2018. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets https: / / arxiv.org / abs / 1606.03657 . BRIEF DESCRIPTION OF DRAWINGS

[0037] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter and with reference to the accompanying drawings, in which

[0038] Figure 1 a training system for training a generative adversarial network is shown, the training system comprising an output interface configured to output log information characterizing the training of the generative adversarial network;

[0039] Figure 2 ​​A flowchart showing a portion of the training of the generative portion of a generative adversarial network is shown;

[0040] Figure 3 An alternative illustration of the training of the generative portion of a generative adversarial network is provided;

[0041] Figure 4 A data synthesis system for synthesizing data using the trained generative portion of a generative adversarial network is shown;

[0042] Figure 5 A data synthesis system for synthesizing missing sensor data outside of a sensor field of view in an autonomous vehicle is shown;

[0043] Figure 6 A computer readable medium including data is shown.

[0044] It should be noted that the various figures are purely illustrative and not drawn to scale. In the various figures, elements corresponding to elements already described can have the same reference number.

[0045] List of Reference Numbers

[0046] The following list of reference numbers is provided to facilitate the explanation of the drawings and should not be interpreted as limiting the claims.

[0047] 100 training system

[0048] 120 input-output interface

[0049] 140 data storage

[0050] 142 generative model data

[0051] 144 training data

[0052] 146 log information

[0053] 200 inner iteration loop for training generative portion

[0054] 210 repeat k iterations

[0055] 220 until batch is empty

[0056] 200 data synthesis system

[0057] 220 input-output interface

[0058] 240 data storage

[0059] 242 generative model data

[0060] 244 synthesized data

[0061] 300 environment

[0062] 310 autonomous vehicle

[0063] 320 image sensor

[0064] 330 electric motor

[0065] 400 computer-readable medium

[0066] 410 non-transitory data. DETAILED DESCRIPTION

[0067] The following relates to training a generative adversarial network (GAN) and using the trained generative adversarial network for data synthesis. In particular, reference is made to Figures 1-3 Training of a GAN is described, with reference to Figure 4 and Figure 5 Data synthesis using a trained GAN is described.

[0068] Figure 1 A training system 100 for training a generative adversarial network is shown. The training system 100 is shown to comprise an input interface 120 configured to access generative model data 142 defining a GAN, which initially can be considered an “untrained” GAN, in that the parameters of the GAN can not yet have been trained, or at least not to a degree that can be considered sufficient. The input interface 120 can further be configured to access training data 144, which can comprise a set of training instances for training the GAN, such as images, text snippets, audio snippets or other types of data on which the GAN is to be trained.

[0069] As shown in Figure 1 The input interface 120 can be a data storage interface 120 to a data storage 140, which can comprise the data 142, 144, as shown in

[0070] It is noted that the input interface 120 can also be an output interface, e.g. an input-output (“I / O”) interface 120. The training system 100 can use the input-output interface 120 to store data such as (parameters of) the trained GAN and / or log information 146, as also described elsewhere in this specification.

[0071] The GAN as defined by the generative model data 142 can comprise a discriminative part which can be configured to discriminate between input instances originating from the generative part and the training data, and output a classification and an attention mask which identifies parts of the respective input instance which contribute to the classification of that respective input instance by the discriminative part, e.g. as originating from the generative part (“fake”) or as originating from the training data (“real”). The GAN as defined by the generative model data 142 can further comprise a generative part which can be configured to receive the attention mask as input and generate a synthetic output instance from i) a noisy sample drawn from the latent space and ii) the attention mask.

[0072] In the following, the generative part of the GAN is also referred to as generative model or simply generator or G, and the discriminative part of the GAN is also referred to as discriminative model or simply discriminator or D.

[0073] The training system 100 is further shown as comprising a processor subsystem 160 which is configured to train the GAN in an iterative manner based on the training data 144, wherein the generative part can generate synthetic output instances from noisy samples drawn from the latent space, and wherein the discriminative part can be trained to discriminate between input instances originating from the generative part and the training data. In particular, the generative part can be trained to generate synthetic output instances from noisy samples which maximize a discriminative error when the discriminative part is applied to the synthetic output instances. Such training of the discriminative part and the generative part can be iterative, wherein in each iteration the weights of the discriminative part and the generative part can be updated, e.g. by computing gradients via backpropagation. Such training can be known per se and can also be referred to in the following as an “outer iteration loop”.

[0074] The processor subsystem 160 can further be configured to establish an inner iteration loop in the iterative training in which the parameters of the generative part are trained by iteratively minimizing a loss function for the generative part. Reference will be made to Figure 2 and Figure 3 This training aspect is further described.

[0075] Although in the above the training of the GAN is described as being performed by the training system 100, it is noted that the training of the GAN can also be performed by another system, e.g. by a system which is configured to perform the training of the GAN and which is configured to provide the trained GAN to the training system 100. Figure 1The trained generative model data can be output by the training system 100, but is not explicitly shown. For example, the generative model data 142 defining the "untrained" GAN can be replaced by the generative model data of the trained GAN during or after training, as the parameters of the GAN can be adapted to reflect the training on the training data 144. In other embodiments, the trained generative model data can be stored separately from the generative model data 142 defining the "untrained" GAN.

[0076] In general, the training system 100 can be embodied as or in a single device or apparatus, such as a workstation or server. The server can be an embedded server. The device or apparatus can comprise one or more microprocessors executing appropriate software. For example, the processor subsystem can be embodied by a single central processing unit (CPU), but also by a system or combination of such CPUs and / or other types of processing units. The software can have been downloaded and / or stored in a corresponding memory, e.g. a volatile memory such as a RAM, or a non-volatile memory such as a flash memory. Alternatively, the processor subsystem of the system can be implemented in the device or apparatus in the form of programmable logic, for example as a field programmable gate array (FPGA). In general, each functional unit of the training system 100 can be implemented in the form of circuitry. The training system 100 can also be implemented in a distributed manner, e.g. involving different devices or apparatuses, such as distributed local or remote, e.g. cloud-based, servers.

[0077] The following examples describe the training of a GAN in more detail. However, the actual implementation of the training can be carried out in various other ways, e.g. on the basis of similar mathematical concepts. The following assumes that the input instances (short: inputs) and the output instances (short: outputs) will be images, but can equally be applied to any other suitable data type.

[0078] The training of the GAN can be defined by the following objective:

[0079]

[0080] where D is a discriminator (e.g. a neural network, such as a convolutional neural network) that determines whether an input is real (i.e. determined to come from the data distribution ) or fake (generated by the generator G), and G is a generator that maps a latent space (z of the latent space is a sample) to an image space in this example. Intuitively, D can be trained to be able to distinguish between real and fake samples, and G can be adversarially trained to try to fool D.

[0081] For this purpose, the discriminator D can be configured to output not only a probability that a given sample is real or fake, but also an attention mask, in this example, an image-based attention mask: a saliency mask s.

[0082]

[0083] It is noted that, An image input instance can be represented by a data vector reshaped to but can also be in a more traditional image definition for The saliency mask s can be a function of parameters of D and of the generated output of the generator G, which in turn can be modeled as a function of z, a latent space vector, and a previous saliency mask s’:

[0084] .

[0085] The generator G can thus map a sample from the latent space and a saliency mask to a generated image. Here, k denotes the dimensionality of the latent sample, i.e. the dimensionality of z.

[0086]

[0087] For generating the saliency map, any suitable saliency method can be used, e.g. as explored for deep learning discriminative models [2]. For example, the saliency map s can be a matrix with values in the range [0, 1] and can indicate which pixels are relevant for the input image classification by D (with values close to 1).

[0088] The training objective can be defined with the following loss function:

[0089] .

[0090] In this loss function, D and G are the discriminator D and the generator G as described above, can be a known GAN loss function (which only takes the traditional outputs of D and G into account, without taking the saliency mask s into account), is a regularizer term, and is an approximation of the mutual information between the current saliency mask s and the quantity as described below:

[0091] .

[0092] In this formula, s' can be the previous saliency mask (as will be further explained below), but it can initially be a default saliency mask which, for example, indicates a matrix of all ones that every pixel is important for classification.

[0093] Intuitively, this can be interpreted as the generator G being trained in such a way that the mutual information between the saliency mask produced by D and the changes proposed by G is maximized. Thus, the generator can specifically look where the discriminator is putting its attention and try to fool it there.

[0094] Value is an approximation of the true mutual information I between s and and can follow the derivation in [3] as follows:

[0095] .

[0096] This derivation indicates that, given that the entropy of the saliency map s is known, the residual can be computed and then , is estimated.

[0097] Note that [3] uses variational autoencoders to derive a differentiable estimate of Depending on the type of input and output instances, the estimate of can take various forms. For example, in case the input and output instances are images and the saliency mask s and the residual are both images, the can be estimated by any suitable differentiable image similarity measure known per se. In a particularly efficient implementation, can be estimated as the dot product of the saliency mask s and the residual .

[0098] When now again considering the general GAN training, which typically involves sampling batches of noise z from the latent space, this is followed by training the discriminator D to distinguish the synthetic output of the generator from real input instances from the training data. Then, the generator G is trained on what can be considered a relative goal, because a new batch of sampled noise z should be recognized as real by the current state of the discriminator D. It is therefore preferred not to freely pair random input samples z with the current saliency mask s, because the current saliency mask is a function of the current state of the generator G and thus of z, as stated previously, .

[0099] Therefore, the previous attention mask s' can be considered to be valid only for the same z. Once a new z is used, the initial attention mask representing equal attention can then be used as the previous saliency mask s' in the first iteration of the inner iterative loop, and determined by s' = s in subsequent iterations.

[0100] Figure 2 A flowchart illustrating a portion of the training for solving the generative part of the GAN described above is shown. In this paper, the parameters of the generator G are trained by keeping z constant, and the discriminator D is used to update the saliency mask s as the discriminator reacts to new synthetic output instances generated by the generator G. This is achieved by introducing an inner iterative loop 200 in the generator training, where the current z remains constant and a new saliency mask is generated as a response to the synthetic output of G. More specifically, in the current iteration of the inner iterative loop 200, D can be applied to the synthetic output. This is in Figure 2 The middle is marked as , and where s' is a previous saliency mask (s' = s), or, in the case that such a previous saliency mask is unavailable (e.g., at the beginning of inner iteration loop 200), s' is an initial saliency mask, such as a saliency mask consisting of all ones (s' = multiple ones). If the discriminator D determines the synthesized output It's fake—it's in Figure 2 The middle is marked as " If the result is a negative "is true", then a saliency mask s (Gen s) can be generated. The generator G can use this saliency mask s to generate the synthesized output. After this, the inner iteration loop can be repeated, for example, within k iterations, as shown by arrow 210. Upon completion of the inner iteration loop 200, a new latent vector z can be obtained, and the inner iteration loop can be repeated, for example, as shown by arrow 220, until the batch of latent vector z is empty or another type of termination criterion has been reached.

[0101] Figure 3 An alternative view of training the generative part G of a generative adversarial network is shown. Here, the discriminator D is shown as being applied to a real input instance—in this case, an image ( Real-Image ), and applied by the generator G from noisy samples z and previous saliency masks. The generated synthetic image. If the discriminator D determines that the image is not real, it outputs the current saliency mask s, and then the generator G uses the current saliency mask s in the next iteration—in which the previous saliency mask is used again (s' = s).

[0102] It is noted that Figure 2and Figure 3 This can be implied that the attention mask is generated by the discriminative part only if the input instance is judged as fake. However, the discriminative part can also be configured to generate an empty saliency mask if the input instance is judged as real, indicating things in the image that the generative part should not change. Furthermore, it will be appreciated that in practice, The output can be a real number, such as a floating-point value between 0 and 1, rather than binary 0 or 1. In such cases, the saliency mask can rarely be completely empty, and the saliency map can be obtained via... The output is generated by backpropagation of the input, that is, The derivative of the output with respect to the derivative of the input.

[0103] Figure 4 A data synthesis system 200 is shown for synthesizing data using the generative part of a trained generative adversarial network. The data synthesis system 200 is shown as including an input interface 220 configured to access generative model data 242, which at least defines the generative part (G) of a GAN trained by a training system 100, as referenced. Figures 1-3 And as described elsewhere. Figure 4 Generative model data 242 accessed from data storage device 240 is shown. In some embodiments, generative model data 242 may further include a discriminative portion (D), while in other embodiments, the generative model data 242 may omit the discriminative portion and instead include only the generative portion.

[0104] Note that, as previously addressed separately Figure 1 The same implementation options described for the input interface 120 and data storage device 140 of the training system 100 can be applied to the input interface 220 and data storage device 240.

[0105] The data synthesis system 200 is further shown to include a processor subsystem 260, which can be configured to use the generative part of the GAN to generate one or more synthetic output instances from noisy samples z drawn from the latent space. In doing so, attention masks that equally identify all parts of the corresponding input instances can be used, such as saliency masks consisting of all ones in the case where the GAN is trained to generate images.

[0106] The data synthesis system 200 may further include an output interface configured to output a synthesized output instance as synthesized data 244. Figure 4In the example of FIG. 2, the input interface is an input-output (“I / O”) interface, which can thus also embody the output interface, and the synthetic data 244 can be stored in the data store 240 via the input-output (“I / O”) interface. However, the output interface can also be separate from the input interface and can be of a different type. In general, the same implementation options as previously described for the input interface 120 of the training system 100 of FIG. 1 can apply to the output interface 220. Figure 1

[0107] In general, the data synthesis system 200 can be embodied as or in a single device or apparatus, such as a workstation or server. The server can be an embedded server. The device or apparatus can comprise one or more microprocessors executing appropriate software. For example, the processor subsystem can be embodied by a single central processing unit (CPU), but also by a system or combination of such CPUs and / or other types of processing units. The software can have been downloaded and / or stored in corresponding memory, e.g. a volatile memory such as a RAM, or a non-volatile memory such as a flash memory. Alternatively, the processor subsystem of the system can be implemented in the device or apparatus in the form of programmable logic, e.g. as a field-programmable gate array (FPGA). In general, each functional unit of the data synthesis system 200 can be implemented in the form of an electrical circuit. The system 200 can also be implemented in a distributed manner, e.g. involving different devices or apparatuses, such as distributed local or remote (e.g. cloud-based) servers.

[0108] Figure 5 A data synthesis system 200 used in an autonomous vehicle 310 is shown, which operates in an environment 300 to synthesize missing sensor data outside the field of view of an image sensor 320. Based on the sensor data and the synthetic data, an electric motor 330 can be controlled. In general, such a data synthesis system can be (part of) a physical entity such as a vehicle, a robot, etc.; or a connected or distributed system of physical entities, e.g. a lighting system; or any other type of physical system, e.g. a building.

[0109] Any of the methods described in this specification can be implemented as computer-implemented methods, special purpose hardware, or a combination of the two. As also described in this specification, any of the methods can be implemented as a computer program product. Figure 6 ​The instructions for a computer, e.g., executable code, illustrated in FIG. 4 can be stored, for example, in the form of a series of physical machine-readable physical marks and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values on a computer readable medium 400. The executable code can be stored in a transitory or non-transitory manner. Examples of computer readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 6 An optical disc 400 is shown. In alternative embodiments of the computer readable medium 400, the computer readable medium 400 can include transitory or non-transitory data 410 representing a trained GAN as described elsewhere in this specification.

[0110] Examples, embodiments or optional features, whether or not they are

[0111] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The verb "comprise" and its conjugations do not exclude the presence of elements or stages other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The expression "at least one of A and B" when preceding the list of elements A and B, should be understood as used in the sense of "at least one of the elements A, at least one of the elements B, and possibly additional elements, including in the case where only one of A and B is present in the list of elements". The expression "at least one of A or B" should be understood as used in the sense of "at least one of the elements A, at least one of the elements B, and possibly additional elements, including in the case where only one of A and B is present in the list of elements". The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A data synthesis system (200) for synthesizing data using the generative part of a trained generative adversarial network, the data synthesis system comprising: - An input interface (220) is configured to access generative model data (242), which at least defines the generative part (G) of a generative adversarial network trained by a training system. The training system mentioned above includes: - The second input interface (120) is configured to access: - Define generative model data for generative adversarial networks that include a generative part (G) and a discriminative part (D) (142). - Training data (144) for generative adversarial networks, wherein the training data (144) includes training images; The discriminative part (D) is configured to discriminate between the input image derived from the generative part (G) and the training data, and output a classification and attention mask(s), wherein the attention mask(s) identifies the parts of the corresponding input image that are helpful for the discriminative part (D) to classify the corresponding input image. The generative part (G) is configured to receive an attention mask (s) as input and generate a synthetic output image from i) noisy samples (z) drawn from the latent space and ii) the attention mask (s); and - A processor subsystem (160) configured to iteratively train a generative adversarial network based on training data by establishing an internal iterative loop (200) in iterative training, wherein the parameters of the generative part (G) are trained by iteratively minimizing a loss function for the generative part (G), wherein minimizing the loss function includes maximizing a mutual information term included in the loss function, the mutual information term representing mutual information between two terms: - The current attention mask(s) of the discriminative part (D), wherein the current attention mask(s) is obtained from the discriminative part (D) by classifying the previously synthesized output image of the generative part (G), the previously synthesized output image of the generative part (G) being generated using the previous attention mask(s') obtained from the discriminative part (D) in previous iterations of the inner iterative loop, and - The change between the current synthesized output image of the generative part (G) generated using the current attention mask(s) and the previous synthesized output image of the generative part (G); and - A second processor subsystem (260) is configured to generate one or more synthetic output images from noisy samples (z) drawn from the latent space using the generative part (G) of a generative adversarial network; - Output interface (220), which is configured to output the one or more composite output images as composite data.

2. The data synthesis system (200) according to claim 1, wherein the processor subsystem (160) is configured to keep the noisy sample (z) used as input to the generative part (G) constant in the internal iterative loop of iterative training.

3. The data synthesis system (200) according to claim 1 or 2, wherein, The processor subsystem (160) is configured to use an initial attention mask (s') as a previous attention mask at the beginning of an internal iteration loop, wherein the initial attention mask (s') equally identifies all parts of the corresponding input image.

4. The data synthesis system (200) according to claim 1 or 2, wherein the mutual information term is an approximation of the true mutual information between the current attention mask(s) and the changes in the synthesized output image.

5. The data synthesis system (200) according to claim 4, wherein, The lower bound of the mutual information term is defined by maximizing the variational information.

6. The data synthesis system (200) according to claim 1, wherein, The discriminative part (D) is configured to generate an attention mask as a spatial overlay of the synthesized output image, which is used as input to the discriminative part (D).

7. The data synthesis system (200) according to claim 1 or 2, wherein the training system further includes a second output interface (120) configured to output log information (146) that at least partially characterizes the training of the generative adversarial network.

8. The data synthesis system (200) according to claim 7, wherein, The processor subsystem (160) is configured to output one or more attention masks generated in the internal iterative loop of the training as log information (146) or as part of log information (146).

9. The data synthesis system (200) according to claim 7 or 8, wherein the processor subsystem (160) is configured to generate log information (146) as metadata for generative model data (142) defining the trained generative adversarial model.

10. A computer-implemented method for synthesizing data using the generative part of a trained generative adversarial network, the method comprising: - Access generative model data, which at least defines the generative part (G) of a generative adversarial network trained through the following steps: - Visit: - Define generative model data for generative adversarial networks, which include a generative part (G) and a discriminative part (D). - Training data for generative adversarial networks, wherein the training data includes training images; The discriminative part (D) is configured to discriminate between the input image derived from the generative part (G) and the training data, and output a classification and attention mask(s), wherein the attention mask(s) identifies the parts of the corresponding input image that are helpful for the discriminative part (D) to classify the corresponding input image. The generative part (G) is configured to receive an attention mask (s) as input and generate a synthetic output image from i) noisy samples (z) drawn from the latent space and ii) the attention mask (s); as well as - The generative adversarial network is iteratively trained based on training data by establishing an internal iterative loop in iterative training, wherein the generative part (G) is trained using the discriminative part (D) by iteratively minimizing the loss function for the generative part (G), wherein minimizing the loss function includes maximizing a mutual information term included in the loss function, the mutual information term representing mutual information between the following two terms: - The current attention mask(s) of the discriminative part (D), wherein the current attention mask(s) is obtained from the discriminative part (D) by classifying the previously synthesized output image of the generative part (G), the previously synthesized output image of the generative part (G) being generated using the previous attention mask(s') obtained from the discriminative part (D) in previous iterations of the inner iterative loop, and - Changes between the current synthesized output image of the generative part (G) generated using the current attention mask(s) and the previous synthesized output image of the generative part (G); - Generate one or more synthetic output images from noisy samples (z) drawn from the latent space using the generative part (G) of a generative adversarial network; and - Output the one or more synthesized output images as synthesis data.

11. A computer-readable medium (400) comprising temporary or non-temporary data (410) representing instructions arranged to cause a processor system to perform the computer-implemented method according to claim 10.

Images