Generating coagulated images

Abstract

Embodiments of the present disclosure relate to generative image condensation. Embodiments are disclosed for providing generative image condensation for unsupervised learning techniques that learn transformations of real data to improve image quality of GANs trained using that image data. In particular, in one or more embodiments, the disclosed systems and methods include generating, by a spatial transformer network, aligned real data sets for real images from unaligned real data sets, providing, by the spatial transformer network, the aligned real images to an adversarial discriminator network to determine whether the aligned real images are similar to aligned synthetic images generated by a generator network, and training, by a training manager, the spatial transformer network to learn updated transformations based on the determination of the adversarial discriminator network.

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to systems and methods for data preparation for training machine learning models. More specifically, one or more embodiments of the present disclosure relate to systems and methods for generative image condensation. BACKGROUND

[0002] Generative adversarial networks (GANs) are a machine learning technique that learns to generate new data that is similar to the data on which it is trained. For example, a GAN trained on cat-based images will generate new images of cats. In recent years, the image quality of GANs has improved rapidly. However, most GANs are typically trained on aligned visual data. When trained using an arbitrarily unaligned image distribution, the resulting GAN suffers substantial degradation in image quality, even for state-of-the-art models such as StyleGAN-v2. Manual alignment of images involves substantial cost and substantial human supervision. Additionally, manual alignment is typically performed heuristically (e.g., in whatever way appears to the person performing the alignment to be the best alignment). However, it is unclear whether this alignment selection is the best alignment for a given dataset.

[0003] These and other problems exist in training GANs. SUMMARY

[0004] Techniques / technology for generative image condensation are introduced herein that provide an unsupervised learning technique that learns a transformation to align real data to be used to train a generative adversarial network (GAN). In particular, in one or more embodiments, the disclosed systems and methods train a spatial transformer network, which is a neural network capable of performing geometric transformations on images, to align real image data with aligned synthetic image data. For example, a generative adversarial network trained on unaligned image data learns to generate synthetic images that are similar to the distribution of the unaligned image data on which the generative adversarial network is trained. GANs trained on unaligned data typically perform worse than GANs trained on aligned data, resulting in synthetic images that are visually undesirable. The input to a GAN is random noise sampled from some prior distribution. If the input random noise provided to a GAN trained on unaligned data is biased toward the mean of the prior distribution, then the GAN produces more faithful synthetic images at the cost of reduced diversity. One result of this reduced diversity is that synthetic images generated by the GAN in response to the biased input noise are aligned in a similar manner.

[0005] The embodiments utilize this to generate a synthetic alignment dataset. Using the synthetic alignment dataset, the spatial transformer network is adversarially trained to learn a transformation that makes the transformed real image data indistinguishable from the aligned synthetic data as determined by the discriminator network. Once trained, the spatial transformer network is able to align real image data used to train the GAN to produce an aligned real image dataset without the large amount of human supervision typically required to manually align the dataset. The aligned real image dataset can then be used to train a new GAN or fine-tune an existing GAN, resulting in a better performing GAN.

[0006] Additional features and advantages of the exemplary embodiments of the present disclosure will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of such exemplary embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0007] The detailed description is described with reference to the accompanying figures that are described below.

[0008] Figure 1 An example environment for performing generative image coalescence is illustrated in accordance with one or more embodiments;

[0009] Figure 2 An example of a generator network trained on unaligned data is illustrated in accordance with one or more embodiments;

[0010] Figure 3 An example of a change in generated images due to bias in input data is illustrated in accordance with one or more embodiments;

[0011] Figure 4 A diagram of a generative image coalescence system is illustrated in accordance with one or more embodiments;

[0012] Figure 5 An example of using a trained spatial transformer to align an unaligned dataset is illustrated in accordance with one or more embodiments;

[0013] Figure 6 An example of training a GAN using aligned data generated by a spatial transformer network is illustrated in accordance with one or more embodiments;

[0014] Figure 7 A diagram of a generative image coalescence system is illustrated in accordance with one or more embodiments;

[0015] Figure 8 A diagram of reversing the alignment of generated data is illustrated in accordance with one or more embodiments;

[0016] Figure 9 An example environment for implementing an image alignment system is illustrated in accordance with an embodiment;

[0017] Figure 10 FIGURE illustrates a schematic diagram of a generative image coalescing system, in accordance with one or more embodiments;

[0018] Figure 11 FIGURE illustrates a sequence diagram of generative image coalescing, in accordance with one or more embodiments;

[0019] Figure 12 FIGURE illustrates a sequence diagram of training a generative adversarial network using training data that has been aligned using generative image coalescing, in accordance with one or more embodiments;

[0020] Figure 13 FIGURE illustrates a flow diagram of a series of acts in a method of generative image coalescing, in accordance with one or more embodiments;

[0021] Figure 14 FIGURE illustrates a schematic diagram of an example environment in which a generative image coalescing system operates, in accordance with one or more embodiments; and

[0022] Figure 15 FIGURE illustrates a block diagram of an example computing device, in accordance with one or more embodiments. DETAILED DESCRIPTION

[0023] One or more embodiments of the present disclosure include a generative image coalescing system that provides an unsupervised learning technique that learns a transformation of real data to improve the image quality of a GAN trained using that image data. For example, a generator network that has been trained on unaligned data is used to generate synthetic images (e.g., "fake image data"). The input to the generator network used by the generator network when generating synthetic image data is biased such that the generator network produces images according to the dominant mode of the data on which it was trained. This produces a set of synthetic image data that has similar alignment that will vary according to the distribution of the dataset used to train the generator network (e.g., "generator model" or "generator"). A spatial transformation network takes unaligned real data and attempts to generate aligned data that appears to be synthetic image data generated by the generator network. A discriminator model (e.g., "adversarial discriminant network" or "discriminator") receives image data generated by the spatial transformation network and the generator network and attempts to determine which is real and which is synthetic. Based on that determination, the spatial transformation network is updated.

[0024] As discussed, GANs trained on misaligned data perform significantly worse than GANs trained on aligned data. In conventional systems, the process of aligning a training dataset is costly and requires a great deal of supervision. However, using the techniques described herein, a spatial transformer network is trained to automatically generate aligned data in an unsupervised manner. For example, after a spatial transformer network has been trained generally as discussed, the spatial transformer network has learned to align image data. Subsequently, the spatial transformer network receives a misaligned dataset and outputs a corresponding aligned (e.g., "condensed") dataset. The aligned dataset can then be used to train a new GAN or fine-tune an existing GAN, thereby improving the performance of the GAN without the costs associated with manually aligning a training dataset. Further, in some embodiments, the generator network of the newly trained GAN is used to train an improved spatial transformer network that produces better aligned datasets, which in turn enables the training of further improved GANs.

[0025] Conventional systems have incorporated spatial transformer networks in generative models. However, these conventional systems only use the spatial transformer network as a layer in the generator network. Additionally, other conventional systems have transformed real data to improve GAN performance. However, such systems require human input to perform the transformation and have not shown the ability to improve the performance of datasets with a large amount of diversity. Unlike past systems, embodiments apply the spatial transformer network to the training data to shrink the distribution of images that the generator must learn by automatically transforming the dataset. This applies to both simple and complex datasets, as the alignment of the dataset is learned by the spatial alignment network without additional user input. Instead, the alignment is learned based on the dominant patterns of the training data. Thus, complex datasets for which there is no obviously best alignment for a human reviewer are aligned based on the dominant patterns of the data, rather than heuristically or by other arbitrary decisions.

[0026] Terminology Definitions

[0027] As used herein, the term "digital visual medium" refers to digital data capable of producing a visual representation. For example, the term "digital visual medium" includes digital images and digital videos.

[0028] As used herein, the term "digital image" or "image" refers to any numeric symbol, picture, icon, or illustration. For example, the term "digital image" includes, but is not limited to, digital files having the following file extensions: JPG, TIFF, BMP, PNG, RAW, or PDF. Thus, a digital image includes a digital file of digital data or an image that can be displayed via a graphical user interface of a display of a computing device.

[0029] As used herein, the term "neural network" refers to a machine learning model that is tuned (e.g., trained) to coarsely estimate an unknown function. In some embodiments, the term neural network includes a model of interconnected neurons that communicate and learn to coarsely estimate a complex function and generate an output based on multiple inputs provided to the model. For example, the term neural network includes one or more machine learning algorithms. In some embodiments, the term neural network includes a deep convolutional neural network (i.e., "CNN"), such as a spatial transformer network (STN). Furthermore, a neural network is an algorithm (or set of algorithms) that implements deep learning techniques to model high-level abstractions in data. The following description and figures generally refer to a CNN that includes lower layers, higher layers, and a loss layer. In various embodiments, a spatial transformer network is a type of CNN capable of performing geometric transformations on an image.

[0030] As used herein, the term "adversarial learning" refers to machine learning algorithms (e.g., generative adversarial networks or adversarial discriminative neural networks) that learn adversarial learning models together. For example, a "generative adversarial neural network" (or simply "GAN") includes a generator network and a discriminator network (e.g., an "adversarial discriminative neural network"). Specifically, the term "adversarial learning" involves solving multiple learning tasks within the same model (e.g., sequentially or in parallel), while leveraging roles and constraints across tasks. In some embodiments, adversarial learning includes employing a loss function that is minimized relative to one model and maximized relative to an adversarial model. For example, image coagulation systems employ adversarial learning to minimize the adversarial loss of the STN and maximize the adversarial loss of the discriminator model. This is used to train the STN to transform the dataset into an output more closely similar to the generator network.

[0031] Figure 1 The illustration depicts an example environment for performing generative image condensation according to one or more embodiments. Figure 1 As illustrated, environment 100 includes a client device 102 with a digital design system 104. The digital design system 104 typically supports the creation, modification, sharing, and / or deletion of graphic content, including raster-based content. In one or more embodiments, the digital design system 104 is a design application, such as... In other embodiments, the digital design system 104 includes multiple applications, functions, and operations related to graphical content, such as application suites, web-based applications, and / or system operations on client device 102.

[0032] Further, the environment 100 includes a server device 106. The server device 106 generates, stores, receives, and / or transmits any type of data, including digital visual media. As shown, the server device 106 includes an artificial intelligence management system 108 that communicates with the digital design system 104 on the client device 102. For example, the artificial intelligence management system 108 uses its components to generate, train, and apply machine learning models. For example, the artificial intelligence management system 108 includes a generative image condensation system 110. As discussed further herein, the generative image condensation system 110 uses adversarial learning to learn to align (e.g., “condense”) data in an unsupervised manner. This includes training an STN to transform input real data to more closely resemble data generated by a generator network. Once the STN has been trained, the STN is used to align an unaligned training dataset to generate an aligned training dataset. The artificial intelligence management system 108 uses the aligned training dataset to train a new GAN or fine-tune an existing GAN.

[0033] The newly trained GAN (e.g., trained model 114) is deployed to the digital design system 104 or a digital design server system 116. In one or more embodiments, the server device 106 includes all or a portion of the digital design system 104, such as within the digital design server system 116. For example, when located in the server device 106, the digital design system 104 includes a portion of an application or software application running on the server device 106 that is downloadable to the client device 102 over one or more networks 112. For example, in some embodiments, the digital design system 104 includes a web-hosted application that allows the client device 102 to interact with content from the digital design server system 116 hosted at the server device 106. In this way, in some embodiments, the server device 106 uses the trained model in conjunction with digital visual media based on input received from a designer using the client device 102.

[0034] Figure 2 FIGURE 2 illustrates an example of a generator network trained on unaligned data, according to one or more embodiments. As discussed, a GAN generates new data that resembles the data on which it is based. For example, in embodiments of the CelebA dataset, the generator 200 has been trained on headshot image data. The training dataset includes paparazzi-style headshots of celebrities, such as available in the CelebA dataset. Because these images are captured in the real world under conditions where the photographer has limited control over the environment or subject of the photograph, the images include representations of people in various poses and at different angles. Rather than manually aligning the dataset, the unaligned dataset is used to train the GAN. Figure 2

[0035] ​To generate new data, the generator of the GAN trained on misaligned data is given an input z 202. The input (e.g., input latent vector) is typically random noise. In this example, the training data sets used to learn for the GAN are extremely challenging, especially because they are misaligned. This forces the GAN to model a much wider range of data distributions than would be the case with aligned training data. As shown in 204, the output of the GAN trained on misaligned data is typically visually poor. It is well known that GANs suffer from the effects of mode collapse (a situation where the generator produces samples from a (typically small) subset of modes from the underlying true distribution). However, even when trained on extremely challenging data distributions, GANs are able to learn at least some mode of the data. For example, the upper right output image is a relatively clear generated image. This mode or modes that the generator is able to produce fairly well will typically look similar. Generating a large amount of output data that looks similar is typically a sign that the GAN is not functioning well. However, embodiments exploit this problem to automatically generate aligned data that represents at least one mode of the training data.

[0036] Figure 3 An example of the variation in generated images due to biasing of the input data is illustrated in accordance with one or more embodiments. As shown in Figure 3 rather than generating images based on random inputs 202, the inputs 300 are biased to produce biased inputs z' 300. When the biased inputs 300 are provided to the generator 200, the generator 200 produces an output image data set 304 that is a significant visual improvement over the output images 306 produced by unbiased inputs 202. For example, in some embodiments, the inputs are biased using what is known as the "truncated trick." It is known that by biasing the input latent vector z towards the "high probability" regions of p z at the expense of diversity, the image quality of the samples is improved. This enables GANs trained on misaligned image data to generate higher quality output images at the expense of those output images that belong to the dominant modes that it has already learned. For example, for an input vector z, any values with magnitudes above a threshold are resampled until they are below the threshold. When the input is biased towards the mean of the distribution of the input random numbers, the highest fidelity output images are obtained. For example, where the values of the input distribution typically vary between -1 and 1, the highest fidelity results are obtained when the input is biased to be close to zero.

[0037] As mentioned, this fidelity comes at the expense of diversity. Thus, the output images are visually similar. However, unlike typical systems, this limited output range is helpful. For example, using this technique, a generator 200 trained on misaligned training data is able to produce an aligned generated dataset. Because the produced output images look visually similar, these output images are all aligned in a similar manner. For example, each image in the output image dataset 304 shows roughly the same portion of a person (e.g., a head close-up), with each image depicting the person in a similar pose. Thus, this output dataset is able to be used to train a STN to align real images in a similar manner. Although embodiments are discussed with respect to aligned data that is generated synthetically, in some embodiments, the aligned data is obtained via a different algorithm, such as a manual or supervised technique.

[0038] Figure 4 A diagram of a generative image condensation system is illustrated in accordance with one or more embodiments. As shown in Figure 4 The generative image condensation system 110 includes a generator 200 that has been trained on misaligned datasets, a spatial transformer 400, and a discriminator 402, as shown in

[0039] In the example of FIG. 4, the generator 200 produces truncated synthetic data 404 based on the biased input z' 302. As discussed, the truncated synthetic data 404 represents data that is generated by the generator with higher fidelity but lower diversity, due to the input z' being biased in a manner that tends to correspond to portions of the input space that have been best learned by the generator. Examples of truncated synthetic data 404 are described above at least with respect to the images 304 of Figure 4 Figure 3 The spatial transformer 400 receives misaligned data and applies one or more transformations to the misaligned data to make it look similar to the truncated synthetic data 404. As discussed, the misaligned data 401 is used to train the generator 200. Although embodiments are discussed with respect to the misaligned data 401 being generated synthetically, in some embodiments, the misaligned data is obtained via a different algorithm, such as a manual or supervised technique.

[0040] Figure 4 ​​The described example uses the same data set used to train the generator 200, but in various embodiments, the unaligned data is any data set that has a similar distribution to the data set used to train the generator 200. For example, if the generator 200 is trained based on cat images, then the unaligned data 401 should also include images of cats.

[0041] The discriminator 402 (e.g., an adversarial discriminative neural network) receives the transformed real data 406 and the truncated synthetic data 404 one at a time and makes a determination 408 as to whether its received data is real or synthetic.

[0042] A typical GAN includes a generator (G) and a discriminator (D). The Wasserstein GAN objective for such a system is:

[0043]

[0044] where x refers to an instance of unaligned real data, such as an image, and z ~ p z (z) is a sample from the prior distribution on which G is based. In this example, L adv is minimized with respect to G and L adv is maximized with respect to D. In this formulation, the distribution of real data is fixed.

[0045] However, embodiments incorporate a spatial transformer network T, such that the formulation becomes:

[0046]

[0047] In this example, T learns a transformation to make the distribution of x closer to G(z). For example, if G has been trained to generate individuals in a portrait pose using an unaligned data set, then when T processes a real image x of a person, the real image x is incentivized to zoom (e.g., crop) the input image so that it now features a portrait pose. Indeed, simply minimizing the above equation with respect to T would yield an extremely degenerate solution, e.g., T zooms a single pixel, making G’s job unimportant. Thus, we first aim to find a “good” T that implicitly minimizes the above equation, but is not degenerate. For example, in some embodiments, the STN is limited in the operations it is configured to perform to reduce the chance of finding a degenerate solution. In some embodiments, such limitations include a minimum crop size that the STN is configured to perform (e.g., as measured in pixels or relative to the size of the input image) or other limitations on the transformations that the STN is configured to perform. In some embodiments, these limitations manifest as soft regularization terms, rather than explicit limitations on the transformations.

[0048] Based on the determination 408 made by the discriminator 402, the spatial transformer 400 and the discriminator are updated (410). For example, the discriminator is trained over a number of epochs while the spatial transformer remains constant, and then the spatial transformer is trained over a number of epochs while the discriminator is being trained. This process continues until convergence is reached. This causes the spatial transformer to be trained such that it produces transformed real images T(x) that are indistinguishable from the generator’s truncated distribution by the discriminator. This problem is represented by the following GAN equation:

[0049]

[0050] As discussed, the generator G has already been trained. Thus, only the spatial transformer T and the discriminator D are trained. Since T belongs to a strictly limited class of functions (e.g., T is configured to perform a limited number of transformations on input images), the only way T can produce images that look similar to synthetic images is for T to learn transformations that align input images x with the same pose of the object produced by G in its truncated mode. Thus, the spatial transformer 400 is trained to output aligned images, where alignment corresponds to the alignment of the truncated mode output by the generator.

[0051] Figure 5 An example of using the trained spatial transformer to align an unaligned dataset is illustrated in accordance with one or more embodiments. As shown in Figure 5 The trained spatial transformer 500 can be used to align input data to generate an aligned dataset, as shown in

[0052] Figure 6 An example of training a GAN using aligned data generated by a spatial transformer network is illustrated in accordance with one or more embodiments. As shown in Figure 6As shown in FIG. 1, the generative image coalescing system 110 includes a generator 600, a trained spatial transformer 500 that has been trained on aligned data, and a discriminator 602. As discussed, GANs trained on aligned data produce synthetic images of higher fidelity than GANs trained on unaligned data. Aligned data is typically performed manually, which is both time consuming and expensive. Additionally, manual alignment is typically arbitrary and does not necessarily reflect the alignment of any of the patterns in the data set. However, as discussed above, by using a trained spatial transformer network 500 that has been trained on aligned data, the improved generator 600 is adversarially trained, which results in a generator that produces synthetic data of higher fidelity.

[0053] As Figure 6 As shown in FIG. 6, the generator 600 receives the trained spatial transformer 500 receives an input z 606. As discussed, the input to the generator includes random noise sampled from some prior distribution. Based on this input, the generator 600 generates synthetic data 610, which is provided to the discriminator 602. The trained spatial transformer 500 receives unaligned real data 608 and applies one or more transformations learned during training, as discussed above at least with respect to Figure 4 The discriminator 602 (e.g., an adversarial discriminative neural network) receives aligned real data 612 and synthetic data 610 one at a time and makes a determination 614 of whether the data it receives is real or synthetic (e.g., fake). Based on the determination 614 made by the discriminator 602, the generator 600 and discriminator are updated (616). For example, the discriminator is trained for a number of epochs while the spatial transformer remains constant, and then the spatial transformer is trained for a number of epochs while the discriminator is being trained. In some embodiments, alternative techniques are used to optimize the discriminator and generator during training. This process continues until convergence is reached.

[0054] Once convergence is reached, the generator 600 has been trained using aligned data and is capable of producing synthetic data of higher fidelity than a generator trained on unaligned data (e.g., generator 200). By using the trained spatial transformer 500 to align the training data, the improved generator 600 is obtained without the need for manual work in aligning the training data set. Additionally, as discussed above, the spatial transformer is trained to align data based on the dominant pattern(s) of the training data rather than arbitrary alignment decisions of a human user. Further, this enables data to be aligned that does not have an alignment that is easily observable by a human user.

[0055] Figure 7 FIG. 6 illustrates a diagram of a generative image coalescing system, in accordance with one or more embodiments. After training the spatial transformer, as discussed at least with respect to Figure 5As discussed, a spatial transformer is used to obtain an aligned training dataset. A new GAN is then trained based on the aligned training dataset, or an existing GAN is fine-tuned, resulting in a generator 700 (e.g., as discussed above at least with respect to Figure 6 As discussed, because the training dataset is aligned, the resulting generator 700 performs better than a previous generator 200 trained based on an unaligned dataset.

[0056] Using the generator 700 trained based on the aligned dataset, the generative image coalescing system 120 trains a new discriminator 702 and spatial transformer 704 similar to the training process discussed above with respect to Figure 5 For example, the spatial transformer 704 outputs transformed real data 706, and the generator 700 outputs truncated synthetic data 708. These are received by the discriminator 702, which makes a determination 710 whether a given received image is real or fake (e.g., synthetic). Based on the determination 710, the spatial transformer 704 and / or the discriminator 702 are updated (712). Because the truncated synthetic data 708 produced by the generator 700 is visually better than synthetic data produced by the previous generator 200, this training process will enable the spatial transformer to learn an improved transformation for aligning unaligned data. In various embodiments, this process is performed iteratively (e.g., training an improved spatial transformer, aligning a training dataset, training an improved generator, etc.) until a desired level of performance of the spatial transformer and / or generator is achieved.

[0057] In some embodiments, because the generator 700 performs better due to being trained based on aligned data, the input z' is not generally biased to be near the mean as previously discussed. For example, the threshold defining how the input vector is truncated is increased, allowing more diversity in the truncated synthetic data 708. Because of the improved performance of the generator (which is now performing better due to being trained using an aligned dataset), the associated fidelity loss is no longer as important. Because the truncated synthetic data 708 has more variability, the training process results in a more robust spatial transformer 704.

[0058] Additionally or alternatively, the restrictions on the transformation performed by the spatial transformer are also relaxed. Using the headshot example, once the generator is effectively modeling a headshot, the generator is extended to learn to model more information about the person (such as the upper body). This allows the restrictions on the spatial transformer (e.g., cropping size) to be extended, enabling it to be scaled down slowly until the alignment functionality is effectively removed.

[0059] Figure 8FIGURE illustrates a diagram of de-alignment of generated data, according to one or more embodiments. As discussed, the techniques described herein enable training of a GAN to generate aligned data. However, typical real data distributions are not aligned. Thus, the preferred output of the GAN should be unaligned in a similar manner so that it more closely mimics the distribution of the original training data set. Accordingly, embodiments enable synthetic data generated by the generator of the trained GAN to be de-aligned based on the transformations used to align the training data.

[0060] As Figure 8 As shown in FIGURE, the biased input z' 800 is received by a generator 802 trained on aligned training data. The generator then produces synthetic data 804. As discussed, because the generator 802 is trained on aligned training data, the generated synthetic data 804 will be aligned in a similar manner. The synthetic data 804 is provided to a reverse spatial transformer 806 to generate de-aligned synthetic data 808. For example, in some embodiments, the transformations performed by the spatial transformer used to align the training data are recorded and used to train the reverse spatial transformer 806. The reverse spatial transformer 806 then progressively de-aligns the synthetic data so that the distribution of the de-alignments performed is the inverse of the distribution of the transformations performed to align the training data. Alternatively, the reverse spatial transformer 806 uses the recorded transformations performed by the STN on the training data and applies the inverse transformations to each synthetic data item as it is generated. The sampling of the recorded transformations to determine the particular transformation to use on a given synthetic data item is controlled to ensure that the de-aligned synthetic data resembles the original alignment of the training data.

[0061] In some embodiments, rather than implementing a reverse spatial transformer 806 to generate de-aligned synthetic data 808, the generator 802 is trained to generate de-aligned synthetic data 808. For example, in one such embodiment, the spatial transformer (such as spatial transformer 704) is frozen (e.g., no longer trained) once it has learned the aligned images correctly. The aligned images using the (now frozen) spatial transformer are used to train a generator, such as generator 700. During the training process in this section, the spatial transformer's prediction of alignment is weakened to the identity function (which corresponds to no alignment). This causes the generator to learn to produce unaligned images that more closely resemble the aligned distribution of the original unaligned data set.

[0062] Figure 9 FIGURE illustrates an example environment in which an image alignment system is implemented, according to an embodiment. As Figure 9As illustrated in the middle, the environment 900 includes a client device 102 with a digital design system 104. The digital design system 104 generally supports the creation, modification, sharing, and / or deletion of graphical content including raster-based content. In one or more embodiments, the digital design system 104 is a design application, such as Adobe® Photoshop®, Adobe® Illustrator®, Adobe® InDesign®, Adobe® XD®, Adobe® Fresco®, and / or Adobe® Draw®. In other embodiments, the digital design system 104 includes a number of applications, functions, and operations related to graphical content, such as an application suite, web-based applications, and / or system operations on the client device 102.

[0063] Further, the environment 100 includes a server device 106. The server device 106 generates, stores, receives, and / or transmits any type of data, including digital visual media. As illustrated, the server device 106 includes all or a portion of the digital design system 104, such as within a digital design server system 116. For example, when located in the server device 106, the digital design system 104 includes a portion of an application or software application running on the server device 106 that is downloadable to the client device 102 over one or more networks 112. For example, in some embodiments, the digital design system 104 includes a web-hosted application that allows the client device 102 to interact with content from the digital design server system 116 hosted at the server device 106. In this way, in some embodiments, the server device 106 uses a trained model in conjunction with digital visual media based on input received from a designer using the client device 102.

[0064] As illustrated, the digital design system 104 and / or the digital design server system 116 includes an image alignment system 902. As discussed, embodiments train a spatial transformer network to align images based on one or more modes of a trained GAN. As discussed, the trained spatial transformer is then used to align training data to be used to train an improved GAN or to fine-tune an existing GAN. Additionally or alternatively, the trained spatial transformer is made available to align arbitrary image inputs for other digital design processes. For example, the trained spatial transformer 904 enables a user with several misaligned images to align the images before or after editing the images using the digital design system 104. In some embodiments, the image alignment system 902 includes multiple trained spatial transformers that have been trained based on different data distributions. In such embodiments, a user selects a trained spatial transformer based on the data that most closely resembles the data that the user wishes to align. Additionally or alternatively, the image alignment system uses each trained spatial transformer to align an input image and presents the resulting aligned images to the user who then selects the alignment to use.

[0065] Figure 10A schematic diagram of a generative image condensation system (such as the "generative image condensation system" described above) according to one or more embodiments is illustrated. As shown, the generative image condensation system 1002 is implemented as part of an artificial intelligence management system 1000 and stored in memory as program code, data structures, etc., and includes, but is not limited to, a training manager 1004, a spatial transformer network 1006, a generative adversarial network 1008, and a storage manager 1010. The generative adversarial network 1008 includes a generator network 1012 and a discriminator network 1014. The storage manager 1010 includes an unaligned real dataset 1016, an aligned training dataset 1018, and an aligned real dataset 1020.

[0066] like Figure 10 As illustrated, the generative image condensation system 1002 includes a training manager 1004. Specifically, the training manager 1004 trains a spatial transformer network to learn transformations applied to input images such that these input images are difficult to distinguish from synthetic images generated by a generator network when compared by a discriminator network. As discussed, the generator network has been trained to generate synthetic data based on truncated inputs, such that the synthetic data belongs to one or more dominant patterns of the training data upon which the previously trained generator was based. During training, the training manager 1004 adversarially trains the spatial transformer and discriminator using the synthetic data generated by the generator network. Thus, the spatial transformer learns to generate images that look similar to the synthetic images by learning transformations that align the input images with the same poses of objects in the synthetic images generated by the generator in its truncated patterns, making these images difficult for the discriminator model to distinguish.

[0067] As shown in the figure, the generative image condensation system 1002 includes a Spatial Transformer Network (STN) 1006, which, as discussed above, is a specialized neural network. As discussed, the STN 1006 is trained using a Generative Adversarial Network 1008. For example, a generator network 1012 generates synthetic images based on the data on which it was trained. Specifically, the generator network 1012 generates images belonging to its truncated patterns, which share general alignments. As discussed, this synthetic alignment data is used to train the STN 1006 to learn the transformations to be performed on real images so that the real images are difficult to distinguish from synthetic images, as determined by the discriminator network 1014. Once trained, the STN 1006 is used to generate a aligned real dataset 1020 from an unaligned real dataset 1016. As discussed, the aligned real dataset 1020 is then used to train an improved GAN.

[0068] As shown, the generative image condensation system 1002 includes a generative adversarial network 1008. As explained above, in one or more embodiments, the generative adversarial network 1008 includes a generator network 1012 and a discriminator network 1014. The generator network 1012 is adversarially trained using a discriminator network (e.g., an adversarial discriminative neural network) to learn to generate synthetic data that is indistinguishable from a training dataset by the discriminator network.

[0069] As Figure 10 illustrated in FIG. 10, the generative image condensation system 1002 also includes a storage manager 1010. The storage manager 1010 maintains data for the generative image condensation system 1002. The storage manager 1010 maintains any type, size, or kind of data as needed to perform the functions of the generative image condensation system 1002. As shown in FIG. 10, the storage manager 1010 includes an unaligned real dataset 1016. The unaligned real dataset 1016 includes one or more digital images, as discussed in additional detail above. Additionally, in various embodiments, the unaligned real dataset includes any type of digital visual media, including digital images and / or digital videos. Figure 10

[0070] As Figure 10 further illustrated in FIG. 10, the storage manager 1010 includes an aligned training dataset 1018. As discussed, the aligned training dataset 1018 includes synthetic data generated by the generator network 1012. The aligned training dataset is generated by the generator in response to input data that has been biased, such as using the truncation trick described above. This results in higher fidelity generated data at the expense of diversity. Thus, this truncated data is generally aligned to one or more dominant patterns of the data that the training generator is based on (e.g., the unaligned real dataset 1016).

[0071] As Figure 10 further illustrated in FIG. 10, the storage manager 1010 includes an aligned real dataset 1020. As discussed, once the STN 1006 has learned to transform image data, the unaligned real dataset 1016 is input to the STN that outputs the aligned real dataset 1020. The aligned real dataset 1020 is then used to train a new GAN or fine-tune an existing GAN to improve the fidelity of data generated by the newly trained GAN. As discussed, these improvements in performance are due to the GAN being trained on aligned data, which shrinks the distribution of the training data, making the patterns of the data easier for the GAN to learn.

[0072] ​In some embodiments, the storage manager 1010 manages various underlying storage devices coupled to the generative image condensation system 1002. The storage devices include one or more non-transitory computer-readable storage media, such as a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices. In some embodiments, the storage manager 1010 manages storage of input and output image data in remote storage services or systems. For example, in some embodiments, the storage manager communicates with storage services that provide block storage, object storage, database storage, etc. over one or more public and / or private networks (e.g., the Internet, a local area network, a storage area network, etc.).

[0073] Each of the components 1004-1010 of the generative image condensation system 1002 and their corresponding elements (as shown in Figure 10 communicate with each other using any suitable communication techniques. It should be recognized that, although the components 1004-1010 and their corresponding elements are shown in Figure 10 separate in Figure 10 , in some embodiments, any of the components 1004-1010 and their corresponding elements are combined into fewer components, such as into a single facility or module, divided into more components, or configured differently according to the needs of a particular embodiment.

[0074] The components 1004-1010 and their corresponding elements include software, hardware, or both. For example, the components 1004-1010 and their corresponding elements include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices. The computer-executable instructions of the generative image condensation system 1002, when executed by one or more processors, cause a client device and / or a server device to perform the methods described herein. Alternatively, the components 1004-1010 and their corresponding elements include hardware (such as a special-purpose processing device) to perform specific functions or groups of functions. Additionally, the components 1004-1010 and their corresponding elements include a combination of computer-executable instructions and hardware.

[0075] Furthermore, in various embodiments, components 1004 to 1010 of the generative image condensation system 1002 are implemented as one or more standalone applications, one or more modules of an application, one or more plugins, one or more library functions called by other applications, and / or cloud computing models. Therefore, in various embodiments, components 1004 to 1010 of the generative image condensation system 1002 are implemented as standalone applications, such as desktop or mobile applications. Furthermore, in various embodiments, components 1004 to 1010 of the generative image condensation system 1002 are implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, in various embodiments, components of the generative image condensation system 1002 are implemented in a suite of mobile device applications or "apps". For illustration, in various embodiments, components of the generative image condensation system 1002 are implemented in digital image or video editing applications (including but not limited to...) PRO or CREATIVE Implemented in software. “ADOBE”, “PHOTOSHOP”, “PREMIERE PRO” and “CREATIVE CLOUD” are registered trademarks or trademarks of Adobe Systems Incorporated in the U.S. and / or other countries.

[0076] Figure 11 A sequence diagram 1100 of generative image condensation according to one or more embodiments is illustrated. For example... Figure 11 As shown, learning to align (e.g., “condense”) data in an unsupervised manner using adversarial learning begins with digit 1, where real image data is provided to spatial transformer 1006 by storage manager 1010. For example, the real image data includes an unaligned training dataset used to train generator 1012. In digit 2, spatial transformer 1006 applies one or more transformations (e.g., geometric transformations, cropping, warping, etc.) to the real image data in an attempt to make the transformed real image data difficult to distinguish from the aligned synthetic data generated by generator 1012, as determined by discriminator 1014.

[0077] In numeral 3, the generator 1012 generates a synthetic image and provides the synthetic image to the discriminator 1014. As discussed, the generator 1012 generates a truncated synthetic image that belongs to one or more dominant modes of the data on which the generator is trained. This is performed by biasing the input vector provided to the generator 1012 from random noise towards the mean of the random noise distribution (e.g. close to zero). For example, a threshold is set such that any input vector whose magnitude exceeds the threshold is re-sampled until the magnitude does not exceed the threshold. This results in a synthetic image that has higher fidelity but lower diversity, which tends to produce images that align with one or more dominant modes of the data on which the generator is trained.

[0078] In numeral 4, the discriminator 1014 determines whether a given image it receives from the spatial transformer or generator is real or fake. Based on the determination, in numeral 5, the spatial transformer is updated. Because the spatial transformer is constrained to transformation operations, the spatial transformer learns to update the transformation it applies to input images to minimize the loss function. In numeral 6, the discriminator 1014 is updated based on the determination. As discussed, because the generator has already been trained, only the spatial transformer 1006 and the discriminator 1014 are trained during this process. In some embodiments, it is preferred to train one for a plurality of training epochs and then the other for another plurality of training epochs. Alternatively, in some embodiments, the training is performed by interleaving the steps of updating the generator with the steps of updating the spatial transformer. In some embodiments, other techniques are used to optimize the network while training. Once the network converges, the training is complete and the spatial transformer has learned how to best align data of the dominant modes of the training data set.

[0079] Figure 12 illustrates a sequence diagram 1200 of training a generative adversarial network using training data that has been aligned using a generative image condensation, in accordance with one or more embodiments, as Figure 12 As shown in numeral 1, training a GAN using aligned data begins with numeral 1, in which real training image data is provided to the spatial transformer 1006 by the storage manager 1010. For example, the real training image data includes an unaligned training data set used to train the generator 1012. In numeral 2, the spatial transformer 1006 applies one or more transformations (e.g. geometric transformations, cropping, warping, etc.) learned during training, as described above at least with respect to Figure 4 and Figure 11 and provides at least one aligned real training image to the discriminator 1014.

[0080] In numeral 3, the generator 1012 generates a synthetic image and provides the synthetic image to the discriminator 1014. As discussed, the generator 1012 generates a synthetic image based on input random noise. In numeral 4, the discriminator 1014 determines whether a given image it receives from the spatial transformer or generator is real or fake. Based on the determination, in numeral 5, the generator is updated. In numeral 6, the discriminator 1014 is updated based on the determination. As discussed, since the spatial transformer has already been trained, only the generator 1012 and the discriminator 1014 are trained during this process. In some embodiments, it is preferred to train one for a plurality of training epochs and then the other for another plurality of training epochs. Alternatively, in some embodiments, the training is performed by interleaving the steps of updating the generator with the steps of updating the spatial transformer. In some embodiments, other techniques are used to optimize the networks while training. Once the networks converge, the training is complete and the spatial transformer has learned how to best align the data of the dominant modes of the training dataset.

[0081] Figures 1 to 12 The corresponding text and examples provide several different systems and apparatuses that enable unsupervised learning of transformations to align real data, thereby improving the image quality of GANs trained using that image data. In addition to the foregoing, an embodiment is described in terms of a flowchart including acts and steps for implementing a method for a specific result. For example, Figure 13 A flowchart illustrating an example method according to one or more embodiments is shown. In various embodiments, the method is performed in relation to Figure 13 The described methods are performed with fewer or additional steps / acts, or the steps / acts are performed in a different order. Additionally, in some embodiments, the steps / acts described herein are repeated or performed in parallel with each other, or with different instances of the same or similar steps / acts.

[0082] Figure 13 A flowchart illustrating a series of acts in a method of generative image coagulation according to one or more embodiments is shown. In one or more embodiments, the method 1300 is performed in a digital media environment including a generative image coagulation system 1002. The method 1300 is intended to illustrate one or more methods according to the present disclosure and is not intended to limit potential embodiments. The method 1300 is described in relation to the generative image coagulation system 1002. Figure 13 Alternative embodiments include additional, fewer, or different steps than those expressly set forth in the method.

[0083] As Figure 13As illustrated in action 1302, the method 1300 includes an act of generating, by a spatial transformer network trained to generate aligned images, an aligned real dataset from an unaligned real dataset. As discussed, a spatial transformer network is a neural network trained to perform a transformation (e.g., a geometric transformation, a warp, a crop, etc.) on input data. The unaligned real dataset includes image data that includes representations of various objects and / or object types. In some embodiments, the unaligned real dataset includes a training dataset used to train a generative adversarial network to generate synthetic images that are similar to the training data.

[0084] As illustrated in action 1304, the method 1300 further includes an act of providing, by the spatial transformer network, at least one real image from the aligned real dataset to an adversarial discriminator network to determine whether the at least one real image is similar to a synthetic image generated by a generator network. As discussed, the adversarial discriminator network receives aligned real images from the spatial transformer network and synthetic data from the generator network. In some embodiments, the generator network is previously adversarially trained based on the unaligned dataset. In some embodiments, the synthetic image is an aligned synthetic image generated by the generator network upon receiving an input vector that has been biased to increase fidelity of the aligned synthetic image. Figure 13 As illustrated in action 1304, the method 1300 further includes an act of providing, by the spatial transformer network, at least one real image from the aligned real dataset to an adversarial discriminator network to determine whether the at least one real image is similar to a synthetic image generated by a generator network. As discussed, the adversarial discriminator network receives aligned real images from the spatial transformer network and synthetic data from the generator network. In some embodiments, the generator network is previously adversarially trained based on the unaligned dataset. In some embodiments, the synthetic image is an aligned synthetic image generated by the generator network upon receiving an input vector that has been biased to increase fidelity of the aligned synthetic image.

[0085] As illustrated in action 1306, the method 1300 further includes an act of training, by a training manager, the generator network to learn to generate synthetic images based on the determination of the adversarial discriminator network. In some embodiments, the spatial transformer network is trained by generating, by the spatial transformer network, an aligned real image for a real image from the unaligned real dataset, providing, by the spatial transformer network, the aligned real image to the adversarial discriminator network to determine whether the aligned real image is similar to an aligned synthetic image generated by the generator network, and training, by the training manager, the spatial transformer network to learn an updated transformation based on the determination of the adversarial discriminator network. As discussed, the image condensation system is trained according to a loss function that minimizes an adversarial loss of the spatial transformer network and maximizes an adversarial loss of the discriminator network. The spatial transformer network has a limited number of transformations configured to be performed on unaligned image data. Thus, during training, the spatial transformation network learns a transformation to make unaligned data indistinguishable from aligned synthetic data. In this way, the spatial transformation network learns how to best align image data in an unsupervised manner. In some embodiments, the updated transformation includes one or more geometric transformations or crops. Figure 13 As illustrated in action 1306, the method 1300 further includes an act of training, by a training manager, the generator network to learn to generate synthetic images based on the determination of the adversarial discriminator network. In some embodiments, the spatial transformer network is trained by generating, by the spatial transformer network, an aligned real image for a real image from the unaligned real dataset, providing, by the spatial transformer network, the aligned real image to the adversarial discriminator network to determine whether the aligned real image is similar to an aligned synthetic image generated by the generator network, and training, by the training manager, the spatial transformer network to learn an updated transformation based on the determination of the adversarial discriminator network. As discussed, the image condensation system is trained according to a loss function that minimizes an adversarial loss of the spatial transformer network and maximizes an adversarial loss of the discriminator network. The spatial transformer network has a limited number of transformations configured to be performed on unaligned image data. Thus, during training, the spatial transformation network learns a transformation to make unaligned data indistinguishable from aligned synthetic data. In this way, the spatial transformation network learns how to best align image data in an unsupervised manner. In some embodiments, the updated transformation includes one or more geometric transformations or crops.

[0086] In some embodiments, the method 1300 further includes an act by the spatial transformer network of applying the updated transformation learned during training to the unaligned real dataset to generate an aligned real dataset. As discussed, GANs trained on aligned data perform better (e.g., higher fidelity, more visually pleasing, etc.) than GANs trained on unaligned data. Because the spatial transformer network has learned how to best align data, the spatial transformer network is used to align the unaligned real dataset to generate an aligned real dataset to be used to train a new GAN or to fine-tune an existing GAN. For example, in some embodiments, the method 1300 further includes an act by the training manager of adversarially training a new generator network using the aligned real dataset.

[0087] In some embodiments, the newly trained GAN is used to train a new spatial transformer network, or to fine-tune an existing spatial transformer network. For example, in some embodiments, the method 1300 further includes acts of generating, by the spatial transformer network, a second aligned real image from the unaligned real dataset for a second real image, providing, by the spatial transformer network, the second aligned real image to the adversarial discriminator network to determine whether the second aligned real image resembles a new synthetic image generated by the generator network, wherein the new synthetic image is generated by the generator network using different input parameters than used to generate the aligned synthetic image, and training, by the training manager, the spatial transformer network to learn other updated transformations based on the determination by the adversarial discriminator network. In some embodiments, the method 1300 further includes an act by the artificial intelligence management system of deploying the new generator network to a digital design system.

[0088] Figure 14 FIG. 13 illustrates a schematic diagram of an example environment 1300 that is suitable to implement a generative image coalescing system 1002 in accordance with one or more embodiments. In various embodiments, the environment 1300 includes a service provider 1302 that includes one or more servers 1304 connected to a plurality of client devices 1306A through 1306N via one or more networks 1308. The client devices 1306A through 1306N, the one or more networks 1308, the service provider 1302, and the one or more servers 1304 communicate with each other or with other components using any communication platform and technology suitable for transmitting data and / or communication signals including any known communication techniques, devices, media, and protocols that support remote data communications (described in greater detail below with respect to FIG. 14). Figure 15 More detail regarding examples of the same are described below.

[0089] Although Figure 14The illustration shows a specific arrangement of client devices 1406A to 1406N, one or more networks 1408, service provider 1402, and one or more servers 1404, but various additional arrangements are possible. For example, bypassing network 1408, client devices 1406A to 1406N communicate directly with one or more servers 1404. Alternatively, client devices 1406A to 1406N communicate directly with each other. In some embodiments, service provider 1402 is a public cloud service provider that owns and operates its own infrastructure in one or more data centers and provides that infrastructure to customers and end users as needed to host applications on one or more servers 1404. Servers include one or more hardware servers (e.g., hosts), each of which securely partitions its computing resources (e.g., processors, memory, disk space, network bandwidth, etc.) among multiple customers, each of which hosts its own applications on one or more servers 1404. In some embodiments, the service provider is a private cloud provider that maintains cloud infrastructure for a single organization. One or more servers 1404 include one or more hardware servers in a similar manner, each hardware server having its own computing resources, which are divided among applications hosted by one or more servers for use by organizational members or their customers.

[0090] Similarly, although Figure 14 Environment 1400 is depicted as having various components, but in various embodiments, environment 1400 has additional or alternative components. For example, in some embodiments, environment 1400 is implemented on a single computing device having generative image condensation system 1002. In particular, generative image condensation system 1002 is implemented wholly or partially on client device 1402A. Alternatively, in some embodiments, environment 1400 is implemented in a distributed architecture across multiple computing devices.

[0091] like Figure 14 As illustrated, environment 1400 includes client devices 1406A to 1406N. Client devices 1406A to 1406N include any computing device. For example, client devices 1406A to 1406N include one or more personal computers, laptops, mobile devices, mobile phones, tablets, dedicated computers, TVs, or other computing devices, including those described below. Figure 15 The described computing device. Although in Figure 14 Three client devices are shown, but it should be understood that client devices 1406A to 1406N include any number of client devices (more or fewer than shown).

[0092] Moreover, such as Figure 14As illustrated in the middle, client devices 1406A-1406N and one or more servers 1404 communicate via one or more networks 1408. The one or more networks 1408 represent a single network or a collection of networks, such as the Internet, a corporate intranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a cellular network, a wide area network (WAN), a metropolitan area network (MAN), or a combination of two or more such networks. Thus, the one or more networks 1408 are any suitable network via which the client devices 1406A-1406N access the service provider 1402 and the servers 1404 (and vice versa). More detail is provided below regarding Figure 15 The one or more networks 1408 are discussed in more detail.

[0093] Further, the environment 1400 also includes one or more servers 1404. The one or more servers 1404 generate, store, receive, and transmit any type of data. For example, the servers 1404 receive data from a client device, such as the client device 1406A, and send data to another client device, such as the client devices 1402B and / or 1402N. The servers 1404 also transmit electronic messages between one or more users of the environment 1400. In one example embodiment, the servers 1404 are data servers. The servers 1404 also include a communication server or a web hosting server. Additional details regarding the servers 1404 are discussed below. Figure 15

[0094] As mentioned, in one or more embodiments, the one or more servers 1404 include or implement at least a portion of the generative image condensation system 1002. In particular, the generative image condensation system 1002 includes an application running on the one or more servers 1404, or in some embodiments, a portion of the generative image condensation system 1002 is downloaded from the one or more servers 1404. For example, the generative image condensation system 1002 includes a web hosting application that allows the client devices 1406A-1406N to interact with content hosted at the one or more servers 1404. To illustrate, in one or more embodiments of the environment 1400, one or more of the client devices 1406A-1406N access a web page supported by the one or more servers 1404. In particular, the client device 1406A runs a web application (e.g., a web browser) to allow a user to access, view, and / or interact with a web page or website hosted at the one or more servers 1404.

[0095] ​After a client device 1406A accesses a webpage or other web application hosted at one or more servers 1404, in one or more embodiments, the one or more servers 1404 enable a user to provide one or more digital images stored at the one or more servers 1404 (e.g., unaligned real data sets 916, such as a camera roll or personal photos of an individual). Upon receiving the unaligned real data sets, the one or more servers 1404 automatically perform the methods and processes described above to train an STN to learn to align the one or more digital images. The one or more servers 1404 provide the trained STN and / or trained GAN to the client device 1406A (e.g., by deploying the one or more models to an application, such as a digital design application on the client device).

[0096] As just described, in some embodiments, the generative image condensation system 1002 is implemented in whole or in part by the individual elements 1402-1408 of the environment 1400. It should be appreciated that while certain components of the generative image condensation system 1002 are described in the previous examples with respect to particular elements of the environment 1400, various alternative implementations are possible. For example, in one or more embodiments, the generative image condensation system 1002 is implemented on any of the client devices 1406A-N. Similarly, in one or more embodiments, the generative image condensation system 1002 is implemented on the one or more servers 1404. Moreover, different components and functionality of the generative image condensation system 1002 are implemented separately among the client devices 1406A-1406N, the one or more servers 1404, and the network 1408.

[0097] As discussed in greater detail below, embodiments of the present disclosure include or utilize special-purpose or general-purpose computer(s) that include computer hardware, such as, for example, one or more processors and system memory. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein are at least partially implemented as instructions embodied in a non-transitory computer- readable medium and executable by one or more computing devices, such as any of the media content access devices described herein. Generally, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.) and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

[0098] Computer-readable media includes any available media that is accessible by a general purpose or special purpose computer system. Computer-readable media that store computer- executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, embodiments of the present disclosure include at least two distinct computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0099] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSDs") (e.g., based on RAM), Flash memory, phase- change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0100] A "network" is defined as one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media include a network and / or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which are accessed by a general purpose or special purpose computer. Combinations of the above are also included within the scope of computer-readable media.

[0101] Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures carry on a modem transmission medium is automatically transferred at least in part to a non-transitory computer-readable storage medium (device) over which the computer executable instructions or data structures are to be executed. For example, computer-executable instructions can be downloaded to and / or stored in RAM, within a network interface module e.g., a "NIC," in order to carry out the activities specified in the computer-executable instructions. For example, from the network or data link, the computer-executable instructions or data structures carried in the transmission medium are detected by the NIC and then captured, buffered, and sent to the RAM of the computer system. Accordingly, it is to be understood that non-transitory computer-readable storage media (devices) are included in the computing system components that are in electrical communication, but not necessarily physically contact, with a general purpose or special purpose computer.

[0102] Computer-executable instructions include, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. In some embodiments, computer-executable instructions are executed on a general purpose computer to cause the general purpose computer to become a special purpose computer implementing elements of the present disclosure. Computer-executable instructions include, for example, binary, intermediate format instructions such as produced by a compiler, or code written in a higher level language that is translated into such intermediate format instructions. The foregoing descriptions of elements of the present disclosure have been presented for purposes of illustration and description. They are not intended to be exhaustive or to be limited to the precise forms disclosed. Various modifications and variations are possible in light of the above teachings. It is intended that the disclosed embodiments be broadly applied and practiced in various ways. Numerous specific details are described to provide a thorough understanding of the embodiments. However, in certain instances, well known methods, procedures, components, and circuits have not been described in detail so as not to obscure pertinent aspects of the embodiments.

[0103] Those skilled in the art will appreciate that embodiments of the present disclosure can be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. Embodiments of the present disclosure can also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0104] Some embodiments of the present disclosure are implemented in a cloud computing environment. In this description and the following claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing provides for the availability, scalability, and management of a shared pool of computing resources via the Internet for on-demand access by a community of consumers. In this description and the following claims, a "computing resource" can comprise, without limitation, computing capacity, infrastructure, platforms, storage, applications, and / or services offered through or provided by a cloud computing environment.

[0105] The cloud computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and the like. The cloud computing model also provides various service models such as, for example, Software as a Service ("SaaS"), Platform as a Service ("PaaS"), and Infrastructure as a Service ("IaaS"). In some embodiments, different deployment models are used to deploy the cloud computing model, such as private cloud, community cloud, public cloud, hybrid cloud, and the like. In this description and the following claims, a "cloud computing environment" is an environment in which cloud computing is employed.

[0106] Figure 15An example computing device 1500 configured to perform one or more of the processes described above is illustrated in block diagram form. It should be appreciated that one or more computing devices, such as computing device 1500, implement the generative image condensation system 1002. As shown, Figure 15 The computing device includes a processor 1502, a memory 1504, one or more communication interfaces 1506, a storage device 1508, and one or more I / O devices / interfaces 1510. In certain embodiments, the computing device 1500 includes fewer or more components than those shown in Figure 15 The components of the computing device 1500 shown in Figure 15 The components of the computing device 1500 shown in

[0107] In particular embodiments, the processor(s) 1502 include hardware, such as a collection of logic gates, to execute instructions. As an example and not by way of limitation, to execute instructions, the processor(s) 1502 fetch (or obtain), decode, and execute instructions from internal registers, an internal cache, the memory 1504, or the storage device 1508. In various embodiments, the processor(s) 1502 include one or more central processing units (CPUs), graphical processing units (GPUs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoC), or other processors, or combinations of processors.

[0108] The computing device 1500 includes the memory 1504 coupled to the processor(s) 1502. The memory 1504 is used for storing data, metadata, and programs for execution by the processor(s). The memory 1504 includes one or more of volatile memory and non-volatile memory such as random access memory (“RAM”), read-only memory (“ROM”), solid state drives (“SSDs”), flash memory, phase change memory (“PCM”), or other types of data storage. The memory 1504 is internal or distributed.

[0109] The computing device 1500 also includes one or more communication interfaces 1506. In various embodiments, the communication interface 1506 includes hardware, software, or both. The communication interface 1506 provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices 1500 or one or more networks. By way of example, and not limitation, communication interface 1506 includes a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network; or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. The computing device 1500 also includes a bus 1512. The bus 1512 includes the hardware, software, or both that couples components of the computing device 1500 to each other.

[0110] The computing device 1500 includes a storage device 1508 that includes storage for storing data or instructions. As an example and not by way of limitation, the storage device 1508 includes a non-transitory storage medium described above. The storage device 1508 includes a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive or a combination of these or other storage devices.

[0111] The computing device 1500 also includes one or more input or output (“I / O”) devices / interfaces 1510 provided to allow a user to provide input to, receive output from, and otherwise transfer data to and from the computing device 1500. These I / O devices / interfaces 1510 include, for example, a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other I / O devices, or a combination of such I / O devices / interfaces 1510. The touch screen is activated by a touch of a finger or a stylus.

[0112] The I / O devices / interfaces 1510 include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (for example, a display screen), one or more output drivers (for example, display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O devices / interfaces 1510 are configured to provide graphical data to a display for presentation to a user. The graphical data represents one or more graphical user interfaces and / or any other graphical content as served by a particular implementation.

[0113] In the foregoing specification, embodiments have been described with reference to specific exemplary embodiments thereof. Various embodiments have been described with reference to details that can vary depending on implementation, and the description above is illustrative only. The embodiments were chosen and described in order to provide what is believed to be the most useful and readily understood description of the principles and theoretical explanations for the various embodiments.

[0114] Embodiments take other specific forms in practice. The described embodiments are to be considered in all respects only as illustrative and not restrictive in character. For example, in some embodiments, the methods described herein are performed with fewer or more steps / actions, or in a different order. Additionally, steps / actions described herein are repeated or performed in parallel with each other, or with different instances of the same or similar steps / actions. Accordingly, the scope of the application is indicated by the appended claims, rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

[0115] In various embodiments described above, unless specifically stated otherwise, disjunctive language such as the phrase “at least one of A, B, or C,” is to be understood as indicating that A, B, or C, or any combination thereof (e.g., A, B, and / or C) can be employed. As such, the disjunctive language is not intended to, nor should it be construed to, imply that a given embodiment requires at least one of A, at least one of B, or at least one of C each individually.

Claims

1. A computer-implemented method comprising: generating, by a spatial transformer network, an aligned real dataset from a misaligned real dataset, wherein the spatial transformer network is trained to learn a transformation to generate an aligned image based on comparing, using an adversarial discriminative network, the generated aligned image generated by the spatial transformer network and an aligned synthetic image generated by a generator network; providing, by the spatial transformer network, at least one aligned real image from the aligned real dataset to the adversarial discriminative network to determine whether the at least one aligned real image resembles a synthetic image generated by the generator network; and training, by a training manager, the generator network to learn to generate a synthetic image based on the determination by the adversarial discriminative network.

2. The computer-implemented method of claim 1, wherein the spatial transformer network is trained by: generating, by the spatial transformer network, an aligned real image for a real image from the misaligned real dataset; providing, by the spatial transformer network, the aligned real image to the adversarial discriminative network to determine whether the aligned real image resembles the aligned synthetic image generated by the generator network; and training, by the training manager, the spatial transformer network to learn an updated transformation based on the determination by the adversarial discriminative network.

3. The computer-implemented method of claim 2, wherein the aligned synthetic image is generated by the generator network after receiving an input vector that has been biased to increase fidelity of the aligned synthetic image.

4. The computer-implemented method of claim 2, further comprising: applying, by the spatial transformer network, the updated transformation learned during training to the misaligned real dataset to generate the aligned real dataset.

5. The computer-implemented method of claim 2, wherein the updated transformation comprises one or more geometric transformations or cropping.

6. The computer-implemented method of claim 1, further comprising: generating, by the spatial transformer network, a second aligned real image for a second real image from the misaligned real dataset; providing, by the spatial transformer network, the second aligned real image to the adversarial discriminative network to determine whether the second aligned real image resembles a new synthetic image generated by the generator network, wherein the new synthetic image is generated by the generator network using different input parameters than used to generate the aligned synthetic image; and training, by the training manager, the spatial transformer network to learn other updated transformations based on the determination by the adversarial discriminative network.

7. The computer-implemented method of claim 1, further comprising: deploying, by an artificial intelligence management system, the generator network to a digital design system.

8. A system comprising: a computing device implementing a generative image condensation system, the generative image condensation system comprising: ​ ​ ​ a spatial transformer network to: generate an aligned real dataset from an unaligned real dataset, wherein the spatial transformer network is trained to learn a transformation to generate an aligned image based on comparing a generated aligned image with an aligned synthetic image generated by a generator network using an adversarial discriminative network, the generated aligned image generated by the spatial transformer network; and provide at least one aligned real image from the aligned real dataset to the adversarial discriminative network to determine whether the at least one aligned real image resembles a synthetic image generated by the generator network; and a training manager to train the generator network to learn to generate a synthetic image based on the determination by the adversarial discriminative network.

9. The system of claim 8, wherein: the spatial transformer network is further to: generate an aligned real image for a real image from the unaligned real dataset; and provide the aligned real image to the adversarial discriminative network to determine whether the aligned real image resembles the aligned synthetic image generated by the generator network; and the training manager is further to train the spatial transformer network to learn an updated transformation based on the determination by the adversarial discriminative network.

10. The system of claim 9, wherein the aligned synthetic image is generated by the generator network upon receiving an input vector that has been biased to increase fidelity of the aligned synthetic image.

11. The system of claim 9, wherein the spatial transformer network is further to: apply the updated transformation learned during training to the unaligned real dataset to generate an aligned real dataset.

12. The system of claim 9, wherein the updated transformation comprises one or more geometric transformations or cropping.

13. The system of claim 8, further comprising: wherein the spatial transformer network is further to: generate a second aligned real image for a second real image from the unaligned real dataset; and provide the second aligned real image to the adversarial discriminative network to determine whether the second aligned real image resembles a new synthetic image generated by the generator network, wherein the new synthetic image is generated by the generator network using different input parameters than used to generate the aligned synthetic image; and wherein the training manager is further to train the spatial transformer network to learn other updated transformations based on the determination by the adversarial discriminative network.

14. The system of claim 8, further comprising: an artificial intelligence management system to deploy the generator network to a digital design system.

15. A system comprising: Apparatus for generating, by a spatial transformer network, an aligned real dataset from a misaligned real dataset, wherein the spatial transformer network is trained to learn a transformation to generate aligned images based on comparing generated aligned images with aligned synthetic images generated by a generator network using an adversarial discriminator network, the generated aligned images generated by the spatial transformer network; Apparatus for providing at least one aligned real image from the aligned real dataset to the adversarial discriminator network to determine whether the at least one aligned real image is similar to a synthetic image generated by the generator network; and Apparatus for training the generator network to learn to generate synthetic images based on the determination by the adversarial discriminator network.

16. The system of claim 15, further comprising: Apparatus for generating an aligned real image for a real image from the misaligned real dataset; Apparatus for providing the aligned real image to the adversarial discriminator network to determine whether the aligned real image is similar to the aligned synthetic image generated by the generator network; and Apparatus for training the spatial transformer network to learn an updated transformation based on the determination by the adversarial discriminator network.

17. The system of claim 16, wherein the aligned synthetic image is generated by the generator network upon receiving an input vector that has been biased to increase fidelity of the aligned synthetic image.

18. The system of claim 16, further comprising: Apparatus for applying the updated transformation learned during training to the misaligned real dataset to generate an aligned real dataset.

19. The system of claim 15, further comprising: Apparatus for generating a second aligned real image for a second real image from the misaligned real dataset; Apparatus for providing the second aligned real image to the adversarial discriminator network to determine whether the second aligned real image is similar to a new synthetic image generated by the generator network, wherein the new synthetic image is generated by the generator network using different input parameters than used to generate the aligned synthetic image; and Apparatus for training the spatial transformer network to learn other updated transformations based on the determination by the adversarial discriminator network.

20. The system of claim 15, further comprising: Apparatus for deploying the generator network to a digital design system.

Images