Controlling neural networks through intermediate latent spaces

Abstract

Embodiments of the present disclosure relate to controlling a neural network through intermediate latent space. A generative neural network control system controls a generative neural network by modifying an intermediate latent space in the generative neural network. The generative neural network includes multiple layers that each generate a set of activation values. An initial layer (and optionally additional layers) receives an input latent vector, and a final layer outputs an image that is generated based on the input latent vector. Data that is input to each layer (except the initial layer) is referred to as data in the intermediate latent space. The data in the intermediate latent space includes activation values (e.g., generated by a previous layer, or modified using various techniques), and optionally includes a latent vector. The generative neural network control system modifies the intermediate latent space to achieve various different effects when generating new images.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to neural networks, and more particularly to controlling neural networks through intermediate latent space. BACKGROUND

[0002] With the development of computer technology, various uses and applications of computers have gradually been formed. One recent development is generative neural networks that can generate images from initial inputs that are often random, such as generative adversarial networks (GANs). These generative neural networks can be provided as part of various digital content creation applications and provide many benefits, such as generating images that can be difficult to separate from image regions captured by a camera.

[0003] Despite these benefits, generative neural networks are not without their problems. One such problem is that it can be difficult to control the output of a generative neural network. Conventional attempts to control the output of a generative neural network include providing a class vector indicating a class of image to be generated as part of the input to an initial layer of the generative neural network, providing additional data to be used as at least part of the input to the initial layer, and the like. However, these attempts have produced limited results. Providing a class vector allows control of what class of image is generated (e.g., dog, cat, landscape), but does not provide any other control (e.g., direction the dog is looking, background of the image). Providing additional data to be used as at least part of the input to the initial layer can be problematic because obtaining the additional data can be difficult and the results of using such additional data are limited. For example, a user can obtain an additional image of a dog looking from a particular direction (e.g., looking at the side of the dog) and provide the additional image as part of the input to an initial layer of the generative neural network, which can generate a dog that looks slightly to the side as in the additional image.

[0004] Accordingly, conventional solutions provide limited control over images generated by generative neural networks, resulting in user dissatisfaction and disappointment with their computers and image generation systems. SUMMARY

[0005] To reduce drawing of conventional image generation systems, a generative neural network control system is described to control a neural network through an intermediate latent space. In one or more implementations, first data for a generator network of a generative adversarial network (GAN) is received, the generator network including a plurality of layers. The plurality of layers includes an initial layer, a first layer, and a second layer. An input selection for a first effect of a new image generated by the generator network is received. Second data is generated by modifying the first data based on the input selection. The modifying includes applying a decomposition vector to activation values generated by the first layer. The second data is provided to the second layer, the second layer being a layer in the generator network that is later than the first layer. Using the generator network, the new image with the first effect is generated with the second data.

[0006] In one or more implementations, first data for a generator network of a generative adversarial network (GAN) is received, the generator network including a plurality of layers. The plurality of layers includes an initial layer, a first layer, and a second layer. An input selection for a first effect of a new image generated by the generator network is received. Second data is generated based on the first data and the input selection. The second data is generated based on modifying activation values generated by the first layer and modifying a latent vector input to the initial layer. The second data is provided to the second layer, the second layer being a layer in the generator network that is later than the first layer. Using the generator network, the new image with the first effect is generated with the second data.

[0007] This summary introduces in a simplified form the concept selected for further description in the detailed description below. Thus, this summary does not purport to identify key or essential features of the claimed subject matter, nor is it intended to operate as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0008] The detailed description is described with reference to the accompanying figures. The entities represented in the figures can indicate one or more entities, and thus, singular or plural forms of the entities can be referred to interchangeably in the discussion.

[0009] Figure 1 is an illustration of a digital media environment in an example implementation that is operable to employ control of a neural network through an intermediate latent space as described herein.

[0010] Figure 2 is an illustration of an example architecture of a generative neural network control system.

[0011] Figure 3 illustrates an example generative neural network.

[0012] Figure 4 illustrates another example generative neural network.

[0013] Figure 5 An example of a translation effect is illustrated.

[0014] Figure 6 An example of a camera motion effect is illustrated.

[0015] Figure 7 An example of a rotation effect of a camera is illustrated.

[0016] Figure 8 An example of modifying activation values of a layer of a generative neural network is illustrated.

[0017] Figure 9 Another example generative neural network is illustrated.

[0018] Figure 10 Another example generative neural network is illustrated.

[0019] Figure 11 Another example generative neural network is illustrated.

[0020] Figure 12 An example of removing artifacts from an image is illustrated.

[0021] Figure 13 An example of changing a style of a generated image is illustrated.

[0022] Figure 14 Another example generative neural network is illustrated.

[0023] Figure 15 An example of generating component images is illustrated.

[0024] Figure 16 An example of combining first image activation values with second image activation values is illustrated.

[0025] Figure 17 Another example of generating a composite image is illustrated.

[0026] Figure 18 Another example of generating a composite image is illustrated.

[0027] Figure 19 is a flowchart depicting a process in an example implementation of controlling a neural network through an intermediate latent space.

[0028] Figure 20 An example system including various components of an example device that can be implemented as described with reference to Figures 1-19 Any type of computing device described and / or utilized to implement various aspects of the technology described herein. DETAILED DESCRIPTION

[0029] SUMMARY

[0030] Current attempts to control images generated by generative neural networks, such as GANs, involve controlling the input to the initial layers of the generative neural network. For example, these attempts are limited to identifying a class of images to be generated (e.g., dogs, cats, landscapes) or providing additional data (e.g., additional images of a dog looking in a particular direction) to be used as at least a portion of the input to the initial layers. However, these attempts provide minimal control over the images generated by the generative neural network and can be problematic due to the need to obtain additional data.

[0031] To overcome these problems, discussed herein are techniques for controlling neural networks through intermediate latent spaces. Generally, in accordance with one or more implementations, a generative neural network includes multiple layers that each generate a set of activation values. An initial layer, also referred to as an input layer, and optionally additional layers, receive an input latent vector. A final layer, also referred to as an output layer, outputs an image that is generated based on the input latent vector. Data that is input to each layer, except the initial layer, is referred to as data in an intermediate latent space. Data in the intermediate latent space includes activation values (e.g., generated by a previous layer or modified using various techniques) and optionally includes latent vectors. The techniques discussed herein modify the intermediate latent space to achieve various different effects when generating new images.

[0032] The intermediate latent space is modified by performing a stitching or blending operation in a spatial region of a layer or across an entire given layer. The stitching or blending can include interpolating values, selecting from different portions of input values, adding or subtracting values (e.g., based on a decomposition as discussed below), etc. By performing the stitching and blending operation in a spatial region or across an entire given layer, a user can affect the output in various ways. Further, by selecting the layer to edit, a user can control the global or local nature of the changes.

[0033] In one or more implementations, a decomposition technique (e.g., principal component analysis) is used to generate a set of decomposition vectors (e.g., eigenvectors) for a particular intermediate latent space. The decomposition vectors are generated by providing a particular number (e.g., on the order of 10,000 to 100,000) of different latent vectors to the generative neural network and generating a set of activation values from a particular layer of the generative neural network. The particular layer can vary based on whether a stronger global effect on the generated image is desired or a stronger local effect on the generated image is desired.

[0034] Using decomposition, each set of activation values generated by a particular layer is unfolded (transformed) into activation vectors, and these activation vectors are analyzed using any of a variety of different public or proprietary techniques to generate a plurality of decomposition vectors. For example, each of these activation values represents a point in a multi-dimensional space (equal to the number of dimensions each vector has). These points create a point cloud in the multi-dimensional space, and various techniques can be used to determine the direction in which this point cloud is at maximum extension (the direction in which the point cloud has maximum variance). The vectors indicating these directions of maximum extension (variance) are the generated decomposition vectors. For example, principal component analysis can be performed on the activation values to generate a plurality of eigen-vectors.

[0035] A small number of decomposition vectors (principal components), such as on the order of tens or hundreds of magnitudes, explain most of the variance of a given layer. These decomposition vectors map well to semantic properties of the output image, such as object locations and orientations, camera poses, etc. The techniques discussed herein use the decomposition vectors to control the geometric properties of the generated images while keeping the style consistent. Additionally or alternatively, output simplification or artifact removal can be performed by projecting the activations onto a small number of principal components. The style can also be optionally changed by changing the latent vectors input to the intermediate latent space.

[0036] In one or more implementations, each layer of the generative neural network receives a latent vector as input. In this case, as part of the decomposition technique, the latent vector is appended to the activation vector (concatenating the activation vector and the latent vector), and the decomposition vectors are generated based on the appended set of activation values and latent vectors.

[0037] In one or more implementations, a decomposition vector is used to modify the activation values generated by one layer of the generative neural network, and the modified activation values are provided as input to the next layer of the generative neural network. Each decomposition vector corresponds to a different effect on the images generated by the generative neural network. For example, modification of the activation values using different ones of the decomposition vectors can allow a user to produce different effects in the images generated by the generative neural network, such as changing the pose of a subject in the image and left and right and up and down camera translations, changing the camera position and left and right and up and down camera rotations, zooming in and out, removal of artifacts, simplification of the scene, etc. The activation values can be modified based on the decomposition vectors in various ways, such as by spreading the activation values into an activation vector. The decomposition vector can be added to the activation vector, subtracted from the activation vector, projected onto the activation vector, etc. The modified activation values are then converted to a form for being input to the next layer (e.g., to a matrix form), and provided to the next layer of the generative neural network. The modified activation values can replace all of the activation values input to the next layer, or can replace only some of the activation values input to the next layer.

[0038] Additionally or alternatively, in the case where the latent vector is input to multiple layers of the generative neural network, a decomposition vector is used to modify the latent vector input to one or more layers of the generative neural network. Each decomposition vector corresponds to a different effect on the images generated by the generative neural network as discussed above. The latent vector can be modified based on the decomposition vector in various ways, such as by adding or subtracting the decomposition vector from the latent vector, and then providing the modified latent vector to the next layer of the generative neural network.

[0039] Different layers of the generative neural network correspond to different quantities through which modification of the activation values and the latent vector have global effects or local effects on the images. Modifications made in earlier layers have more global effects on the generated images, while modifications made in later layers have more local effects on the images. Global effects refer to modifications that affect a large number of images, while local effects refer to modifications that affect a small number of images. Thus, by selecting the appropriate layer, a user can control the degree of global or local effects produced on the images.

[0040] In one or more implementations, the activation values or latent vector input is modified after an initial image is generated by the generative neural network. This allows a user to view the initial image, provide input requesting that one or more effects be performed, and have the generative neural network generate a new image with the requested effects. Additionally or alternatively, the activation values or latent vector input can be modified as part of generating the initial image by the generative neural network. For example, a user can provide input requesting that one or more effects be performed, and the initial image generated by the generative neural network has the requested effects.

[0041] Additionally or alternatively, the intermediate latent space can be modified to change the style of the image generated by the generative neural network, such as changing the coloring of the image or the background of the image. The style of the generated image can be changed, for example, by applying the initial latent vector to the first one or more layers of the generative neural network but applying a different latent vector (e.g., generated randomly or pseudorandomly) to the second one or more later layers of the generative neural network. Thus, different latent vectors are applied to different layers of the generative neural network, thereby allowing a user to control the style of the generated image.

[0042] Additionally or alternatively, the intermediate latent space can be modified to have the generative neural network generate an image that is a composite of two other images. Such a composite image is generated by having the generative neural network generate two images, also referred to as source images. This allows a user to control the output of the generative neural network to enable the generation of high-quality composite or blended images. The activation values from the two source images for a particular layer of the generative neural network can be combined in various ways, such as by stitching the activation values from the different source images (using different portions of the activation values), by blending (e.g., interpolating) the activation values from the different source images, and so on, to generate a composite or blended image based on the two source images.

[0043] The techniques discussed herein allow for various controls over images generated by a generative neural network that were previously not possible. By manipulating the data in the intermediate latent space, various different effects on the generated images can be achieved. Additionally, the techniques discussed herein allow for simple controls over images generated by a generative neural network. Simple and direct input (e.g., user requests) for different effects can be received. For example, an input request to zoom in or out (e.g., a selection of a “zoom in” or “zoom out” button, a selection of a “zoom out by 3x” or “zoom in by 2x” button) can indicate a requested effect without requiring access to an image zoomed in or out by the requested amount. This alleviates the need for a user to provide additional data or additional supervision in order to control the output of the generative neural network.

[0044] Additionally, the techniques discussed herein allow a generative neural network to quickly generate images. An input indicating a requested effect can be received by the generative neural network and used immediately to generate an image having the requested effect. No additional training of the generative neural network is required in order to generate a desired effect. Additionally, the techniques discussed herein employ simple modifications to the generative neural network coefficients (activation values) or latent vectors input to the intermediate latent space. This alleviates the need to significantly modify or alter the generative neural network in order to achieve effects requested by a user.

[0045] Additionally, the techniques discussed herein recognize that a generative neural network models the concept of style in generated images. By providing suitable modifications to the intermediate latent space (e.g., altering latent vectors input to the intermediate latent space), the techniques discussed herein allow a user to control the style of generated images.

[0046] In the following discussion, an example environment is described that can employ the techniques described herein. Example processes that can be performed in the example environment, as well as other environments, are also described. Thus, the example processes are not limited to the example environment, and the example environment is not limited to performance of the example processes.

[0047] Example Environment

[0048] Figure 1 is an illustration of a digital media environment 100 in an example implementation that is operable to employ intermediate latent space control of neural networks as described herein. The illustrated environment 100 includes a computing device 102 that can be configured in a variety of ways. The computing device 102 may, for example, be configured as a mobile device (e.g., assume a handheld configuration, such as a tablet or mobile phone), a wearable device (e.g., an augmented reality or virtual reality headset, a smart watch), a laptop computer, a desktop computer, a game console, a car computer, etc. Thus, the computing device 102 can range in scope from complete resource devices (e.g., personal computers, game consoles) with substantial memory and processor resources to a low-resource device (e.g., a mobile device) with minimal memory and / or processing resources. Additionally, although a single computing device 102 is shown, the computing device 102 can represent multiple different devices, such as a plurality of servers used by an enterprise to perform operations "in the cloud" as described in Figure 20

[0049] ​The computing device 102 is illustrated as including a generative neural network 104 trained to generate images and a generative neural network control system 106. The generative neural network 104 can be any of various different types of neural networks, such as a generative adversarial network (GAN). Note that a GAN typically includes a generator network and a discriminator network. After training the GAN, the discriminator network is no longer needed. Thus, the generative neural network 104 can be, for example, the generator network of a GAN.

[0050] The generative neural network 104 under the control of the generative neural network control system 106 processes and transforms digital content 108, which is illustrated as being maintained in a storage 110 of the computing device 102. Such processing includes creating the digital content 108 (including by manipulating an intermediate latent space as discussed herein) and presenting the digital content 108 in a user interface, e.g., by a display device. The storage 110 can be any of various different types of storage, such as random access memory (RAM), flash memory, solid state drives, disk drives, etc. Although illustrated as being implemented locally at the computing device 102, functionality of the generative neural network 104 or the generative neural network control system 106 can also be implemented in whole or in part via functionality available over a network 112, such as part of a web service or "in the cloud."

[0051] The generative neural network 104 is composed of multiple layers, with activation values being generated by one layer and passed to successive layers. The generative neural network control system 106 applies various controls on images generated by the generative neural network 104 by modifying these activation values in various ways as discussed in more detail below. For example, the generative neural network 104 receives an initial latent vector 114, such as a random value. A class vector indicating a class of image to be generated (e.g., dog, cat, man, woman, car, landscape, etc.) can be included as part of the initial latent vector 114 or can be provided separately to the generative neural network 104 (e.g., input to an initial layer of the generative neural network 104 and optionally to additional layers). By controlling the generative neural network 104, the generative neural network control system 106 allows the neural network 104 to generate different images from the initial latent vector 114. For example, the generative neural network control system 106 can manipulate an intermediate latent space of the generative neural network 104 to control the generative neural network 104 to generate an image of a dog that appears to have been captured from a direction looking directly at the dog's face (as illustrated by image 116) or an image of a dog that appears to have been captured from a direction looking at the side of the dog's face (as illustrated by image 118).

[0052] Note that although the generative neural network control system 106 is illustrated as being implemented locally at the computing device 102, functionality of the generative neural network control system 106 can also be implemented in whole or in part via functionality available over a network 112, such as part of a web service or "in the cloud." Figure 1The generative neural network control system 106 is illustrated as a standalone system, but can be implemented as part of another program or system. For example, the generative neural network control system 106 can be implemented as part of a digital content editing or creation system, as part of an operating system, etc.

[0053] Although illustrated as a neural network, the generative neural network 104 can be a generative machine learning system implemented using a variety of different machine learning techniques. Discussion herein regarding generative neural networks similarly refers to other generative machine learning systems. A machine learning system refers to a computer representation that can be tuned (e.g., trained) based on input to roughly estimate unknown functions. In particular, a machine learning system can include a system that learns known data and makes predictions thereon using algorithms to learn to generate outputs that reflect patterns and attributes of the known data by analyzing the known data. For example, a machine learning system can include decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, artificial neural networks, deep learning, etc.

[0054] Generally, the functionality, features, and concepts described with respect to the above and below examples can be employed in the context of the example systems and processes described herein. Further, the functionality, features, and concepts described with respect to different figures and examples in this document can be interchanged and are not limited to implementation in the context of a particular figure or process. Moreover, blocks associated with different representative processes and corresponding figures herein can be applied together and / or combined in different ways. Accordingly, the individual functionality, features, and concepts described with respect to different example environments, devices, components, figures, and processes herein can be used in any suitable combination, and are not limited to the particular combinations represented by the examples listed in this specification.

[0055] Neural Network Control System Architecture

[0056] Figure 2 is an illustration of an example architecture of the generative neural network control system 106. The generative neural network control system 106 includes a decomposition vector determination module 202, an image editing module 204, an image style module 206, an image synthesis module 208, an output module 210, and an input module 212.

[0057] The generative neural network 104 is a neural network that has been trained to generate images. Additionally or alternatively, additional training or fine-tuning of the generative neural network 104 can optionally be performed contemporaneously with or subsequent to use of the techniques discussed herein.

[0058] The decomposition vector determination module 202 implements functionality to generate one or more decomposition vectors for a set of activation values (also referred to simply as an activation set) generated by a layer of the generative neural network 104. The image editing module 204 implements functionality to perform a translation or transformation of an image generated by the generative neural network 104, such as zooming in or out, translating left or right, etc. Additionally or alternatively, the image editing module 204 also implements functionality to clean up an image generated by the generative neural network 104 by removing artifacts, or to simplify an image generated by the generative neural network 104 by removing details.

[0059] The image style module 206 implements functionality to change the style of an image generated by the generative neural network 104, such as changing the coloring of the image or the background of the image. The image synthesis module 208 implements functionality to cause the generative neural network 104 to generate an image that is a synthesis of two other images. The output module 210 generates a user interface 214 of a display that indicates the types of control functionality that the generative neural network control system 106 can impose on the generative neural network 104. The input module 212 implements functionality to receive user input 216 that indicates the control functionality that the user desires the generative neural network control system 106 to impose on the generative neural network 104.

[0060] To control the generative neural network 104, the generative neural network control system 106 provides various information to the generative neural network 104 and optionally receives various information from the generative neural network 104. In one or more implementations, the generative neural network 104 receives an initial latent vector 220 from which the generative neural network 104 will generate an image. Optionally, the generative neural network control system 106 also receives the initial latent vector 220. The initial latent vector 220 can be generated in any of a variety of different ways, such as randomly or pseudo-randomly. The initial latent vector 220 can also include or be accompanied by a class vector that identifies a class of object (e.g., dog, cat, man, woman, car, landscape, etc.) that the generative neural network 104 is to generate. The class of object can be identified in a variety of ways, such as from user input that selects the class, from a configuration setting of the generative neural network 104, etc.

[0061] In one or more implementations, the generative neural network control system 106 obtains the updated latent vector 222 (also referred to as the altered latent vector) and provides it to the generative neural network 104. As discussed in greater detail below, the updated latent vector 222 is used in place of the initial latent vector 220 in certain layers of the generative neural network 104. In one or more implementations, a set of activation values 224 generated by one layer of the generative neural network 104 is provided to the generative neural network control system 106. The set of activation values is modified by the generative neural network control system 106, and the modified activation values 226 are provided to the generative neural network control system 106. As discussed in greater detail below, the modified activation values 226 are used in generating the image 228.

[0062] Although illustrated as including all of the modules 202-212, in various alternatives, one or more of the modules 202-212 need not be included in the generative neural network control system 106. For example, if the generative neural network control system 106 is not to include functionality to cause the generative neural network 104 to generate an image that is a composite of two other images, then the image compositing module 208 need not be included in the generative neural network control system 106.

[0063] Decomposition

[0064] Figure 3 An example generative neural network 300 is illustrated. The generative neural network 300 can be, for example, the generative neural network 104 of Figure 1 or Figure 2 The generative neural network 300 includes a plurality (n) of layers, illustrated as layers 302, 304, 306, and 308. In general, each layer performs one or more operations or functions on received data, and generates output data referred to as activation values or simply activations. The operations or functions performed in any given layer vary based on the architecture of the generative neural network 300. The example generative neural network 300 is a feedforward neural network that receives a latent vector 312 as input to an initial layer of the generative neural network 300 (layer 302 in the illustrated example) but is not directly input to other layers of the generative neural network 300.

[0065] For example, the generative neural network 300 can be implemented as one or more convolutional neural networks (CNNs). A CNN is formed from layers of nodes (i.e., neurons) and can include various layers that perform various operations or functions, such as input functions, output functions, convolution functions, pooling functions, activation functions, fully connected functions, normalization functions, and the like.

[0066] The data input to each layer (except layer 1) is referred to as the data in the intermediate latent space. In contrast, the latent vector 312 input to layer 1 of the generative neural network 300 is referred to as the data in the latent space of the generative neural network, or the data in the initial latent space. The data in the intermediate latent space includes activation values ​​(e.g., one or both of the following: activation values ​​generated by a previous layer, or activation values ​​generated using the techniques discussed herein), and optionally includes latent vectors as discussed in more detail below. The techniques discussed herein modify the intermediate latent space to achieve a variety of different effects when generating new images.

[0067] To generate a new image 310, a latent vector 312 is input to the initial layer of generative neural network 300, illustrated as layer 302. In layer 302, one or more functions 314 are performed on latent vector 312, generating various activation values ​​316. Activation values ​​316 are provided as input to layer 304. In layer 304, one or more functions 318 are performed on activation values ​​316, generating various activation values ​​320. Activation values ​​320 are provided as input to layer 306. In layer 306, one or more functions 322 are performed on activation values ​​320, generating various activation values ​​324. Activation values ​​324 are provided as input to the next layer. Finally, the activations from the penultimate layer are provided to layer 308. In layer 308, one or more functions 326 are performed on the activation values ​​received from the previous layer, generating various activation values ​​328. Activation values ​​328 are output as the generated new image 310.

[0068] The decomposition vector determination module 202 generates one or more decomposition vectors 330 based on the activation values ​​generated by one or more layers of the generative neural network 300. To generate the one or more decomposition vectors 330, a particular number of different latent vectors 312 are provided to the generative neural network 300, and the activation values ​​320 generated for those latent vectors 312 are received and maintained by the decomposition vector determination module 202. The particular number of latent vectors 312 can vary, but is typically approximately 10,000 to 100,000 latent vectors in order to provide a large number of examples over which decomposition can be performed.

[0069] exist Figure 3 In the illustrated example, activation values ​​320 are provided to the decomposition vector determination module 202 as well as to (or instead of) one or more functions 322. The decomposition vector determination module 202 is thus able to generate one or more decomposition vectors 330 based on the activation values ​​generated in the layer 304 (also referred to as generating one or more decomposition vectors 330 for the layer 304).

[0070] The decomposition performed by the decomposition vector determination module 202 refers to analyzing the set of activation values received from a layer of the generative neural network 300 to identify one or more vectors that represent the set of activation values. The activation values received from a layer of a particular latent vector 312 are unfolded (transformed) into an activation vector. This results in a large number (e.g., 10,000 to 100,000) of activation vectors, each of which can have a high dimensionality (e.g., hundreds or thousands of elements). This set of activation vectors can be analyzed using any of a variety of different public or proprietary techniques in order to generate the decomposition vectors 330. For example, each of the activation vectors in this set represents a point in a multi-dimensional space (but each vector has multiple dimensions). These points create a point cloud in the multi-dimensional state, and various techniques can be used to determine the direction in which this point cloud is at maximum extension (the direction in which the point cloud has maximum variance). The vectors that indicate these directions of maximum extension (variance) are the decomposition vectors 330.

[0071] In one or more implementations, the decomposition vector determination module 202 is implemented using principal component analysis (PCA) to characterize the shape of the point cloud. The decomposition vector determination module 202 generates a set of eigenvectors from the point cloud, each of which is one of the decomposition vectors 330. The decomposition vector determination module 202 also generates and stores a set of eigenvalues for the point cloud.

[0072] Additionally or alternatively, other decomposition techniques can be used to generate the decomposition vectors 330, including other eigen-decomposition techniques that generate eigenvectors. By way of example, the decomposition vector determination module 202 can be implemented using sparse PCA, independent component analysis (ICA), non-negative matrix factorization (NNMF), and so on.

[0073] Although Figure 3 The example illustrates generating one or more decomposition vectors 330 for the layer 304, but additionally or alternatively, the decomposition vector determination module 202 can generate decomposition vectors 330 for any other layer of the generative neural network 300. The decomposition vector determination module 202 typically generates one or more decomposition vectors for a second layer (the layer 304) or a later layer (e.g., the layers 306-308) in the generative neural network 300, but can also generate one or more decomposition vectors for a first layer (the layer 302) in some cases. In addition to generating decomposition vectors 330 from activation values output by a layer other than the layer 304, the decomposition vector determination module 202 can generate one or more decomposition vectors for a layer other than the layer 304 in a similar manner as discussed above with respect to generating decomposition vectors 330 for the layer 304. For example, one or more decomposition vectors are generated for the layer 306 using the activation values 324.

[0074] It should be noted that one or more decomposition vectors 330 can be generated for each of multiple different layers simultaneously or consecutively. For example, a set of multiple latent vectors 312 can be provided to generative neural network 300, and for each latent vector 312, decomposition vector determination module 202 receives and maintains activation values 320 of layer 304 and activation values 324 of layer 306. After having received activation values 320 and 324 for all latent vectors in the set of multiple latent vectors 312, decomposition vector determination module 202 generates a set of decomposition vectors 330 for layer 304 and a set of decomposition vectors for layer 306, thus generating decomposition vectors for multiple different layers simultaneously.

[0075] By another example, a first set of multiple latent vectors 312 can be provided to generative neural network 300, and for each latent vector in the first set, decomposition vector determination module 202 receives and maintains activation values 320 of layer 304. A second set of multiple latent vectors 312 can be provided to generative neural network 300 (the same or different latent vectors as the first set), and for each latent vector in the second set, decomposition vector determination module 202 receives and maintains activation values 324 of layer 306. Decomposition vector determination module 202 then generates a set of decomposition vectors 330 for layer 304 and a set of decomposition vectors for layer 306, thus generating decomposition vectors for multiple different layers consecutively. Decomposition vector determination module 202 can generate the set of decomposition vectors for layer 304 after having received activation values 320 and 324 for all latent vectors in the first and second sets of multiple latent vectors 312, or after having received activation values for all latent vectors in the first set of multiple latent vectors, but before receiving activation values for all latent vectors in the second set of multiple latent vectors.

[0076] It should be noted that when generating decomposition vectors 330, generative neural network 300 can but need not generate a new image 310 for each latent vector in multiple latent vectors 312. Rather, after having provided activation values of the final layer for which decomposition vectors are being generated to decomposition vector determination module 202, generative neural network 300 can stop performing functions and stop generating activation values of latent vectors 312. For example, if decomposition vector determination module 202 generates decomposition vectors 330 only for layer 304, then after providing activation values 320 to decomposition vector determination module 202, generative neural network 300 can stop operating on the generated activation values, and thus functions 322 need not use activation values 320 to generate activation values 324.

[0077] Figure 4 Another example generative neural network 400 is illustrated. Generative neural network 400 can be, for example,Figure 1 or Figure 2 generative neural network 104. The generative neural network 400 includes multiple (n) layers, illustrated as layers 402, 404, 406, and 408, and generates a new image 410. Generally, similar to the layers of the generative neural network 300 of Figure 3 each layer performs one or more operations or functions on received data and generates output data, referred to as activation values or simply activations. However, the example generative neural network 400 differs from the generative neural network 300 of Figure 3 in that a latent vector 412 is input to multiple layers (e.g., each layer) of the generative neural network 400.

[0078] To generate the new image 410, the latent vector 412 is input to an initial layer of the generative neural network 400, illustrated as layer 402. In the layer 402, one or more functions 414 are performed on the latent vector 412, which generates various activation values 416. The activation values 416, as well as the latent vector 412, are provided as input to layer 404. In the layer 404, one or more functions 418 are performed on the activation values 416 and the latent vector 412, which generates various activation values 420. The activation values 420, as well as the latent vector 412, are provided as input to layer 406. In the layer 406, one or more functions 422 are performed on the activation values 420 and the latent vector 412, which generates various activation values 424. The activation values 424, as well as the latent vector 412, are provided as input to the next layer. Ultimately, the activation from the penultimate layer, as well as the latent vector 412, are provided as input to layer 408. In the layer 408, one or more functions 426 are performed on the activation values received from the previous layer and the latent vector 412, which generates various activation values 428. The output activation values 428 are output as the generated new image 410. Thus, the generative neural network 400 operates similar to the generative neural network 300 of Figure 3 except that, in addition to performing one or more functions on activation values received from a previous layer and the latent vector 412, one or more functions are performed in one or more layers of the generative neural network 400.

[0079] The decomposition vector determination module 202 generates one or more decomposition vectors 430 based on the activation values generated by one or more layers of the generative neural network 400 in a similar manner as discussed above with respect to the generative neural network 300 of Figure 3 Figure 4 ​In the example of FIG. 4, the decomposed vector determination module 202 expands (transposes) the activation values received from the layer for a particular latent vector 412 into an activation vector and appends the particular latent vector 412 to the activation vector (e.g., concatenates the activation vector with the latent vector). Although the dimensional space is larger due to appending the latent vector 412 to the activation vector generated from the activation values, the decomposed vector determination module 202 then proceeds as discussed above with respect to Figure 3 The general discussion continues.

[0080] Additionally or alternatively, similar to the discussion above with respect to the generative neural network 300 of Figure 3 The decomposed vector determination module 202 can generate one or more decomposed vectors based on activation values generated by one or more layers of the generative neural network 400. In this case, the decomposed vector determination module 202 ignores (does not consider) the latent vector 412 in generating the one or more decomposed vectors.

[0081] Referring back to Figure 2 , the decomposed vector determination module 202 obtains a decomposed vector for one or more layers of the generative neural network 104. The decomposed vector determination module 202 can obtain a decomposed vector as discussed above with respect to Figure 3 Additionally or alternatively, the decomposed vector can have been previously determined and obtained by the decomposed vector determination module 202 (e.g., from storage at a computing device implementing the generative neural network control system 106, from a remote device or service, etc.).

[0082] For example, one or more decomposed vectors can be generated and stored for each of multiple layers (e.g., all layers) of the generative neural network 104. These decomposed vectors can be generated by the generative neural network control system 106 or by another device or system (e.g., accessed via the network 112).

[0083] As discussed above, many (e.g., 10,000 to 100,000) Figure 3 The input latent vector 312 or Figure 4 In one or more implementations, the class vector is fixed (e.g., the same for all latent vectors 312 or 412) for the latent vectors input to the generative neural network to generate the decomposed vector. Additionally or alternatively, different class vectors can be used.

[0084] Image Editing, Cleaning, and Simplification

[0085] In one or more implementations, the image editing module 204 implements functionality to perform a translation or transformation of the image generated by the generative neural network 104, such as zooming in or out, translating left or right, etc. The image editing module 204 performs the translation or transformation by modifying one or both of the activation values from one or more layers and the latent vector input to one or more layers using one or more decomposition vectors 230. Additionally or alternatively, the image editing module 204 implements functionality to clean up artifacts or simplify the image generated by the generative neural network 104, as discussed in more detail below.

[0086] Each decomposition vector 230 corresponds to a different effect (translation or transformation) on the image 228 generated by the generative neural network 104. The decomposition vectors 230 can correspond to a variety of different effects. For example, the decomposition vectors 230 can correspond to a translation (moving an object in the image, such as a dog, a person, a car, etc.) from left to right or right to left, from top to bottom or bottom to top, combinations thereof, etc. By way of another example, the decomposition vectors 230 can correspond to a camera motion (changing a view of an object in the image to produce the effect of moving a camera that captured the image in a particular direction) from left to right or right to left, from top to bottom or bottom to top, combinations thereof, etc. By way of another example, the decomposition vectors 230 can correspond to zooming in or out on an object in the image.

[0087] By way of another example, the decomposition vectors 230 can correspond to a camera rotation (changing a view of an object in the image to produce the effect of moving a camera that captured the image around the object) from left to right or right to left, from top to bottom or bottom to top, combinations thereof, etc. By way of another example, the decomposition vectors 230 can correspond to changing a dimension (e.g., height or width) of an object in the image, such as making the object taller or shorter, wider or narrower, combinations thereof, etc.

[0088] Figure 5 An example 500 of a translation effect is illustrated. The example 500 illustrates translating an object in the image, a dog, from left to right. As illustrated at 502, the dog is in the left side portion of the image. At 504, the dog has been translated slightly to the right, and at 506, the dog has been translated to the right and is approximately in the center of the image. At 508, the dog has been translated to the right side portion of the image.

[0089] Figure 6An example 600 of camera motion effects is illustrated. The example 600 illustrates a camera motion from the top right to the bottom left. As illustrated at 602, the view of the dog in the image appears to be taken from a camera that is above and to the right of the dog. At 604, the view of the dog in the image appears to be taken from a camera that is below and to the left of the camera in 602, and at 606, the view of the dog in the image appears to be taken from a camera that is even further below and further left of the camera of 604. At 608, the view of the dog in the image appears to be taken from a camera that is further below and to the left of the camera of 606.

[0090] Figure 7 An example 700 of camera rotation effects is illustrated. The example 700 illustrates a camera rotation from right to left. As illustrated at 702, the view of the dog in the image appears to be taken from a camera that is to the right of the dog (from the camera’s perspective). At 704, the view of the dog in the image appears to be taken from a camera that is rotated around the dog to the left of the camera at 702, and at 706, the view of the dog in the image appears to be taken from a camera that is rotated around the dog even further to the left of the camera at 704. At 708, the view of the dog in the image appears to be taken from a camera that is to the left of the dog (from the camera’s perspective).

[0091] Referring back to Figure 2 The effects of a particular decomposition vector 230 corresponding to a particular generative neural network 104 can be determined in various ways, such as empirically. However, the effects of a particular decomposition vector 230 corresponding to a particular generative neural network 104 remain the same for different class vectors. Thus, a user interface can be displayed or otherwise presented to a user (e.g., as part of a digital content creation application) that allows the user to generate an image using a generative neural network 104 and select a particular effect (e.g., zoom in or zoom out), and the digital content creation application uses the appropriate decomposition vector 230 to perform the selected effect.

[0092] Additionally, different layers of a generative neural network 104 correspond to different amounts of global effects or local effects that modifications to activation values or latent vector inputs have on the image 228. Modifications made at earlier layers (e.g., layer 2 of Figure 3 have more global effects on the image 228, while modifications made at later layers (e.g., the fifth or sixth layer) have more local effects on the image 228. Global effects refer to modifications that affect a large amount of the image, while local effects refer to modifications that affect a small amount of the image.

[0093] The results of modifying the activation values or latent vector inputs in different layers for different effects and different generative neural networks can be determined in various ways, e.g., empirically. However, the results of modifying the activation values or latent vector inputs in a particular layer of a particular generative neural network 104 remain the same for different class vectors. Thus, a user interface can be displayed or otherwise presented to a user (e.g., as part of a digital content creation application) that allows the user to generate an image using the generative neural network 104 and select a particular result (e.g., more global or more local), and the digital content creation application uses the appropriate decomposition vector 230 to obtain the selected result.

[0094] In one or more implementations, the activation values or latent vector inputs are modified after an initial image is generated by the generative neural network 104. This allows a user to view the initial image, request that one or more effects be performed, and have the generative neural network 104 generate a new image with the requested effects. Additionally or alternatively, the activation values or latent vector inputs can be modified as part of generating an initial image by the generative neural network 104. For example, a user can request that one or more effects be performed, and the initial image generated by the generative neural network 104 has the requested effects.

[0095] In one or more implementations, the image editing module 204 performs a translation or transformation by modifying the activation values from one or more layers of the generative neural network 104 using one or more decomposition vectors 230. Figure 8 An example of modifying activation values of a layer of a generative neural network 800 is illustrated. The generative neural network 800 can be, for example, the generative neural network 104 of Figure 1 or Figure 2 The generative neural network 800 includes a plurality (n) of layers, illustrated as layers 802, 804, 806, and 808, and generates a new image 810. In general, similar to the layers of the generative neural network 300 of Figure 3 each layer performs one or more operations or functions on received data and generates output data, referred to as activation values or simply activations. In the example of the generative neural network 300 of Figure 8 In the example, the image editing module 204 modifies the modified activation values generated at the layer 804 based on the decomposition vector 812 obtained by the decomposition vector determination module 202. The modification of the activation values generated at the layer 804 is in the intermediate latent space between the layers 804 and 806.

[0096] To generate the new image 810, the latent vector 814 is input to the first layer 802. In the layer 802, one or more functions 816 are performed on the latent vector 814, which generates various activation values 818. The activation values 818 are provided as input to the layer 804. In the layer 804, one or more functions 820 are performed on the activation values 818, which generates various activation values 822. The activation values 822 are provided as input to the image editing module 204. The image editing module 204 uses the decomposition vector 812 to modify the activation values 822, and provides the modified activation values 824 as input to the layer 806.

[0097] In the layer 806, one or more functions 826 are performed on the modified activation values 824, and the one or more functions 826 generate various activation values 828. The activation values 828 are provided as input to the next layer. Eventually, the activation from the penultimate layer is provided as input to the layer 808. In the layer 808, one or more functions 830 are performed on the activation values received from the previous layer, which generates various activation values 832. The output activation values 832 are output as the generated new image 810.

[0098] The image editing module 204 modifies the activation values 822 based on the decomposition vector 812. In one or more implementations, the image editing module 204 unrolls the activation values 822 into an activation vector, and adds or subtracts the decomposition vector 812 from the activation vector, resulting in a modified activation vector. The image editing module 204 converts the modified activation vector into the same format (e.g., a matrix) as the activation values 822 to generate the modified activation values 824.

[0099] The value that is added to or subtracted from the activation vector is determined based on the size of the decomposition vector 812. By controlling the value that is added to or subtracted from the activation vector, the corresponding activation vector is controlled to move how far in the direction corresponding to the decomposition vector 812. For example, user input indicating the amount and intensity of the desired effect can be received. A lesser amount corresponds to a smaller value, and a greater amount corresponds to a larger value. For example, if the user input requests a small amount of the desired effect, then the image editing module 204 adds a fraction (e.g., ¼) of the size of the decomposition vector 812 to the corresponding activation vector. However, if the user input requests a large amount of the desired effect, then the image editing module 204 adds a multiple (e.g., 2.0) of the size of the decomposition vector 812 to the corresponding activation vector.

[0100] The modification to the activation values 822 based on adding the decomposition vector 810 to the corresponding activation vector results in a translation or transformation in one direction (e.g., rightward translation, upward translation, zoom-in, etc.). On the other hand, the modification to the activation values 822 based on subtracting the decomposition vector 812 from the corresponding activation vector elements results in a translation or transformation in the opposite direction (e.g., leftward translation, downward translation, zoom-out, etc.).

[0101] In one or more implementations, where an initial image has already been generated, it is not necessary to re-input the latent vector 814 to the generative neural network 812 in order to perform a desired effect. Rather, the activation values from the layers 802-808 can have been stored and are accessible by the image editing module 204. Thus, the image editing module 204 can take the activation values 822 as previously generated, modify the activation values 822, and provide the modified activation values to the layer 806. Thus, new activation values 828 and 832 will be generated for the new image 810, but it is not necessary to re-generate the previously generated activation values 818 and 822, thereby improving the performance of the generative neural network 800 and the generative neural network control system 106.

[0102] Figure 9 Another example generative neural network 900 is illustrated. The generative neural network 900 can be, for example, the generative neural network 104 of Figure 1 or Figure 2 The generative neural network 900 includes a plurality (n) of layers illustrated as layers 902, 904, 906, and 908 and generates a new image 910. In general, similar to the layers of the generative neural network 800 of Figure 8 each layer performs one or more operations or functions on received data and generates output data referred to as activation values or simply activations. However, the example generative neural network 900 differs from the generative neural network 800 of Figure 8 in that a latent vector 912 is input to the plurality of layers (e.g., each layer) of the generative neural network 900.

[0103] In Figure 9In the example of FIG. 9, the image editing module 204 modifies the activation values generated at layer 904 based on the decomposition vector 914 obtained by the decomposition vector determination module 202. To generate the new image 910, the latent vector 912 is input to the initial layer 902. In layer 902, one or more functions 916 are performed on the latent vector 912, which generates various activation values 918. The activation values 918, as well as the latent vector 912, are provided as input to layer 904. In layer 904, one or more functions 920 are performed on the activation values 918 and the latent vector 912, which generates various activation values 922. The activation values 922 are provided as input to the image editing module 204. The image editing module 204 uses the decomposition vector 914 to modify the activation values 922 and provides the modified activation values 924 as input to layer 906.

[0104] In layer 906, one or more functions 926 are performed on the modified activation values 924 and the latent vector 912, and the one or more functions 926 generate various activation values 928. The activation values 928, as well as the latent vector 912, are provided as input to the next layer. Eventually, the activation from the penultimate layer, as well as the latent vector 912, are provided as input to layer 908. In layer 908, one or more functions 930 are performed on the activation values received from the previous layer and the latent vector 912, which generates various activation values 932. The output activation values 930 are output as the generated new image 910.

[0105] The image editing module 204 can modify the activation values 922 based on the decomposition vector 914 in various ways similar to the discussion above regarding the example of FIG. 9. Figure 8 However, because the decomposition vector 914 is generated based on the concatenated activation vector and latent vector, the image editing module 204 adds (or subtracts) the portion of the decomposition vector 914 corresponding to the activation vector to (or from) the activation vector. Thus, in the example of FIG. 9, although the image editing module 204 modifies the activation values received from a layer, the same latent vector 912 is input to multiple different layers. Figure 9

[0106] Figure 10 Another example generative neural network 1000 is illustrated. The generative neural network 1000 can be, for example, the generative neural network 104 of FIG. 1. Figure 1 or Figure 2 The generative neural network 1000 includes multiple (n) layers illustrated as layers 1002, 1004, 1006, and 1008 and generates a new image 1010. Generally, the layers 1002, 1004, 1006, and 1008 are similar to the layers 102, 104, 106, and 108 of FIG. 1. Figure 9 ​of the generative neural network 900, each layer performs one or more operations or functions on received data and generates output data referred to as activation values or simply activations. However, the example generative neural network 1000 differs from the generative neural network 900 in that the latent vector 1012 is input to some layers of the generative neural network 1000, but is modified by the image editing module 204 based on the decomposition vector 1014 and the modified latent vector is input to other layers. Figure 9

[0107] To generate the new image 1010, the latent vector 1012 is input to an initial layer 1002. In the layer 1002, one or more functions 1016 are performed on the latent vector 1012, which generates various activation values 1018. The activation values 1018, as well as the latent vector 1012, are provided as input to a layer 1004. In the layer 1004, one or more functions 1020 are performed on the activation values 1018 and the latent vector 1012, which generates various activation values 1022. The activation values 1022 are provided as input to the image editing module 204.

[0108] The image editing module 204 uses the decomposition vector 1014 to modify the activation values 1022 and provides the modified activation values 1024 as input to a layer 1006. Additionally, the image editing module 204 uses the decomposition vector 1014 to modify the latent vector 1012 and provides the modified latent vector 1026 as input to all layers after the layer 1004, such as the layers 1006 and 1008.

[0109] In the layer 1006, one or more functions 1028 are performed on the modified activation values 1024 and the modified latent vector 1026, and the one or more functions 1028 generate various activation values 1030. The activation values 1030, as well as the modified latent vector 1026, are provided as input to the next layer. Ultimately, the activation from the penultimate layer, as well as the modified latent vector 1026, are provided as input to a layer 1008. In the layer 1008, one or more functions 1032 are performed on the activation values received from the previous layer and the modified latent vector 1026, which generates various activation values 1034. The output activation values 1034 are output as the generated new image 1010.

[0110] The image editing module 204 can modify the latent vector 1012 in the same manner as described above with respect to the activation values 1022. Figure 8 ​corresponding to the activation vector of the decomposition vector 1014 to the activation vector (or subtracting the portion of the decomposition vector 1014 corresponding to the activation vector from the activation vector), and adding the portion of the decomposition vector 1014 corresponding to the latent vector to the latent vector (or subtracting the portion of the decomposition vector 1014 corresponding to the latent vector from the latent vector). Thus, in the example of the generative neural network 1000, the image editing module 204 modifies both the activation values and the latent vector 1012 received from a layer, and inputs the modified latent vector into a subsequent layer of the generative neural network 1000. Figure 10

[0111] Figure 11 Another example generative neural network 1100 is illustrated. The generative neural network 1100 can be, for example, the generative neural network 104 of Figure 1 or Figure 2 The generative neural network 1100 includes a plurality (n) of layers, illustrated as layers 1102, 1104, 1106, and 1108, and generates a new image 1110. In general, similar to the layers of the generative neural network 800 of Figure 8 each layer performs one or more operations or functions on received data and generates output data, referred to as activation values or simply activations. However, the example generative neural network 1100 differs from the generative neural network 800 of Figure 8 in that a latent vector 1112 is input to some layers of the generative neural network 1100, but is modified based on a decomposition vector 1114 by the image editing module 204, and the modified latent vector is input to other layers.

[0112] To generate the new image 1110, the latent vector 1112 is input to the initial layer 1102. In the layer 1102, one or more functions 1116 are performed on the latent vector 1112, which generates various activation values 1118. The activation values 1118, as well as the latent vector 1112, are provided as input to the layer 1104. In the layer 1104, one or more functions 1120 are performed on the activation values 1118 and the latent vector 1112, which generates various activation values 1122. The activation values 1122 are provided as input to the layer 1106.

[0113] ​The image editing module 204 modifies the latent vector 1112 using the decomposition vector 1114 and provides the modified latent vector 1124 as input to all layers after the layer 1104, such as the layers 1106 and 1108. In the layer 1106, one or more functions 1126 are performed on the activation values 1122 and the modified latent vector 1124, and the one or more functions 1126 generate various activation values 1128. The activation values 1128, as well as the modified latent vector 1124, are provided as input to the next layer. Ultimately, the activation from the penultimate layer and the modified latent vector 1124 are provided as input to the layer 1108. In the layer 1108, one or more functions 1130 are performed on the activation values received from the previous layer and the modified latent vector 1124, which generates various activation values 1132. The output activation values 1132 are output as the generated new image 1110.

[0114] The image editing module 204 can modify the activation values 1112 based on the decomposition vector 1114 in various ways similar to the discussion above with respect to the example of the generative neural network 1100. Figure 8 However, because the decomposition vector 1114 is generated based on the concatenated activation vector and the latent vector, the image editing module 204 adds (or subtracts) the portion of the decomposition vector 1114 corresponding to the latent vector to (or from) the latent vector. Thus, in the example of the generative neural network 1100, the image editing module 204 modifies the latent vector 1112 and inputs the modified latent vector 1124 into subsequent layers of the generative neural network 1100, but does not modify the activation values generated by any particular layer. This allows the translation and transformation techniques discussed herein to be performed, for example, even in cases where the activation values in the layers of the generative neural network 1100 are not available for modification, such as in cases where the generative neural network 1100 is implemented in hardware, such as an application-specific integrated circuit (ASIC). Figure 11

[0115] With reference to Figure 2 ​The generative neural network control system 106 can be implemented in various different ways as discussed above. In the examples discussed above, a decomposition vector 230 is used to modify one or both of the initial latent vector 220 and the activation values 224 at a particular layer at a time. Additionally or alternatively, one or both of the initial latent vector 220 and the activation values 224 can be modified at different layers used to generate the image 228. For example, one decomposition vector 230 can be used to modify one or both of the initial latent vector 220 and the activation values 224 after layer 2 of the generative neural network 104, and a different decomposition vector 230 can be used to modify one or both of the initial latent vector 220 and the activation values 224 after layer 3 of the generative neural network 104. By way of another example, one decomposition vector 230 can be used to modify the initial latent vector 220 after layer 2 of the generative neural network 104, and a different decomposition vector 230 can be used to modify the modified latent vector after layer 4 of the generative neural network 104.

[0116] Additionally or alternatively, one or more of the initial latent vector 220 and the activation values 224 can be modified using multiple different decomposition vectors 230 at the same layer in order to achieve two different effects. For example, one decomposition vector 230 can be used to modify one or both of the initial latent vector 220 and the activation values 224 after layer 2, and then a different decomposition vector 230 can also be used to modify one or both of the initial latent vector 220 and the activation values 224 after layer 2.

[0117] Thus, as can be seen from the discussion herein, the image editing module 204 allows various edits (such as transformations and translations) to be performed on images generated by a generative neural network. The techniques discussed herein are used on a trained generative neural network, thus no additional training data or other data is needed. For example, additional example images do not have to be provided to the generative neural network in order to have a particular effect on the generated images.

[0118] Image Cleaning

[0119] In one or more implementations, the image editing module 204 implements functionality to clean up an image generated by the generative neural network 104 by removing artifacts or to simplify an image generated by the generative neural network 104 by removing details. In some cases, the generative neural network 104 generates visual artifacts as part of the image 228. These visual artifacts are anomalies in the image 228, such as noise, incorrect colors, etc. The image editing module 204 removes the artifacts and simplifies the image by projecting the activation values from one or more layers of the generative neural network 104 onto the linear subspace determined by the decomposition vector determination module 202.

[0120] Figure 12 An example 1200 of removing artifacts from an image is illustrated. The example 1200 illustrates an initial image 1202 of a restaurant. An artifact, illustrated with an oval 1204, is present in the lower right corner of the image 1202. A new image 1206 is generated by the generative neural network 104 based on a modified latent vector from the image editing module 204 that removes the artifact in the image 1202.

[0121] Referring back to Figure 2 , the image editing module 204 removes artifacts and simplifies images by modifying activation values from one or more layers of the generative neural network 104 using the plurality of decomposition vectors 230. The image editing module 204 modifies the activation values by unfolding the activation values from a layer of the generative neural network 104 into an activation vector and creating a modified activation vector by projecting the activation vector onto the top N decomposition vectors 230. The image editing module 204 converts the modified activation vector into the same format as the activation values (e.g., a matrix) and provides the modified activation values to the next layer in the generative neural network 104.

[0122] For example, referring back to Figure 8 , the activation values 822 are provided as input to the image editing module 204, which modifies the activation values 822 based on the plurality of decomposition vectors 812. The image editing module 204 unfolds the activation values 822 into an activation vector and projects the activation vector onto the top N decomposition vectors, resulting in a modified activation vector. The image editing module 204 converts the modified activation vector into the same format as the activation values 822 (e.g., a matrix) to generate modified activation values 824.

[0123] By another example, referring back to Figure 9 , the activation values 922 are provided as input to the image editing module 204, which modifies the activation values 922 based on the plurality of decomposition vectors 914. The image editing module 204 unfolds the activation values 922 into an activation vector and projects the activation vector onto the top N decomposition vectors 914, resulting in a modified activation vector. The image editing module 204 converts the modified activation vector into the same format as the activation values 922 (e.g., a matrix) to generate modified activation values 924.

[0124] Referring back to Figure 2The number N of decomposition vectors 230 on which the activation vectors project can vary. In one or more implementations, the number N of decomposition vectors 230 is between 30 and 100, which results in the removal of artifacts from the image 228. Additionally or alternatively, the number N of decomposition vectors 230 can be fewer, such as between 10 and 20, to simplify the image 228. Simplifying the image 228 refers to making the image 228 appear more like an average image of the class by removing particular details. For example, if the class is a restaurant, particular details such as windows, curtains, pictures, etc. can be removed and replaced with simple, unadorned walls.

[0125] In one or more implementations, the image editing module 204 replaces all of the activation values input to the next layer with the modified activation values generated by the image editing module 204. For example, the activation values used by the one or more functions 826 are the modified activation values 824 generated by the image editing module 204. Figure 8 In one or more implementations, the image editing module 204 replaces all of the activation values input to the next layer with the modified activation values generated by the image editing module 204. For example, the activation values used by the one or more functions 826 are the modified activation values 824 generated by the image editing module 204.

[0126] Additionally or alternatively, the image editing module 204 replaces only some of the activation values input to the next layer with the modified activation values generated by the image editing module 204. For example, the activation values used by the one or more functions 826 are a combination of the modified activation values 824 and the activation values 822. Replacing only some of the activation values 822 with the modified activation values allows the image editing module 204 to apply the clean-up and simplification only to particular spatial regions of the generated image. Figure 8 Additionally or alternatively, the image editing module 204 replaces only some of the activation values input to the next layer with the modified activation values generated by the image editing module 204. For example, the activation values used by the one or more functions 826 are a combination of the modified activation values 824 and the activation values 822. Replacing only some of the activation values 822 with the modified activation values allows the image editing module 204 to apply the clean-up and simplification only to particular spatial regions of the generated image.

[0127] In one or more implementations, each activation value corresponds to a particular portion of the generated image, such as a particular pixel or set of pixel classes (e.g., one activation value can correspond to 16 pixels in a 4x4 grid in the upper right corner of the generated image). Which activation value corresponds to which portion of the generated image can be determined in any of a variety of different ways. For example, the activation values can be arranged in a matrix format, and the dimensions of the matrix can be compared to the dimensions of the image 228 to easily determine which portion of the generated image corresponds to which activation value. The comparison can be performed by the image editing module 204 in a variety of ways, such as automatically.

[0128] By allowing the image editing module 204 to apply the clean-up and simplification only to particular spatial regions of the generated image, user input can be received that specifies particular regions of the image 228 in which artifacts exist or in which the user desires the image to be simplified. For example, the image 228 can be initially generated by the generative neural network 104. User touch input that roughly draws a circle or other geometric shape around the artifacts can be received, verbal input that specifies particular regions of the image 228 can be received (e.g., verbal input of “upper right corner” can be received), etc.

[0129] In response to such user input, the image editing module 204 determines which portions (e.g., pixels) of the image 228 are identified by the user input, and also determines which activation values correspond to the identified portions of the image 228. The image editing module 204 uses the decomposition vector 230 to modify the activation values corresponding to the identified portions of the image 228. For example, referring again to Figure 8 , the image editing module 204 receives the activation values 822 and modifies those of the activation values 822 that correspond to the identified portions of the image 228. The modified activation values 824 provided to the layer 806 include the modified activation values of the activation values 822 that correspond to the identified portions of the image 228, and the activation values 822 (unmodified) that correspond to the portions of the image 228 that were not identified by the user input.

[0130] As discussed above, different layers of the generative neural network 104 correspond to different amounts of global or local effect of the modification to the activation values or latent vector input on the image 228. Accordingly, the image editing module 204 can modify the activation values at earlier layers (e.g., layer 2) to more globally clean up or simplify the modification, or at later layers (e.g., layer 5 or layer 6) to more locally clean up or simplify the modification. Figure 8 or Figure 9 layer 2) to more globally clean up or simplify the modification, or at later layers (e.g., layer 5 or layer 6) to more locally clean up or simplify the modification.

[0131] It should be noted that, where an initial image has already been generated, it is not necessary to re-input the latent vector 220 to the generative neural network 104 in order to perform the desired effect. Rather, the activation values from the layers of the generative neural network 104 can have been stored and are accessible by the image editing module 204. Accordingly, the image editing module 204 can take the activation values from one layer as previously generated, modify the activation values as discussed above to clean up or simplify the image, and provide the modified activation values to the next layer. Accordingly, new activation values for the next layer and later layers of the generative neural network 104 will be generated to create a new image, but the previously generated activation values for the earlier layers do not have to be regenerated, thereby improving the performance of the generative neural network 104 and the generative neural network control system 106.

[0132] Image Style

[0133] In one or more implementations, the image style module 206 implements functionality to change the style of an image generated by the generative neural network 104. The style of an image refers to the unique appearance of the image, such as the coloring of the image or the background of the image. For example, the style of a generated image can be changed by applying an initial latent vector to a first one or more layers of the generative neural network, but applying a different latent vector to a second one or more later layers of the generative neural network. Thus, a different latent vector is applied to different layers of the generative neural network. However, the activation values between the layers are not necessarily modified (although these activation values can be modified to produce additional effects as discussed herein).

[0134] By freezing the activations of the early layers using the initial latent vector, the spatial structure of the generated image remains consistent. However, by changing the activations of the later layers with the new latent vector, additional aspects such as texture and color (which can be referred to as the style of the generated image) can be added or changed. The higher the number of earlier layers that use the initial latent vector, the less change to the generated image (e.g., the more consistent the geometric structure of the generated image), allowing the user to control the degree of randomization by selecting the number of earlier layers. For example, changing to a new latent vector in an earlier layer (e.g., layer 2) results in more change to the geometric structure or spatial structure of the generated image (e.g., the shape of the generated dog), while changing to a new latent vector in a later layer (e.g., layer 5) results in changing the lighting conditions of the generated image or changing the background of the generated image (e.g., changing from a grassy background or sandy background to a snowy background).

[0135] Figure 13 An example 1300 of changing the style of a generated image is illustrated. The example 1300 illustrates an initial image 1302 of a dog generated using an initial latent vector. A new image 1304 is generated by the generative neural network 104 based on a new latent vector generated by the image style module 206 and input to the later layers of the generative neural network 104, which changes the background of the image but keeps the dog substantially the same. Similarly, a new image 1306 is generated by the generative neural network 104 based on another new latent vector generated by the image style module 206 and input to the later layers of the generative neural network 104, which further changes the background of the image but keeps the dog substantially the same.

[0136] Figure 14 Another example generative neural network 1400 is illustrated. The generative neural network 1400 can be, for example, Figure 1 or Figure 2The generative neural network 104 of FIG. The generative neural network 1400 includes a plurality of (n) layers illustrated as layers 1402, 1404, 1406, and 1408 and generates a new image 1410. In general, similar to Figure 8 Each layer of the generative neural network 800 performs one or more operations or functions on the received data and generates output data called activation values ​​or simply activations. However, the example generative neural network 1400 is different from Figure 8 The difference between the generative neural network 800 and the generated neural network 800 is that the initial latent vector 1412 is input to one or more earlier layers of the generated neural network 1400, but the new latent vector 1424 is input to one or more later layers. The earlier layers of the generated neural network 1400 refer to layers of the generated neural network 1400 that are closer to the input of the initial latent vector 1412 than the later layers, and the later layers of the generated neural network 1400 refer to layers of the generated neural network 1400 that are closer to the output of the new image 1410 than the earlier layers.

[0137] To generate a new image 1410, an initial latent vector 1412 is input to a first layer 1402. In layer 1402, one or more functions 1416 are performed on the initial latent vector 1412, which generates various activation values ​​1418. The activation values ​​1418 and the initial latent vector 1412 are provided as input to layer 1404. In layer 1404, one or more functions 1420 are performed on the activation values ​​1418 and the initial latent vector 1412, which generates various activation values ​​1422. The activation values ​​1422 are provided as input to layer 1406.

[0138] The image style module 206 generates a new latent vector 1424. The image style module 206 can generate the new latent vector 1424 in any of a variety of different ways (such as randomly, pseudo-randomly, according to other rules or criteria, etc.). It should be noted that the image style module 206 does not necessarily base the generation of the new latent vector 1424 on Figure 2 230. In the event that the initial latent vector 1412 includes a class vector, the image style module 206 typically includes the same class vector in the new latent vector 1424. Additionally or alternatively, the image style module 206 may generate a new class vector (such as randomly, pseudo-randomly, according to other rules or criteria, etc.) to include in the new latent vector 1424.

[0139] Image style module 206 provides new latent vector 1424 as input to all layers following layer 1404 (e.g., layers 1406 and 1408). In layer 1406, one or more functions 1426 are performed on activation values ​​1422 and new latent vector 1424, and one or more functions 1426 generate various activation values ​​1428. Activation values ​​1428 and new latent vector 1424 are provided as input to the next layer. Finally, the activations from the penultimate layer and new latent vector 1424 are provided as input to layer 1408. In layer 1408, one or more functions 1430 are performed on the activation values ​​received from the previous layer and new latent vector 1424, which generates various activation values ​​1432. Activation values ​​1432 are output as the generated new image 1410.

[0140] This process can be repeated multiple times, with the image style module 206 generating a different new latent vector 1424 each time, thereby producing a different new image 1410 with a different style. Figure 14 An example is illustrated in which the initial latent vector 1412 is provided to the layers 1402 and 1404 , and the new latent vector 1424 is provided to the layer 1406 , but later, the initial latent vector 1412 may be provided to an additional layer (eg, layer 1404 ) instead of the new latent vector 1424 .

[0141] It should be noted that Figure 14 In the example of , the image style module 206 generates a new latent vector 1424 and inputs the new latent vector 1424 into subsequent layers of the generative neural network 1400, but does not modify the activation values ​​generated by any particular layer. This allows the content discussed herein to be performed, for example, even in situations where the activation values ​​in the layers of the generative neural network 1400 are not available for modification, such as when the generative neural network 1400 is implemented in hardware, such as an ASIC.

[0142] Synthetic Image

[0143] The image synthesis module 208 implements functionality for causing the generative neural network 104 to generate an image that is a synthesis of two other images. Such a synthesized image is generated by causing the generative neural network 104 to generate two images (also referred to as source images). These source images can be generated using any of the effects discussed herein, such as transformations, translations, style changes, etc.

[0144] The activation values of the image synthesis module 208 receive, for each of the two source images, activation values for the two source images from a particular layer of the generative neural network 104. As discussed in more detail below, the image synthesis module 208 combines these activations in various ways, such as by using different portions of the activation values from the different source images, by interpolating the activation values, etc. These combined activation values are then input to the next layer of the generative neural network 104, which continues to generate the synthesized image.

[0145] Figure 15 An example of generating a synthesized image is illustrated. Figure 15 A portion of a generative neural network 104, such as Figure 1 or Figure 2 The generative neural network includes multiple layers, illustrated as layer 1506 and layer 1508, and generates a new synthesized image 1510. Although the activation values generated by those earlier layers are not used in generating the new synthesized image 1510 and thus need not be included in the portion 1500, the generative neural network optionally includes additional layers earlier than layer 1506.

[0146] In general, similar to the layers of the generative neural network 300, Figure 3 each layer performs one or more operations or functions on the received data and generates output data, referred to as activation values or simply activations. In the example of the generative neural network 300, Figure 15 The image synthesis module 208 generates the activation values 1512 based on the activation values 1514 received from the generative neural network 104 that generated the first image and the activation values 1516 received from the generative neural network 104 that generated the second image. The activation values 1514 and 1516 can be received from the same generative neural network 104 or, alternatively, from different generative neural networks.

[0147] To generate the new synthesized image 1510, the image synthesis module 208 uses the activation values 1514 and 1516 from the previously generated first and second images to generate the activation values 1512 and provides the activation values 1512 as input to the layer 1506. In the layer 1506, one or more functions 1518 are performed on the activation values 1512 and the one or more functions 1518 generate various activation values 1520. The activation values 1520 are provided as input to the next layer. Eventually, the activation from the penultimate layer is provided as input to the layer 1508. In the layer 1508, one or more functions 1522 are performed on the activation values received from the previous layer, which generates various activation values 1524. The output activation values 1524 are output as the generated new synthesized image 1510.

[0148] In one or more implementations, activation values 1514 and 1516 are received from a particular layer of the generative neural network 104 for two different initial latent vectors 220. These activation values 1514 and 1516 are activation values generated from the same layer as the layer preceding layer 1506 (e.g., layer 2 in the illustrated example). The image synthesis module 208 can operate on the received activation values 1514 and 1516 (e.g., in matrix form). Additionally or alternatively, the image synthesis module 208 can unroll the activation values 1514 into a first image activation vector and unroll the activation values 1516 into a second image activation vector, and operate on these activation vectors. Figure 15

[0149] The image synthesis module 208 can generate the activation values 1512 based on the first image activation values 1514 and the second image activation values 1516 in various ways, in order to generate different effects for the new synthesized image 1510. In one or more implementations, the image synthesis module 208 combines the first image activation values 1514 with the second image activation values 1516 to generate the activation values 1512. This combination is performed by selecting activation values from a portion of the first image activation values 1514 and using those selected activation values as corresponding activation values in the activation values 1512. Similarly, activation values from a portion of the second image activation values 1516 are selected and used as corresponding activation values in the modified activation values 1512.

[0150] Figure 16 An example of combining first image activation values with second image activation values is illustrated. Figure 16 A simplified example of arranging activation values into a 10 by 5 matrix is illustrated. First image activation values 1602 are illustrated, which are obtained for a first initial latent vector 220 from a layer of the generative neural network 104, and second image values 1604 are illustrated, which are obtained for a second initial latent vector 220 from the same layer of the generative neural network 104. A portion 1606 of the activation values 1602 (illustrated with a diagonal line from the top left to the bottom right) is selected to be combined with a portion 1608 of the activation values 1604 (illustrated with a diagonal line from the bottom left to the top right). Modified activation values 1610 are generated by using the activation values in these portions 1606 and 1608 as illustrated.

[0151] Although illustrated as using the left half of the activation values 1602 for the left half of the activation values 1610 and the right half of the activation values 1604 for the right half of the activation values 1610, these can be reversed. For example, the right half of the activation values 1602 can be used for the left half of the activation values 1610, and the left half of the activation values 1604 can be used for the right half of the activation values 1610, which can be reversed.​

[0152] Referring back to Figure 15 , the two portions of the activation values 1514 and 1516 are combined to generate the activation values 1512. The two portions can be any of a variety of different geometric shapes. For example, each portion can be roughly half of the activation values (such as the upper half and the left half, the lower half and the right half). By way of another example, each portion can be roughly a quarter of the activation values (such as the upper right quarter, the upper left quarter, the lower left quarter, and the lower right quarter). By way of another example, one portion can be a circular region of the center of the activation values, and the second portion can be the remaining region of the activation values.

[0153] In one or more implementations, the portions of the activation values from the two source images are user-selected. This can be a user selection of a predetermined portion (such as half or a quarter as discussed above) or a user selection of any of a variety of geometric shapes. For example, a user touch input can be received that draws a circle or other geometric shape on one of the source images. Similar to the discussion above regarding receiving a user selection of an artifact, in response to such user input, the image synthesis module 208 determines which portions (such as pixels) of the source image are identified by the user input, and also determines which activation values correspond to the identified portions of the source image. Thus, the activation values generated from the initial latent vector for that source image are the portions of that source image that are used as modified activation values. The remaining activation values in the modified activation values are those from the other source image.

[0154] In one or more implementations, similar to the discussion above, the style of the new synthesized image 1510 is changed by providing the latent vector 1526 to the layers of the generative neural network portion. Thus, the latent vector 1526 controls the style of the new synthesized image 1510, and the combination of the activation values controls the layout and content of the new synthesized image 1510. In Figure 15 the illustrated example, these later layers are the layer 1506 and the layers after it. The image synthesis module 208 generates the latent vector 1526 based on the latent vectors (also referred to herein as source latent images) that are input to the later layers of the generative neural network used to generate the source images. The image synthesis module 208 combines the source latent images to create the latent vector 1526.

[0155] The source latent images can be combined in a variety of different ways. In one or more implementations, the latent vector 1526 is generated by interpolating between the source latent images (such as using linear interpolation, spherical interpolation, etc.). For example, this provides the new synthesized image 1510 with a style that is set to the mean of the source latent images.

[0156] Additionally or alternatively, the source latent vectors can be combined in different ways. For example, a user input can be received by the image synthesis module 208 that specifies a degree to which each of the source latent vectors is weighted. For example, a user can specify that the style should be 75% of the first source image and 25% of the second source image. The image synthesis module 208 then uses a weighted combination of the source latent vectors that corresponds to the user input. For example, following the previous example, the image synthesis module 208 generates the latent vector 1526 by summing 75% of the source latent vector corresponding to the first source image and 25% of the source latent vector corresponding to the second source image.

[0157] Additionally or alternatively, the source latent vectors need not all be combined. For example, a random number can be used in place of one or more of the source latent vectors, resulting in a new style similar to that discussed above. Thus, in this case, the image synthesis module 208 need not receive one or more of the activation values 1514 and 1516.

[0158] Similar to the discussion above, the image synthesis module 208 can provide activation values for any of the various different layers of the generative neural network 104. Earlier layers correspond to global changes in the new synthesized image 1510, resulting in more gradual, more natural transitions to the human eye. Later layers correspond to local changes with less natural transitions (e.g., that appear to have been cut and pasted together with a pair of scissors). Thus, controlling the degree of sharpness of the transition between the source images (e.g., by user input) is controlled by selecting different layers of the generative neural network 104 for which to modify the activation values.

[0159] Although discussed herein as combining two images, it should be noted that any number of images can similarly be combined to generate a new synthesized image 1510.

[0160] Figure 17 An example 1700 of generating a synthesized image is illustrated. The example 1700 illustrates two source images 1702 and 1704. A new synthesized image 1706 is generated by the generative neural network 104 based on activation values generated by the image synthesis module 208 and input to certain layers of the generative neural network 104 as discussed above. As illustrated, a left portion 1708 of the source image 1704 is the basis for a left portion of the new synthesized image 1706, and a right portion 1710 of the source image 1702 is the basis for a right portion of the new synthesized image 1706.

[0161] Referring back to Figure 15The image synthesis module 208 can optionally generate the activation values 1512 based on the first image activation values 1514 and the second image activation values 1516 in other ways. In one or more implementations, the image synthesis module 208 combines the first image activation values 1514 and the second image activation values 1516 by linearly interpolating between the activation values 1514 and 1516. Each value in the activation values 1512 is generated by interpolating corresponding values in the first image activation values 1514 and the second image activation values 1516, such as using linear interpolation, spherical interpolation, and the like.

[0162] Additionally or alternatively, the image synthesis module 208 can generate the activation values 1512 in other ways. For example, a user input can be received by the image synthesis module 208 that specifies a degree to which each of the source images is to be weighted. For example, a user can specify that the style should be based on 75% of the first source image and 25% of the second source image. The image synthesis module 208 then uses a weighted combination of the activation values 1514 and 1516 that corresponds to the user input. For example, following the previous example, the image synthesis module 208 generates each value in the activation values 1512 by summing 75% of a corresponding value in the first image activation values 1514 and 25% of a corresponding value in the second image activation values 1516.

[0163] With this combination of the activation values 1514 and 1516, for example, such that each of the activation values 1512 is based at least in part on a corresponding value of the activation values 1514 and a corresponding value of the activation values 1516, blended images can be generated. These blended images provide a combination of the corresponding source images in all regions of the image.

[0164] In one or more implementations, similar to the discussion above, the style of the new synthesized image 1510 is changed by providing the latent vector 1526 to the later layers of the portion 1500. Thus, the latent vector 1526 controls the style of the new synthesized image 1510, and the combination of the activation values controls the layout and content of the new synthesized image 1510. In Figure 15 In the illustrated example, these later layers are the layer 1506 and the layers after it. The new latent vector 1526 is generated based on latent vectors that are input to the later layers of the generative neural network used to generate the source images (also referred to herein as source latent images). The image synthesis module 208 combines the source latent vectors to create the latent vector 1526.

[0165] As discussed above, the source latent vectors can be combined in a variety of different ways. In one or more implementations, the latent vector 1526 is generated by linearly interpolating between the source latent vectors, such as using linear interpolation, spherical interpolation, and the like. Additionally or alternatively, the source latent vectors can be combined in different ways. For example, user input specifying the degree to which each of the source latent vectors is weighted can be received by the image synthesis module 208 and used to generate the latent vector 1526. Additionally or alternatively, the source latent vectors need not all be combined. For example, a random number can be used in place of one or more of the source latent vectors.

[0166] Figure 18 Another example 1800 of generating a synthetic image is illustrated. The example 1800 illustrates two source images 1802 and 1804. The source image 1802 is generated using the category vector "tiger" and the source image 1804 is generated using the category vector "owl." The new synthetic image 1806 is generated by the generative neural network 104 based on the activation values generated by the image synthesis module 208 and input to certain layers of the generative neural network 104 as discussed above. As illustrated, the new synthetic image 1806 is a hybrid image that is a combination of the tiger of the source image 1802 and the owl of the source image 1804.

[0167] Referring back to Figure 15 It is noted that the same or different latent vectors can be used as input latent vectors in generating the source images. Although different input latent vectors can be used, using the same input latent vectors for the source images generally results in better alignment of the objects in the source images. Additionally, the same or different category vectors can be used as input category vectors in generating the source images according to the desired effect.

[0168] Referring back to Figure 2 It is noted that any of the various techniques discussed herein can be combined. For example, the various translation, transform clean-up, simplification, and the like techniques discussed with respect to the image editing module 204 can be performed in conjunction with the effects discussed with respect to one or both of the image style module 206 and the image synthesis module 208. These different techniques can be performed at the same level or different levels of the generative neural network 104. For example, the various translation, transform clean-up, simplification, and the like techniques discussed with respect to the image editing module 204 can be performed in one layer, while the effects discussed with respect to one or both of the image style module 206 and the image synthesis module 208 can be performed in different layers of the generative neural network 104.

[0169] It should also be noted that in cases where an image has already been generated and the intermediate latent space between layer X and layer X+1 is being modified when generating a new image, the previously generated activation values for layer X and earlier layers can but need not be regenerated. Rather, the previously generated activation values for layer X and earlier layers can have been stored previously so that they can be retrieved when generating a new image.

[0170] Example Process

[0171] The following discussion describes techniques that can be implemented utilizing the previously described systems and devices. Aspects of the processes can be implemented in hardware, firmware, software, or combinations thereof. The processes are shown as a collection of blocks in a logical flow graph, which can be Figures 1-18 .

[0172] Figure 19 is a flow diagram 1900 that depicts a process in an example implementation of controlling a neural network through an intermediate latent space. In this example, first data is received for a generator network, which includes multiple layers (block 1902). The generator network is at least a generator portion of, for example, a GAN. The multiple layers of the generator network include an initial layer, a first layer, and a second layer. The first layer is later in the generator network than the initial layer (input layer), and the second layer is later in the generator network than the first layer. For example, the first data includes a latent vector that is input to the initial layer of the generator network, or an activation value that is generated by the first layer of the generator network.

[0173] An input selection is received for a first effect on a new image generated by the generator network (block 1904). The input selection can be, for example, a user selection or user request, a selection or request from another device or system, etc.

[0174] Second data is generated by modifying the first data based on the input selection (block 1906). The modification includes modifying the latent vector that is input to the initial layer, or modifying the activation value generated by the first layer, such as by applying a decomposition vector to the activation value generated by the first layer, applying a decomposition vector to the latent vector that is input to the initial layer, interpolating between activation values generated by the first layer from two source image latent vectors, selecting from among activation values generated by the first layer from two source image latent vectors, or any combination thereof.

[0175] The second data is provided to the second layer (block 1908). For example, the second data is provided to the second layer instead of the first data.

[0176] The generator network is used to generate new images with the first effect using the second data (block 1910). Although a first effect is discussed, a variety of effects can be applied to the new images as discussed above.

[0177] Example System and Device

[0178] Figure 20 An example system is illustrated generally at 2000, including an example computing device 2002 that represents one or more computing systems and / or devices that can implement various techniques described herein. This is illustrated by including a generative neural network 104 and a generative neural network control system 106. The computing device 2002 can be, for example, a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computing device or computing system.

[0179] The example computing device 2002 as illustrated includes a processing system 2004, one or more computer-readable media 2006, and one or more I / O interfaces 2008 that are communicatively coupled with one another. Although not shown, the computing device 2002 can also include a system bus or other data and command transfer system that couples the various components within the computing device 2002 to one another. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a serial bus, and / or a local bus, using any of a variety of bus architectures. Various other examples can also be contemplated, such as control lines and data lines.

[0180] The processing system 2004 is representative of the functionality performed by hardware as an processing system 2004. Accordingly, the processing system 2004 is illustrated as including hardware elements 2010 that can be configured to perform a variety of functions as a processor, functional blocks, and so forth. This can include use of a dedicated integrated circuit and / or other logic device formed using one or more semiconductors, which can be fabricated using a process. The hardware elements 2010 are not limited to a particular physical structure, and can include any of a variety of components operating together to perform a functionality. For example, the hardware elements 2010 can include a processor, such as a central processing unit (CPU), a microcontroller, an integrated circuit, a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other processing logic that can in part be fabricated into a

[0181] The computer-readable storage medium 2006 is illustrated as including memory / storage 2012. The memory / storage 2012 represents the memory / storage capacity associated with one or more computer-readable media. The memory / storage 2012 can include volatile media (such as RAM) and / or non-volatile media (such as read only memory (ROM), floppy disk, etc.). The memory / storage 2012 can include fixed and removable media, for example RAM, ROM, a fixed hard disk drive, etc. The computer-readable media 2006 can be configured in a variety of other ways as further described below.

[0182] The input / output interface(s) 2008 are representative of functionality to allow a user to enter commands and information to computing device 2002, and further allow information to be presented to the user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, cursor control device (e.g., a mouse), microphone, scanner, touch functionality (e.g., capacitive or other sensors that are configured to detect physical touch), a camera (e.g., which can employ visible or non-visible wavelengths such as infrared frequencies to recognize movement as gestures that do not involve touch), and so forth. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, tactile-response devices, and so forth. Thus, the computing device 2002 can be configured in a variety of ways as further described below to support user interaction.

[0183] In this document, various technologies are described in the general context of software, hardware elements, or program modules. Generally, such modules include routines, programs, objects, elements, components, data structures, and so forth that perform particular tasks or implement particular abstract data types. As used herein, the terms "module," "functionality," and "component" generally represent software, firmware, hardware, or a combination thereof. The features of the technologies described herein are platform-independent, meaning that the technologies can be implemented on a variety of commercial computing platforms having a variety of processors.

[0184] Implementations of the described modules and techniques can be stored or transmitted using some form of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 2002. By way of example, and not limitation, computer-readable media can comprise "computer-readable storage media" and "computer-readable signal media."

[0185] "Computer-readable storage media" refers to media and / or devices that enable persistent storage of information and / or dynamic storage of information in a computer- readable format that is readable by a computer. The computer-readable storage media is a non- transitory machine-readable medium. The computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in a method or technology for storage of information such as computer readable instructions, data structures, program modules, logical elements / circuits, or other data. Examples of computer-readable storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture that are appropriate for storage of desired information and that are accessed by a computer.

[0186] "Computer-readable signal media" refers to a signal-bearing medium that is configured to transmit computer readable instructions to the hardware of the computing device 2002, such as via a network. The signal media typically can be implemented in analogue and / or digital formats. The computer-readable instructions are typically implemented in the form of software programs, software modules, data files, data structures, or other data. The computer-readable signal media includes any information delivery tense. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0187] As previously described, hardware elements 2010 and computer-readable media 2006 are representative of modules, programmable device logic and / or fixed device logic implemented in the hardware form that can be employed in some embodiments to implement at least portions of the technology described herein, such as to perform one or more instructions. Hardware can include components of an integrated circuit or on-chip system, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations in silicon or other hardware. In this context, hardware can operate as a processing device that performs program tasks defined by instructions and / or logic that are implemented in the hardware. Hardware can also operate as a hardware logic component that provides the function(s) of the module upon the occurrence of one or more events.

[0188] Combinations of the above can also be employed to implement the various techniques described herein. Accordingly, software, hardware, or executable modules can be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage media and / or by one or more hardware elements 2010. The computing device 2002 can be configured to implement particular instructions and / or functions corresponding to the software and / or hardware modules. Accordingly, implementation of a module that is executable by the computing device 2002 as software can be achieved at least partially in hardware, for example, through use of computer-readable storage media and / or hardware elements 2010 of the processing system 2004. The instructions and / or functions can be executable / operable by one or more articles of manufacture (for example, one or more computing devices 2002 and / or processing systems 2004) to implement techniques, modules, and examples described herein.

[0189] The techniques described herein can be supported by various configurations of the computing device 2002 and are not limited to the specific examples of the techniques described herein. Functionality can also be implemented all or in part within the platform 2016 via "cloud" 2014 described below.

[0190] The cloud 2014 includes and / or is representative of the platform 2016 for resources 2018. The platform 2016 abstracts underlying functionality of hardware (for example, servers) and software resources of the cloud 2014. The resources 2018 can include applications and / or data that can be utilized while computer processing is executed on servers that are remote from the computing device 2002. Resources 2018 can also include services provided over the Internet and / or through a subscriber network, such as a cellular or Wi-Fi network.

[0191] The platform 2016 can abstract resources and functions to connect the computing device 2002 with other computing devices. The platform 2016 can also serve to abstract scaling of resources to provide correspondingly scalable levels of access to the resources 2018 provided by the platform 2016. In the interconnected device embodiment, implementation of functionality described herein can therefore be distributed throughout the system 2000. For example, the functionality can be implemented in part on the computing device 2002 as well as via the platform 2016 that abstracts the functionality of the cloud 2014.

[0192] CONCLUSION

[0193] While the application has been described in language specific to structural features and / or methodological acts, it is to be understood that the application defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed application.

Claims

1. A method implemented by at least one computing device in a digital content creation digital media environment, the method comprising: Receiving first data for a generator network of a generative adversarial network (GAN), the generator network having a plurality of layers, the plurality of layers including an initial layer, a first layer, and a second layer, wherein the first data includes a latent vector input to the initial layer or an activation value generated by the first layer; receiving an input selection of a first effect for a new image generated by the generator network; generating second data by modifying the first data based on the input selection, the modifying comprising: applying a decomposition vector to an activation value generated by the first layer, wherein the decomposition vector is generated based on the activation value; providing the second data to the second layer, which is a later layer than the first layer in the generator network; and The new image having the first effect is generated using the generator network and the second data.

2. The method of claim 1, wherein the first data comprises the activation values ​​generated by the first layer, the method further comprising: obtaining a plurality of sets of activation values ​​from the first layer of the generator network, each set of activation values ​​being generated from one of a plurality of different latent vectors input to the generator network; generating a plurality of decomposition vectors from the plurality of sets of activation values ​​by performing a decomposition technique on the plurality of sets of activation values; as well as Generating the second data includes generating the second data by applying one of the plurality of decomposition vectors to the first data.

3. The method of claim 2, applying one of the plurality of decomposition vectors to the first data comprises: The one of the plurality of decomposition vectors is added to the first data or subtracted from the first data.

4. The method according to claim 2, generating the second data further comprising: The second data is generated by applying one of the plurality of decomposition vectors to a latent vector input to the initial layer of the generator network.

5. The method of claim 2, applying a decomposition vector from the plurality of decomposition vectors to the first data comprises: The first data is projected onto a set of multiple decomposition vectors. The method of claim 5 , wherein the set of multiple decomposition vectors comprises a set of eigenvectors.

7. The method according to claim 1, further comprising: obtaining a plurality of sets of activation values ​​from the first layer of the generator network, each set of activation values ​​being generated from one of a plurality of different latent vectors input to the initial layer of the generator network; generating a plurality of decomposition vectors from the plurality of sets of activation values ​​by performing a decomposition technique on the plurality of sets of activation values; as well as Generating the second data includes generating the second data by applying one of the plurality of decomposition vectors to the latent vector input to the initial layer.

8. The method of claim 7, wherein applying one of the plurality of decomposition vectors to the latent vector input to the initial layer comprises: The one of the decomposition vectors is added to the latent vector input to the initial layer or subtracted from the latent vector input to the initial layer.

9. The method according to claim 1, generating the second data comprises: The second data is generated by generating a new latent vector different from a latent vector input to the initial layer of the generator network.

10. The method according to claim 9, generating the new latent vector comprises: The new latent vector is randomly generated.

11. The method according to claim 1, generating the second data comprises: obtaining a first activation value generated by the first layer of the generator network from a first source image latent vector; obtaining a second activation value generated by the first layer of the generator network from a second source image latent vector; generating the second data by interpolating between the first activation value and the second activation value; as well as Modifying the first data includes replacing the first data with the second data.

12. The method according to claim 11, generating the second data further comprising: The second data is generated by generating a new latent vector different from a latent vector input to the initial layer of the generator network.

13. A computing device in a content creation digital media environment, comprising: processor; as well as A computer-readable storage medium having a plurality of instructions of an application stored thereon, wherein the instructions, in response to execution by the processor, cause the processor to perform operations, the operations comprising: Receiving first data for a generator network of a generative adversarial network (GAN), the generator network having a plurality of layers, the plurality of layers including an initial layer, a first layer, and a second layer, wherein the first data includes a latent vector input to the initial layer or an activation value generated by the first layer; receiving an input selection of a first effect for an image generated by the generator network; generating second data based on the first data and the input selection, wherein generating the second data is based on modifying activation values ​​generated by the first layer and modifying latent vectors input to the initial layer; providing the second data to the second layer, which is a later layer than the first layer in the generator network; and The image having the first effect is generated using the generator network and the second data.

14. The computing device of claim 13 , wherein modifying the activation values ​​generated by the first layer comprises: A decomposition vector is applied to the activation values ​​generated by the first layer.

15. The computing device of claim 14, the operations further comprising: obtaining a plurality of sets of activation values ​​from the first layer of the generator network, each set of activation values ​​being generated from one of a plurality of different latent vectors input to the generator network; generating a plurality of decomposition vectors from the plurality of sets of activation values ​​by performing a decomposition technique on the plurality of sets of activation values; as well as Modifying the activation values ​​generated by the first layer includes applying a decomposition vector from the plurality of decomposition vectors to the activation values ​​generated by the first layer and the latent vector.

16. The computing device of claim 15, wherein applying one of the plurality of decomposition vectors comprises: The one of the decomposition vectors is added to or subtracted from the activation values ​​and the latent vector generated by the first layer.

17. The computing device of claim 13, wherein generating the second data comprises: The second data is generated by generating a new latent vector different from a latent vector input to the initial layer of the generator network.

18. The computing device of claim 13, wherein generating the second data comprises: obtaining a first activation value generated by the first layer of the generator network from a first source image latent vector; obtaining a second activation value generated by the first layer of the generator network from a second source image latent vector; as well as The second data is generated by selecting a portion of the first activation value as a first portion of the second data, selecting a portion of the second activation value as a second portion of the second data, and replacing the first data with the second data.

19. The computing device of claim 13, wherein generating the second data comprises: obtaining a first activation value generated by the first layer of the generator network from a first source image latent vector; obtaining a second activation value generated by the first layer of the generator network from a second source image latent vector; generating the second data by interpolating between the first activation value and the second activation value; as well as Modifying the activation value includes replacing the activation value generated by the first layer with the second data.

20. A system comprising: A generator network of a generative adversarial network (GAN), the generator network having a plurality of layers, the plurality of layers comprising an initial layer, a first layer, and a second layer; an input module implemented at least in part in hardware, the input module to receive an input selection of a first effect for an image generated by the generator network; An image editing module, implemented at least partially in hardware, configured to: receiving first data, the first data comprising a latent vector input to the initial layer or an activation value generated by the first layer, the first layer being a layer later than the initial layer in the generator network, modifying the first data based on the input by applying a decomposition vector generated based on the activation value to the activation value generated by the first layer, thereby obtaining second data, and providing the second data instead of the first data to the second layer, the second layer being a later layer than the first layer in the generator network; And the generator network generates the image with the first effect using the second data.

Images