Image transformation using interpretable transformation parameters

Abstract

A computer-implemented method of performing an image-to-image conversion. The method may include obtaining one or more machine learning generator models. The one or more machine learning generator models may be configured to receive an input image and a user-specified condition vector that parameterizes one or more expected values ​​for one or more defined characteristics of an output image. The one or more machine learning generator models may be configured to perform one or more transformations on the input image based at least in part on the user-specified condition vector to generate an output image having one or more expected values ​​for the one or more defined characteristics. The method may include receiving an input image and a user-specified condition vector. The method may include using the machine learning generator model to generate an output image having one or more expected values ​​for the one or more characteristics.

Description

Technical Field

[0001] The present disclosure generally relates to using (multiple) machine learning models to perform image transformations according to interpretable user-specified parameters. Background Art

[0002] Image transformations have been typically performed manually using image manipulation software. As an example, an image depicting an environment in summer can be transformed into an image depicting the same environment in winter using image manipulation techniques. Recently, machine learning models (e.g., neural networks, etc.) have been trained and used to perform these image transformations.

[0003] However, these image transformation models generally do not provide fine-grained control over the specific interpretable properties of the image to be transformed. For example, such image transformation models generally do not provide control over the extent to which the transformation is applied. Furthermore, such image transformation models generally can only provide a single type of image transformation. Summary of the invention

[0004] Aspects and advantages of the embodiments of the present disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0005] One example aspect of the present disclosure is directed to a computer-implemented method for performing image-to-image conversion. The method may include obtaining a machine learning generator model by a computing system including one or more computing devices. The machine learning generator model may be configured to receive an input image and a user-specified condition vector that parameterizes one or more expected values ​​for one or more defined characteristics of an output image. The machine learning generator model may also be configured to perform one or more transformations on the input image based at least in part on the user-specified condition vector to generate an output image having one or more expected values ​​for one or more defined characteristics. The method may include receiving, by the computing system, an input image and a user-specified condition vector. The method may include generating, by the computing system, an output image having one or more expected values ​​for one or more characteristics using the machine learning generator model.

[0006] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0007] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0009] Figure 2 A block diagram depicts an example computing device according to an example embodiment of the present disclosure.

[0010] Figure 3 A block diagram depicts an example computing device according to an example embodiment of the present disclosure.

[0011] Figure 4 A graphical diagram depicting an example machine learning model and associated training signals for output image generation according to an example embodiment of the present disclosure.

[0012] Figure 5 A graphical diagram depicting an example machine learning model and associated training signals for reconstructing input image generation according to an example embodiment of the present disclosure.

[0013] Fig. 6A Depicting a graphical representation of an example day-to-night image transformation with different user desired values ​​applied to defined characteristics of an input image according to an example embodiment of the present disclosure.

[0014] Figure 6B Depicting graphical representations of example light source image transformations applied to one or more defined characteristics of an input image at different user desired values ​​according to example embodiments of the present disclosure.

[0015] Figure 6C Depicting graphical representations of example day-to-night and weather transformations simultaneously applied to defined characteristics of an input image at different user desired values ​​according to an example embodiment of the present disclosure.

[0016] Figure 7 is a flow chart depicting an example method of generating a reconstruction of an input image according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Overview

[0018] Example embodiments of the present disclosure are directed to performing transformations of images using (multiple) machine learning models and interpretable user-specified parameters that control the transformations. More specifically, the machine learning model can be used to perform (multiple) transformations on an image based on (multiple) different parameters of a user-specified condition vector. A user can specify desired values ​​for defined characteristics (e.g., transformation area, degree of transformation, etc.) for one or more transformations of (multiple) defined characteristics (e.g., transforming summer to winter, transforming daytime lighting to nighttime lighting, transforming the position of a light source, etc.). The user's desired values ​​can be parameterized in the user-specified condition vector. As an example, a user may wish to transform an image depicting a summer environment into an image depicting a winter environment. The user-specified condition vector may include parameters indicating the type (s) of the user's desired transformation (e.g., summer to winter, etc.) and / or the desired degree of transformation (e.g., full winter transformation, partial winter transformation, etc.).

[0019] (Multiple) machine learning models can receive an input image and a user-specified condition vector, and transform the input image based on the user's desired values ​​for defined characteristics of the output image. Specifically, the output image can correspond to the input image transformed to have the desired values ​​for the (multiple) defined characteristics. Thus, in the example given above, if the user-specified condition vector indicates a 60% transformation of the input image depicting a summer environment, the output image may depict the same scene, but 60% is transformed to a winter environment. In this manner, the machine learning model can perform one or more transformations on the input image based on the user's desired values ​​for the defined characteristics of the (multiple) transformations. In other words, a user can be provided with continuous fine-grained control over specific interpretable characteristics of the transformed image.

[0020] More specifically, existing image transformation models generally fail to provide users with control over desired values ​​of characteristics used for image transformations. As an example, methods such as Pix2Pix provide binary image transformations (e.g., complete transformation or no transformation), but fail to provide finer-grained control over the characteristics of the transformation (e.g., degree of transformation, etc.) (see Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, Image-to-Image Translation with Conditional Adversarial Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134 (2017)). In addition, methods such as Pix2Pix necessitate the use of paired training data (e.g., paired training data with known ground truth correspondences) (see Isola et al. above).

[0021] In response to these problems, the present disclosure proposes an adversarial training method for machine learning, which combines the use of user-specified conditional vectors to provide parameterized image transformations without the need for paired training data. More specifically, in one example, the present disclosure proposes an adversarial network (e.g., a generative adversarial network, etc.) that utilizes one or more machine learning generator models and one or more machine learning discriminator models. One or more machine learning generator models can be configured to receive an input image and a user-specified conditional vector. The user-specified conditional vector can parameterize one or more expected values ​​for one or more defined characteristics of an output image. The machine learning generator model can be configured to perform one or more transformations on the input image based at least in part on the user-specified conditional vector to generate an output image with expected values ​​for the defined characteristics.

[0022] One or more machine learning discriminator models can be configured to receive an output image, a target image, and a user-specified condition vector. The (multiple) discriminator models can also be configured to generate a discriminator output that selects one of the output image and the target image as having one or more expected values ​​for one or more defined characteristics. At least one of the (multiple) machine learning generator models can be trained based at least in part on the discriminator output. Thus, in this manner, the (multiple) machine learning generator models can be trained to apply one or more transformations to the input image according to the expected values ​​of the (multiple) defined characteristics of the output image.

[0023] More specifically, one or more machine learning generator models can be configured to receive an input image and a user-specified condition vector. The user-specified condition vector can parameterize one or more desired values ​​for one or more defined characteristics of the output image. In some embodiments, the user-specified condition vector can be a real, continuous-valued vector. As an example, the vector can include values ​​(e.g., parameterized user desired values) that are both real and continuous (e.g., 0.5, 1, .2235, -5, etc.) that correspond to one or more desired characteristics of the output image.

[0024] In some embodiments, a user-specified condition vector may describe the extent to which one or more transformations are applied to an input image. As an example, a user-specified condition vector may include a parameterized user desired value (e.g., 1, 0, 0.5, etc.) describing one or more transformations to be performed and the desired extent to which the transformation(s) (e.g., transforming daytime lighting characteristics to nighttime lighting characteristics, etc.) should be performed. For example, a condition vector may describe a nighttime to daytime transformation to be applied at a value of 0.5, thereby producing a partially transformed output image. As another example, a condition vector may describe a summer to winter transformation to be applied at a value of 1.0, thereby producing a fully transformed output image.

[0025] One or more machine learning generator models may be configured to perform one or more transformations on an input image based at least in part on a user-specified condition vector to generate an output image having one or more expected values ​​for one or more defined characteristics. The user-specified condition vector may specify the transformation(s) to be performed on the input image. As an example, the user-specified condition vector may specify a summer to winter transformation to be performed on the input image (e.g., transforming the environment depicted in the input image from a summer environment to a winter environment). As another example, the user-specified condition vector may specify a summer to winter transformation and a day to night transformation (e.g., transforming the time of day depicted in the input image from daytime lighting characteristics to nighttime lighting characteristics). The user-specified condition vector may describe multiple transformations and multiple defined characteristics associated with each corresponding transformation.

[0026] In some embodiments, the user-specified condition vector may specify one or more regions of the input image to which one or more transformations are applied. As an example, the user-specified condition vector may include (multiple) values ​​specifying that (multiple) transformations should be applied to the upper half of the input image. As another example, the user-specified condition vector may include (multiple) values ​​specifying that the transformation should be applied to the lower left portion of the input image. It should be noted that the user-specified condition vector may be multi-dimensional, and therefore one or more regions of the input image may be specified in one or more dimensions. As an example, in some embodiments, the user-specified condition vector may describe a transformation that applies a light source to an image. The user-specified condition vector may also specify the location of the light source in three dimensions (e.g., x-coordinate, y-coordinate, z-coordinate, etc.).

[0027] The defining characteristics of the output image may describe one or more features, style, content, or any other aspect of the output image. In some embodiments, the defining characteristics may include style characteristics that describe one or more style transformations to be performed on the input image. As an example, a digital image generated by a digital camera may be transformed into an oil painting (e.g., an image-to-painting transformation). The machine learning generator model may be trained so that the style transformation can replicate various styles of known artists (e.g., Monet, Van Gogh, Picasso, etc.). For example, a digital input image generated by a digital camera may be transformed into an output image that replicates the style of Picasso. It should be noted that the above style transformation may also be performed in reverse by the machine learning generator model (e.g., transforming a Picasso painting into a "digital image" style). Both of the above style transformations may be applied to the input image to a certain degree. For example, an image-to-painting transformation may be applied to the input image with an expected value of 0.5, which may produce an output image that is partially transformed into the style of an oil painting. The style characteristics may describe any kind of subjective and / or objective transformation to be applied to the image (e.g., oil painting style, drawing style, animation style, camera focal length / focus, etc.).

[0028] In some embodiments, the defining characteristics may include light characteristics and / or light source position characteristics. As an example, a digital image depicting a face may be transformed so that a light source is presented in the image projection light at a specific intensity. As previously described, the light source may be placed three-dimensionally during the transformation. For example, the desired value of the light source may place the light source behind the face at an intensity of 0.2. For another example, the desired value of the light source may place the light source in front of and to the right of the face at an intensity of 0.8. In this manner, the light source transformation may illuminate and / or dim various aspects of the input image in a three-dimensional manner. It should be noted that multiple light sources may be included in the light source transformation based on the desired value of the light source transformation characteristics used to define.

[0029] In some embodiments, the defining characteristics may include color characteristics. As an example, an input image lacking color (e.g., black and white, grayscale, etc.) may be transformed into an output image containing color (e.g., an RGB image, a CMYK image, etc.). In some embodiments, the defining characteristics may include environmental characteristics. Environmental characteristics may include any kind of weather effects (e.g., rain, snow, hail, etc.), seasonal effects (summer, autumn, winter, spring, etc.), or any other kind of environment-related transformations (e.g., transforming water bodies into ice during winter, etc.). As an example, an environmental transformation may add rain, snow, or other weather effects to an image. As another example, an environmental transformation may change the seasonal environment of an image from summer to winter (e.g., adding snowfall, changing the color of leaves and / or vegetation, freezing water bodies, changing the angle and intensity of sunlight and / or shadows, etc.).

[0030] In some embodiments, the (multiple) environmental characteristics of the output image may include the time of day. More specifically, the (multiple) environmental characteristics may determine the lighting of the environment depicted in the output image. As an example, an input image acquired at 2:00 pm (e.g., with daytime lighting conditions) in an outdoor environment may be transformed into an output image of the same outdoor environment acquired at 10:00 pm (e.g., with nighttime lighting conditions). In this manner, the environmental characteristics may transform a "daytime" image into a "nighttime" image, and vice versa. It should be noted that the time of day may be transformed in conjunction with other environmental characteristics of the output image. As an example, a summer daytime image may be transformed into a winter nighttime image. As another example, a winter image without rain may be transformed into a summer image depicting rain.

[0031] In some embodiments, the defining characteristics of the output image may include characteristic characteristics. The characteristic characteristics may include features depicted in the input and / or output image (e.g., objects, people, environments, organisms, text, etc.). As an example, the input image may depict a horse in a rural environment. The characteristic characteristics of the output image may include the coloring of the horse. In this manner, for example, the coloring of the horse may be changed to the coloring of a zebra or any other kind of coloring pattern. As another example, the input image may depict a vehicle (e.g., a car, a helicopter, etc.) in the background. The characteristic characteristics of the output image may include changing the color of the vehicle (e.g., red to blue), the type of vehicle (e.g., changing the vehicle from a first type of vehicle to a second type of vehicle), or removing the vehicle. The characteristic characteristics of the output image may include any aspect of any feature of the output image.

[0032] In some embodiments, one or more machine learning generator models may include an encoder portion and a decoder portion. The user-specified conditional vector may be provided to the decoder portion of one or more machine learning generator models. More specifically, a series of layers in the encoder portion (e.g., convolutional layers in a convolutional neural network) may encode an input image into a compressed representation that can be used as an input to a series of residual blocks (e.g., residual blocks of (multiple) residual neural networks). A stack of connection layers may learn to transform the original values ​​of the user-specified conditional vector into an alternative representation. At the lowest layer (e.g., bottleneck layer) of the (multiple) machine learning generator model, the image representation and the alternative representation of the user-specified conditional vector may be mixed by connecting to one or more regions along the feature dimension of the image. A series of convolutional layers may decode the combined (e.g., connected) data into an output image.

[0033] One or more machine learning generator models may be or otherwise include one or more neural networks (e.g., deep neural networks), etc. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a residual neural network, and / or various other types of neural networks. In some embodiments, one or more machine learning generator models may be a residual neural network including connections (e.g., skip connections) between layers of the residual neural network. The connections between layers may be used to send the user-specified conditional vector to a matching layer (e.g., a mirror layer) of the network before the user-specified conditional vector is reduced to a minimum alternative representation. In this manner, the training signal (e.g., an optimization function) may be more efficiently back-propagated through the machine learning generator model.

[0034] The machine learning discriminator model can be configured to receive an output image, a target image, and a user-specified condition vector. The target image can be an image from a set of target images representing a desired output image. As an example, if the output image has been transformed to include defined stylistic characteristics of a Picasso painting, the target image can be an image from a set of Picasso paintings. As another example, if the output image has been transformed to include defined environmental characteristics of a winter environment, the target image can be an image from a set of images depicting a winter environment.

[0035] The one or more machine learning discriminator models may be configured to generate a discriminator output that selects one of an output image and a target image as having one or more expected values ​​for one or more defined characteristics. The (multiple) machine learning discriminator models may consider including a user-specified conditional vector in the transformation. More specifically, the (multiple) machine learning discriminator model may select one of an output image and a target image based on the defined characteristics of the output image based at least in part on the user-specified conditional vector.

[0036] In some embodiments, one or more machine learning discriminator models may include multiple layers configured to evaluate different aspects of the discriminator output. As an example, the initial layer of the (multiple) discriminator model (e.g., the convolutional layer of a convolutional neural network) may operate only on the output image to extract an appropriate deep representation (e.g., a latent space representation of the output image). The (multiple) machine learning discriminator model may transform the user-specified conditional vector through a series of fully connected layers (e.g., the convolutional layers of a convolutional neural network) in a manner similar to the (multiple) machine learning generator model. The learned representations of the input image and the user-specified conditional vector can be mixed by connecting to (multiple) spatial locations along the feature dimension in a manner similar to the (multiple) machine learning generator model. A series of layers (e.g., the convolutional layers of a convolutional neural network) can operate on the connected representations to produce classification scores for (multiple) patches of the output image. The selected output can be obtained by analyzing (multiple) patches of the output image (e.g., performing (multiple) mean average pooling operations, etc.).

[0037] One or more machine learning generator models can be trained based at least in part on the discriminator output using various training or learning techniques such as, for example, back propagation of errors (e.g., truncated back propagation through time). In some embodiments, the machine learning discriminator model(s) and the machine learning generator model(s) can be trained in an adversarial manner (e.g., generative adversarial networks). The specific training signal(s) used to train or retrain the machine learning models and methods for training or retraining the machine learning models are discussed in depth in the following figures.

[0038] One or more machine learning discriminator models may be or otherwise include one or more neural networks (e.g., deep neural networks), etc. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a residual neural network, and / or various other types of neural networks. In some embodiments, the (multiple) machine learning discriminator models and the (multiple) machine learning generator models may be components of a generative adversarial network (GAN). In some embodiments, the one or more machine learning discriminator models may be a residual neural network including connections (e.g., skip connections) between the layers of the residual neural network. The user-specified conditional vector may be sent from the first layer of the network to the matching layer (e.g., mirror layer) through connections between layers before the user-specified conditional vector is transformed into an alternative representation. Alternatively or additionally, in some embodiments, the user-specified conditional vector may be input at other points in the network architecture. As an example, the user-specified conditional vector may be input to the same layer as the input image. As another example, the user-specified conditional vector may be connected to the output of some other (multiple) layers (e.g., after (multiple) transformations are applied to the image, etc.). In this manner, the training signal (e.g., optimization function) can be back-propagated through the (multiple) machine learning discriminator models more efficiently.

[0039] In some embodiments, the (multiple) machine learning generator model may also be configured to receive an output image and a user-specified condition vector. Based at least in part on the output image and the condition vector, the one or more machine learning generator models may perform one or more reconstruction transformations on the output image to generate a reconstructed input image. The one or more reconstruction transformations may be configured to remove one or more transformations applied to the input image. As an example, the output image may include transformed environmental characteristics (e.g., winter to summer, rain to snow, etc.). The (multiple) machine learning generator model may perform one or more reconstruction transformations on the output image to remove one or more transformations (e.g., performing a winter to summer transformation to remove a previous summer to winter transformation, etc.).

[0040] In some embodiments, one or more machine learning discriminator models may also be configured to receive an input image, a reconstructed input image, and a user-specified condition vector. The (multiple) machine learning discriminator models may generate a reconstructed discriminator output that selects one of the reconstructed input image and the input image. One or more generator models may also be trained based at least in part on the reconstructed discriminator output. In this manner, the (multiple) machine learning generator models may be trained to enforce cycle consistency and thereby enhance the generative capabilities of the (multiple) machine learning generator models. The generation of the (multiple) reconstructed discriminator outputs and the associated training uses will be discussed in more detail with reference to the accompanying drawings.

[0041] The present disclosure provides many technical effects and benefits. As an example technical effect and benefit, the system and method of the present disclosure can achieve significant progress in image transformation technology compared to other methods. Although other methods known in the art are generally limited to binary image transformations (e.g., full daytime lighting, full nighttime lighting, etc.), the present disclosure allows the transformation to be applied to the input image to a specific degree specified by the user. This functionality provides the user with significant control over the application of the transformation. As an example, a user can apply a day to night transformation to a lesser extent to reproduce the lighting characteristics of a sunset. As another example, a user can apply a summer to winter transformation to a lesser extent to reproduce the environmental characteristics of early winter.

[0042] As another technical effect and benefit, the (multiple) generalized machine learning generator models of the present disclosure allow training without specific paired training data. More specifically, the (multiple) models of the present disclosure can be trained using training images with labels that only determine the set identity (e.g., a collection of images depicting various winter environments). Other models known in the art necessitate the use of paired training data (e.g., images of an environment acquired in the summer and paired images of the same environment acquired in the winter). Paired training data for the transformations performed by the (multiple) models of the present disclosure may prove almost impossible to collect (e.g., a winter to summer transformation would necessitate waiting for a seasonal change to collect paired training data, etc.). Therefore, the proposed (multiple) models can eliminate the need to spend a lot of time and resources collecting paired training data to train the (multiple) generator models.

[0043] As another example technical effect and benefit, the systems and methods of the present disclosure allow for (multiple) generalized machine learning generator models, which in turn allow multiple transformations to be applied to an input image. Other methods known in the art can typically only be trained to apply one transformation to an input image, or can only be performed in a non-intuitive manner. However, the present disclosure enables (multiple) machine learning generator models to be trained to apply multiple transformations to an input image. As an example, a model can be trained to transform an input image from daytime lighting characteristics and summer environmental characteristics to nighttime lighting characteristics and winter environmental characteristics. In this way, the model can be trained to provide more realistic and complete image transformations than other methods known in the art. In addition, the ability to perform multiple transformations with one model reduces the number of machine learning models required, thereby reducing the use of computing resources (e.g., processor usage, memory usage, etc.) associated with training multiple models.

[0044] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0045] Example devices and systems

[0046] Figure 1 A block diagram of an example computing system 100 is depicted, in accordance with an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and an image transformation computing system 150 communicatively coupled via a network 180.

[0047] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0048] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0049] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model(s) 120 may be or may otherwise include various machine learning models, such as a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, a residual neural network, or other forms of neural networks.

[0050] In some implementations, the machine learning model(s) 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single neural network 120.

[0051] Additionally or alternatively, the machine learning model(s) 140 may be included in, or otherwise stored and implemented by, a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model(s) 140 may be implemented by the server computing system 130 as part of a web service. Thus, the machine learning model(s) 120 may be stored and implemented at the user computing device 102, and / or one or more networks 140 may be stored and implemented at the server computing system 130.

[0052] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input. In some embodiments, the user input may be used to specify one or more desired values ​​for a user-specified condition vector, which will be referenced Figure 3 and Figure 4 Discuss in more detail.

[0053] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0054] In some implementations, server computing system 130 includes, or is otherwise implemented by, one or more server computing devices. In instances where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a serial computing architecture, a parallel computing architecture, or some combination thereof.

[0055] As described above, the server computing system 130 may store or otherwise include (multiple) machine learning models 140. For example, (multiple) machine learning models 140 may be or may otherwise include a feedforward neural network, a deep neural network, a recurrent neural network, a residual neural network, and a convolutional neural network.

[0056] The user computing device 102 and / or the server computing system 130 may train and / or evaluate the machine learning model(s) 120 and / or 140 via interaction with an image transformation computing system 150 communicatively coupled via a network 180. The image transformation computing system 150 may be separate from the server computing system 130, or may be part of the server computing system 130.

[0057] The image transformation computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the image transformation computing system 150 to perform operations. In some embodiments, the image transformation computing system 150 includes one or more server computing devices, or is otherwise implemented by one or more server computing devices.

[0058] The image transformation computing system 150 may include a model trainer 160 that trains and / or evaluates the (multiple) machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, back propagation of errors. In some embodiments, performing back propagation of errors may include performing truncated back propagation through time. The model trainer 160 may perform a number of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0059] Specifically, model trainer 160 can train (multiple) machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 can be (but is not limited to) unpaired training data (e.g., a collection of images that share one or more defining characteristics, such as depicting a winter environment at night).

[0060] The image transformation computing system 150 may also include (multiple) image transformation models 159. In some embodiments, the (multiple) image transformation model 159 may include (multiple) machine learning generator models and (multiple) machine learning discriminator models configured to perform image transformation on an image based on a user-specified condition vector. In some embodiments, the (multiple) machine learning generator models and (multiple) machine learning discriminator models of the (multiple) image transformation model 159 may be trained by the model trainer 160 in an adversarial manner (e.g., as a generative adversarial network (GAN) or the like). The image transformation computing system 150 may also be optionally communicatively coupled with various other devices (e.g., the user computing device 102) to provide (multiple) trained image transformation models (e.g., machine learning generator models, (multiple) machine learning discriminator models, etc.) to the various other devices and / or receive data from the various other devices (e.g., receiving images from the user computing device 102 as inputs to the (multiple) image transformation models 159, sending the transformed images to the computing device 102, etc.).

[0061] Each of the model trainer 160 and the image transformation model(s) 159 may include computer logic for providing the desired functionality. Each of the model trainer 160 and the image transformation model(s) 159 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some embodiments, each of the model trainer 160 and the image transformation model(s) 159 may include a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, each of the model trainer 160 and the image transformation model(s) 159 may include one or more sets of computer executable instructions stored in a tangible computer readable storage medium (e.g., a RAM hard disk or optical or magnetic media).

[0062] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications on the network 180 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0063] Figure 1An example computing system that can be used to implement the present disclosure is shown. Other computing systems can also be used. For example, in some embodiments, the user computing device 102 can include a model trainer 160 and a training data set 162. In these embodiments, the network 120 can be trained and used locally at the user computing device 102. In some of these embodiments, the user computing device 102 can implement the model trainer 160 to personalize the network 120 based on user-specific data.

[0064] Furthermore, although the present disclosure is described with specific reference to neural networks, the systems and methods described herein can be applied to other multi-layer machine learning model architectures.

[0065] Figure 2 Depicted is a block diagram of an example computing device 10 according to an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0066] Computing device 10 includes many applications (e.g., applications 1 to N). Each application contains its own machine learning library and (multiple) machine learning models. For example, each application may include (multiple) machine learning generator models and / or (multiple) machine learning discriminator models. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, image capture applications, image transformation applications, image upload applications, etc.

[0067] like Figure 2 As shown, each application can communicate with many other components of the computing device (such as, for example, one or more sensors, context managers, device state components, and / or additional components). In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to the application.

[0068] Figure 3 Depicted is a block diagram of an example computing device 50 according to an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0069] The computing device 50 includes a number of applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, an image capture application, an image transformation application, an image upload application, etc. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).

[0070] The central intelligence layer includes many machine learning models. For example, Figure 3 As shown, a corresponding machine learning model (e.g., a machine learning generator model, a machine learning discriminator model, etc.) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system of the computing device 50.

[0071] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository for data of computing devices 50. Figure 3 As shown, the central device data layer can communicate with many other components of the computing device (such as, for example, one or more sensors, context managers, device state components, and / or additional components). In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0072] Example training environment and methods

[0073] Figure 4 A graphical diagram depicting an example machine learning model and associated training signals for output image generation according to an example embodiment of the present disclosure. An input image 406 and a user-specified condition vector 404 may be received as input by (multiple) machine learning generator models 408. The input image 406 may be a digital image (e.g., a digital image captured by a digital image capture device, a scanned image, etc.). In some embodiments, as described for example in relation to Figure 5 As discussed in more detail, the input image 406 may be the output of the machine learning generator model(s). The user-specified condition vector 404 may parameterize one or more desired values ​​for one or more defined characteristics of the output image. In some implementations, the user-specified condition vector 404 may be a real and continuously-valued vector. As an example, the vector may include values ​​(e.g., parameterized user desired values) that are both real and continuous (e.g., 0.5, 1, .2235, -5, etc.) that correspond to one or more desired characteristics of the output image.

[0074] In some embodiments, the user-specified condition vector 404 may describe the extent to which one or more transformations are applied to the input image. As an example, the user-specified condition vector 404 may include a parameterized user desired value (e.g., 1, 0, 0.5, etc.) describing the one or more transformations to be performed and the desired extent to which the transformation(s) (e.g., transforming daytime lighting characteristics to nighttime lighting characteristics, etc.) should be performed. For example, the condition vector 404 may describe a night-to-day transformation to be applied at a value of 0.5, thereby producing a partially transformed output image. As another example, the condition vector 404 may describe a summer-to-winter transformation to be applied at a value of 1.0, thereby producing a fully transformed output image.

[0075] In some embodiments, the user-specified condition vector 404 can specify one or more regions of the input image 406 to which one or more transformations are applied. As an example, the user-specified condition vector 404 can include (multiple) values ​​specifying that the (multiple) transformations should be applied to the upper half of the input image 406. As another example, the user-specified condition vector 404 can include (multiple) values ​​specifying that the transformations should be applied to the lower left portion of the input image 406. It should be noted that the user-specified condition vector 404 can be multi-dimensional, and thus can specify one or more regions of the input image 406 in one or more dimensions. As an example, in some embodiments, as will be referred to in Figure 6B As discussed in more detail, the user-specified condition vector 404 may describe a transformation that applies a light source to an input image 406 .

[0076] The (multiple) machine learning generator models 408 may include an encoder portion 408A and a decoder portion 408B. The user-specified conditional vector 404 may be provided to the decoder portion 408B of one or more machine learning generator models 408. More specifically, a series of layers in the encoder portion 408A (e.g., convolutional layers in a convolutional neural network) may encode the input image 406 into a compressed representation that may be used as an input to a series of residual blocks (e.g., residual blocks of (multiple) residual neural networks). A stack of connected layers may learn to transform the original values ​​of the user-specified conditional vector 404 into an alternative representation. At the lowest layer (e.g., bottleneck layer) of the (multiple) machine learning generator model 408, the image representation (e.g., a lower dimensional representation of the input image 406) and the alternative representation of the user-specified conditional vector 404 may be mixed by connecting to one or more regions along the feature dimension of the input image 406. A series of convolutional layers may decode the combined (e.g., connected) data into an output image 412.

[0077] One or more machine learning generator models 408 may be or otherwise include one or more neural networks (e.g., deep neural networks), etc. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a residual neural network, and / or various other types of neural networks. In some embodiments, one or more machine learning generator models 408 may be a residual neural network including connections (e.g., skip connections) between layers of the residual neural network. The connections between layers may be used to send the user-specified conditional vector to a matching layer (e.g., a mirror layer) of the network before the user-specified conditional vector is reduced to a minimum alternative representation. In this manner, the training signal (e.g., the discriminator output 414) may be more efficiently back-propagated through the (multiple) machine learning generator models.

[0078] The machine learning generator model(s) 408 may be configured to perform one or more transformations on the input image 406 based at least in part on the user-specified condition vector 404 to generate an output image 412 having one or more desired values ​​for one or more defined characteristics of the output image 412. The user-specified condition vector 404 may specify the transformation(s) to be performed on the input image 406. As an example, the user-specified condition vector 404 may specify application of a summer-to-winter transformation to the input image 406 (e.g., transforming an environment depicted in the input image from a summer environment to a winter environment). As another example, the user-specified condition vector 404 may specify a summer-to-winter transformation and a day-to-night transformation (e.g., transforming a time of day depicted in the input image from daytime lighting characteristics to nighttime lighting characteristics). The user-specified condition vector 404 may describe a plurality of transformations and a plurality of defined characteristics associated with each respective transformation.

[0079] The defined characteristics of the output image 412 may describe one or more features, style, content, or any other aspect of the output image 412. In some embodiments, the defined characteristics may include style characteristics that describe one or more style transforms to be performed on the input image 406. As an example, the input image 406 (e.g., a digital image generated by a digital camera) may be transformed into a painting (e.g., an image to painting transformation). The (multiple) machine learning generator models 408 may be trained so that the style transform can replicate various styles of known artists (e.g., Monet, Van Gogh, Picasso, etc.). For example, a digital input image 406 generated by a digital camera may be transformed into an output image 412 that replicates the style of Picasso. It should be noted that as will be referred to in Figure 5As discussed in more detail, the style transformation described above may also be performed in reverse by the machine learning generator model(s) 408 (e.g., transforming a Picasso painting to a "digital image" style). Both of the above style transformations may be applied to the input image 406 to a certain degree. For example, the image-to-painting transformation may be applied to the input image 406 with an expected value of 0.5, which may produce an output image 412 that is partially transformed to the style of a painting. The style characteristics may describe any kind of subjective and / or objective transformation to be applied to an image (e.g., a painting style, a drawing style, an animation style, a camera focus / focal length, etc.).

[0080] In some embodiments, the defined characteristics may include light characteristics and / or light source position characteristics. As an example, the digital image 406 depicting a face may be transformed so that the light source appears in the image projection light at a specific intensity. Figure 6B As discussed in more detail, the light sources can be placed three-dimensionally during the transformation. For example, the desired value of the light source can place the light source behind the face at an intensity of 0.2. As another example, the desired value of the light source can place the light source in front of and to the right of the face at an intensity of 0.8. In this manner, the light source transformation can illuminate and / or dim various aspects of the input image 406 in a three-dimensional manner. It should be noted that multiple light sources can be included in the light source transformation based on the desired value used to define the light source transformation characteristics.

[0081] In some embodiments, the defining characteristics may include color characteristics. As an example, an input image 406 lacking color (e.g., black and white, grayscale, etc.) may be transformed into an output image 412 containing color (e.g., an RGB image, a CMYK image, etc.). In some embodiments, the defining characteristics may include environmental characteristics. Environmental characteristics may include any kind of weather effects (e.g., rain, snow, hail, etc.), seasonal effects (summer, autumn, winter, spring, etc.), or any other kind of environment-related transformations (e.g., transforming a body of water into ice during winter, etc.). As an example, an environmental transformation may add rain, snow, or other weather effects to the output image 412. As another example, an environmental transformation may change the seasonal environment of the input image 406 from summer to winter (e.g., adding snowfall, changing the color of leaves and / or vegetation, freezing a body of water, changing the angle and intensity of sunlight and / or shadows, etc.).

[0082] In some embodiments, the (multiple) environmental characteristics of the output image 412 can include the time of day. More specifically, the (multiple) environmental characteristics can determine the lighting of the environment depicted in the output image 412. As an example, an input image 406 acquired in an outdoor environment at 2:00 p.m. (e.g., with daytime lighting conditions) can be transformed into an output image 412 of the same outdoor environment acquired at 10:00 p.m. (e.g., with nighttime lighting conditions). In this manner, the environmental characteristics can transform a "daytime" image into a "nighttime" image, and vice versa. It should be noted that the time of day can be transformed in conjunction with other environmental characteristics of the output image. As an example, a summer daytime image can be transformed into a winter nighttime image. As another example, a winter image without rain can be transformed into a summer image depicting rain.

[0083] In some embodiments, the defining characteristics of the output image 412 may include characteristic characteristics. The characteristic characteristics may include features (e.g., objects, people, environments, organisms, text, etc.) depicted in the input image 406 and / or the output image 412. As an example, the input image 406 may depict a horse in a rural environment. The characteristic characteristics of the output image 412 may include the coloring of the horse. In this manner, for example, the coloring of the horse may be changed to the coloring of a zebra or any other kind of coloring pattern. As another example, the input image 406 may depict a vehicle (e.g., a car, a helicopter, etc.) in the background. The characteristic characteristics of the output image 412 may include changing the color of the vehicle (e.g., red to blue), the type of the vehicle (e.g., changing the vehicle from a first type of vehicle to a second type of vehicle) or removing the vehicle. The characteristic characteristics of the output image 412 may include any aspect of any feature of the output image 412.

[0084] The machine learning discriminator model 410 may be configured to receive an output image 412, a target image 402 from a set of target images, and a user-specified condition vector 404. The target image 402 may be an image from a set of target images that represents a desired output image 412. As an example, if the output image 412 has been transformed to include defined stylistic characteristics of a Picasso painting, the target image 402 may be an image from a set of Picasso paintings. As another example, if the output image 412 has been transformed to include defined environmental characteristics of a winter environment, the target image 402 may be an image from a set of images that depict a winter environment.

[0085] The machine learning discriminator model(s) 410 can be configured to generate a discriminator output 414 that selects one of the output image 412 and the target image 402 as having one or more desired values ​​for one or more defined characteristics. The machine learning discriminator model(s) 410 can consider including the user-specified conditional vector 404 in the transformation. More specifically, the machine learning discriminator model(s) 410 can select one of the output image 412 and the target image 402 based on the defined characteristics of the output image 412 based at least in part on the user-specified conditional vector 404.

[0086] The one or more machine learning discriminator models 410 may include multiple layers configured to evaluate different aspects of the discriminator output 414. As an example, the initial layers (e.g., convolutional layers of a convolutional neural network) of the (multiple) machine learning discriminator model(s) 410 may operate only on the output image 412 to extract an appropriate deep representation (e.g., a latent space representation of the output image 412). The (multiple) machine learning discriminator model(s) 410 may transform the user-specified conditional vector 404 through a series of fully connected layers (e.g., convolutional layers of a convolutional neural network) in a manner similar to the (multiple) machine learning generator model(s) 408. The learned representations of the input image 406 and the user-specified conditional vector 404 may be blended by concatenating to (multiple) spatial locations along a feature dimension in a manner similar to the (multiple) machine learning generator model(s) 408. A series of layers (e.g., convolutional layers of a convolutional neural network) may operate on the concatenated representations to produce classification scores for (multiple) patches of the output image 412. The selected output (eg, discriminator output 414 ) may be obtained by analyzing the patch(es) of the output image 412 (eg, performing mean average pooling operation(s), etc.).

[0087] One or more machine learning discriminator models 410 may be or otherwise include one or more neural networks (e.g., deep neural networks), etc. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a residual neural network, and / or various other types of neural networks. In some embodiments, the (multiple) machine learning discriminator models 410 and the (multiple) machine learning generator models 408 may be components of a generative adversarial network (GAN). In some embodiments, the one or more machine learning discriminator models 410 may be a residual neural network including connections (e.g., skip connections) between layers of the residual neural network. The user-specified conditional vector 404 may be sent from the first layer of the network to the matching layer (e.g., mirror layer) through connections between layers before the user-specified conditional vector 404 is reduced to the lowest alternative representation. In this manner, the training signal (e.g., discriminator output 414) may be evaluated using an objective function and more efficiently back-propagated through the (multiple) machine learning discriminator models 410. As an example, the (multiple) machine learning discriminator model(s) 410 may be trained on the difference between real data associated with the discriminator output 414 (e.g., a defined transform quality of the output image 412 ) and the discriminator output 414 from the (multiple) machine learning discriminator model(s) 410 .

[0088] One or more machine learning generator models 408 can be trained using various training or learning techniques (e.g., back propagation of errors (e.g., truncated back propagation through time)) based at least in part on an objective function evaluation of the discriminator output 414. In some embodiments, the (multiple) machine learning discriminator model 410 and the (multiple) machine learning generator model 414 can be trained in an adversarial manner (e.g., a generative adversarial network). As an example, in some embodiments, training the (multiple) machine learning generator model based on the discriminator output 414 can include performing stochastic gradient descent to optimize an objective function that evaluates the discriminator output 414 (e.g., minimize the discriminator output 414, etc.). In this manner, the (multiple) machine learning discriminator model 410 can be trained to optimize the value of the discriminator output 414, while the (multiple) machine learning generator model can be trained to minimize the value of the discriminator output 414 (e.g., in an adversarial manner).

[0089] Figure 5 A graphical diagram depicting an example machine learning model and associated training signal for reconstructing input image generation according to an example embodiment of the present disclosure. In some embodiments, as depicted, the machine learning generator model(s) and the machine learning discriminator model(s) may be in the context of Figure 4More specifically, the machine learning generator model(s) 508 and the machine learning discriminator model(s) 510 can be run in a parallel configuration to perform transformations on an output image 412 (e.g., an output image 412 generated by the machine learning generator model(s) 408).

[0090] The machine learning generator model(s) 508 (e.g., including an encoder portion 508A and a decoder portion 508B) may receive the output image 412 and the user-specified condition vector 404 as input. The machine learning generator model(s) 508 may be configured to generate the output image 412 and the user-specified condition vector 404 in a manner similar to Figure 4 The reconstructed input image 512 is generated in the same manner as the (multiple) machine learning generator model 408 of the embodiment of the present invention. The reconstructed input image 512 can be a reconstruction of the input image 406 from the output image 412. More specifically, by generating the reconstructed input image 512 based on the output image 412 and the user-specified condition vector 404, the (multiple) machine learning generator model 508 can reverse the parameterized transformation specified by the vector and originally applied by the (multiple) machine learning generator model 408.

[0091] As an example, the input image 406 may be an image depicting a summer environment with daytime lighting conditions. Figure 4 The (multiple) machine learning generator model 408 can generate an output image 412 depicting a winter environment with nighttime lighting conditions based on the user-specified condition vector 404 and the input image 406. The (multiple) machine learning generator model 508 can receive the output image 412 and the user-specified condition vector 404 and generate a reconstructed input image 512 as an output. The reconstructed input image 512 can depict a reconstruction of the depiction of the input image 406 (e.g., a summer environment with daytime lighting conditions). In this manner, the (multiple) machine learning generator model 508 can reconstruct the input image from the output image by applying a transformation that inverts the transformation applied by the (multiple) machine learning generator model 408.

[0092] The reconstructed input image 512 and the input image 406 may be received as input by the (multiple) machine learning discriminator model 510. The (multiple) machine learning discriminator model 510 may evaluate the difference between the reconstructed input image 512 and the input image 406 to output a reconstructed discriminator output 514. The reconstructed discriminator output may be used as a training signal for the optimization function. The optimization function may evaluate the reconstructed discriminator output 414 and, based at least in part on the reconstructed discriminator output 414, modify the values ​​of one or more parameters of the (multiple) machine learning discriminator model 510 and / or the (multiple) machine learning generator model 508 based on the optimization function. More specifically, the output may be back-propagated through the (multiple) machine learning model (e.g., 508 and 510) to determine the values ​​associated with one or more parameters of the (multiple) model to be updated. The one or more parameters may be updated to reduce the difference evaluated by the optimization function (e.g., using an optimization process such as a gradient descent algorithm).

[0093] In some embodiments, the reconstruction discriminator output 514 can be used as Figure 4 The training signals of the (multiple) machine learning generator model 408 and the (multiple) machine learning discriminator model 410 of the reconstructed discriminator output 514 can be modified. Based at least in part on the optimization function that evaluates the reconstructed discriminator output 514, the values ​​of one or more parameters of the (multiple) machine learning discriminator model 410 and / or the (multiple) machine learning generator model 408 can be modified. More specifically, the output can be back-propagated through the (multiple) machine learning model to determine the values ​​associated with one or more parameters of the (multiple) model to be updated. The one or more parameters can be updated to reduce the difference evaluated by the optimization function (e.g., using an optimization process such as a gradient descent algorithm).

[0094] In this manner, the performance of the mirror model architecture (e.g., model(s) 508 and 510) can be used as a training signal for the transformed model architecture (e.g., model(s) 408 and 410) to enforce consistency between the transformations. More specifically, the addition of the mirror model architecture can be used to enforce structure on the generated output of the machine learning generator model 408 to minimize the bias of the transformation.

[0095] In some embodiments, the problem evaluated by the objective function can be formulated such that given an input image domain X (e.g., input image 406) and a target image domain Y (e.g., target image 402), a parameterizable transformation G can be learned. P :X,P→Y. For example, G P may represent the transformations performed by the (multiple) machine learning generator models 408, and F PThe transformation performed by the (multiple) machine learning generator model 508 can be represented. As represented by P, the (multiple) model is aware of (e.g., can take as input) the user-specified conditional vector 404. More specifically, the transformation can be learned by the (multiple) machine learning generator model 408, which generates effective samples y∈Y by taking as input the input image 406x∈X and applying p∈P (e.g., the value of the user-specified conditional vector 404) parameterization to the transformation. The (multiple) target image 402 can be from a (multiple) set of unpaired training samples. As an example, in a training scenario, the unpaired target image 402 can be obtained from (multiple) sets of unpaired training samples. The target image is selected from the set of unpaired input images An input image 406 is selected from the set of.

[0096] More specifically, in some embodiments, two unpaired image domains X (e.g., input image(s) 406) and Y (e.g., target image(s) 402 from a training image set) can be mapped to machine learning discriminator model(s) 410 (e.g., ) and (multiple) machine learning discriminator models 510 (e.g., ), where x~(X) and y~(Y) represent two data distributions. The user-specified expected value can be parameterized as a user-specified conditional vector 404. As an example, the user-specified conditional vector 404 can be represented as p∈ As discussed above, by obtaining the user-specified condition vector 404 (e.g., ) as input, (multiple) machine learning generator models 410 (e.g., G P ) and (multiple) machine learning generator models 510 (e.g., F P ) can be known parameterically.

[0097] As an example, the objective function of the system (e.g., Figure 4 and Figure 5 The combined architecture of ) can be expressed as:

[0098]

[0099]

[0100]

[0101] Among them, (multiple) machine learning generator models 410 (e.g., G P ) and (multiple) machine learning generator models 510 (e.g., F P) can be optimized to minimize the objective function L, while (multiple) machine learning discriminator models (e.g., and ) is optimized to maximize the objective function L. Can be used as a measurement mirror architecture (e.g. Figure 4 The model architecture depicted in Figure 5 ), and λ can represent the loss signal of cycle consistency in the overall objective function L. More specifically, Figure 4 The model architecture can be signaled by the loss To express, Figure 5 The model architecture can be signaled by the loss and the cycle consistency between models can be expressed by To express.

[0102] Fig. 6A A graphical representation depicting an example day-to-night image transformation with different user-desired values ​​for a defined characteristic applied to an input image according to an example embodiment of the present disclosure. 600A depicts an example transformation performed by a (multiple) machine learning generator model of the present disclosure. As depicted, the (multiple) machine learning generator model can receive a user-specified condition vector. The user-specified condition vector can include parameterized values ​​that the user desires for the defined characteristic of the (multiple) transformation to be applied. In this example, the user can specify a day-to-night transformation with a value of 0% for the input image. The transformation can be applied to the input image with a degree of 0%, resulting in image 602. The user-specified condition vector can specify successively larger values ​​for the degree of the day-to-night transformation, resulting in increasingly larger degrees of transformation in images 604, 606, 608, and 610.

[0103] For image 610, the user-specified condition vector specifies a 100% degree of transformation, resulting in an image depicting daylight lighting conditions being completely transformed into an image depicting nighttime lighting conditions. As depicted, the user-specified condition vector can provide the user with fine-grained control over the degree of transformation applied to the input image (e.g., various degrees of transformation applied to images 604, 606, and 608).

[0104] It should be noted that in some embodiments, in addition to Fig. 6A In addition to the day-to-night transformation depicted in , a second transformation can be applied, such as Figure 6CAs shown. As an example, a summer-to-winter transformation can be applied in conjunction with a day-to-night transformation, with the summer-to-winter transformation being controlled by the user using additional parameterized values ​​in a user-specified condition vector. For example, a user-specified condition vector may specify a value of 1.0 for the day-to-night transformation (e.g., depicting full nighttime lighting conditions) and a value of 0.3 for the summer-to-winter transformation (e.g., depicting mild winter ambient conditions, such as a small amount of snow). In this manner, defining characteristics of an image (e.g., degree of day or night, etc.) can be manipulated using parameterized values ​​(e.g., real and continuous values, etc.) of the defining characteristics of the transformation(s) with sufficient specificity to enable a user to have fine-grained control over the transformations applied.

[0105] Figure 6B A graphical representation of an example light source image transformation applied to one or more defined characteristics of an input image at different user desired values ​​according to an example embodiment of the present disclosure is depicted. 600B depicts an example transformation performed by (multiple) machine learning generator models of the present disclosure. As depicted, (multiple) machine learning generator models can receive a user-specified condition vector. The user-specified condition vector may include parameterized values ​​that the user expects for the defined characteristics of the (multiple) transformations to be applied. In this example, a user can specify a light source transformation that can be positioned two-dimensionally (e.g., adding a light source to an image). At 612, the user can specify a horizontal position value (e.g., x value) of -0.67 and a vertical position value (e.g., y value) of 0.00. As depicted, the user can keep the vertical position value constant and increase the value of the horizontal position value. As the horizontal position value increases, the transformation can apply light from the light source to different locations of the image. For example, as the x value slowly increases, the position of the light source in images 612-620 moves from right to left.

[0106] It should be noted that the location of the transformation (e.g., light source, etc.) can be additionally specified in three dimensions in the user-specified condition vector. As an example, a depth parameter (e.g., z value) can be added as an additional parameter to change the depth of the light source in the image. As an example, if the image 616 has a z value of 1.0, the light source can appear to be generating light from behind the face depicted in the image.

[0107] Figure 6CA graphical representation depicting an example day-to-night transformation and a weather transformation simultaneously applied to a defined characteristic of an input image at different user desired values ​​according to an example embodiment of the present disclosure. 600C depicts two example transformations performed by a (multiple) machine learning generator model of the present disclosure. As depicted, the (multiple) machine learning generator model can receive a user-specified condition vector. The user-specified condition vector can include parameterized values ​​that the user desires for the defined characteristic of the transformation to be applied. In this example, the user can specify a day-to-night transformation (e.g., transforming daytime lighting conditions to nighttime lighting conditions) and a weather transformation (e.g., adding a raining condition to an image, etc.). Each of the two transformations can be applied to a different extent based on the values ​​included in the user-specified condition vector. Figure 622 depicts an input image before the transformations are applied by the machine learning model (or, as an example, the case where the user-specified condition vector provides a value of 0 for both transformations). Image 624 depicts the input image after the day-to-night transformation has been applied with a value of 0.25 p1 and the weather transformation has been applied with a value of 0.25 p2. As depicted, the user can gradually increase the values ​​(p1 and p2) associated with the transformations to increase the degree of transformation applied. For example, as the p1 and p2 values ​​are gradually increased, images 624-630 are transformed to include a higher degree of nighttime lighting conditions and rainy weather conditions (e.g., darker clouds, raining, etc.), while image 630 depicts both transformations being fully applied to the image.

[0108] Figure 7 is a flowchart illustrating an example method for training one or more machine learning models according to an example embodiment. Method 700 may, for example, use Figure 1 (multiple) computing devices. Figure 7 The steps performed in a particular order are depicted for purposes of illustration and discussion. One of ordinary skill in the art will understand, using the disclosure provided herein, that the individual steps of any one of the methods described herein may be omitted, rearranged, performed simultaneously, expanded, and / or modified in various ways without departing from the scope of the present disclosure.

[0109] At 702, method 700 includes obtaining one or more machine learning generator models. The (multiple) machine learning generator model(s) may be configured to receive an input image and a user-specified condition vector that parameterizes one or more desired values ​​for one or more defined characteristics of an output image. The (multiple) machine learning generator model(s) may also be configured to perform one or more transformations on the input image based at least in part on the user-specified condition vector to generate an output image having one or more desired values ​​for the one or more defined characteristics. The (multiple) machine learning generator model(s) may be or otherwise include one or more neural networks (e.g., deep neural networks), etc. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a residual neural network, and / or various other types of neural networks.

[0110] At 704, method 700 may include receiving an input image and a user-specified condition vector. The input image may be a digital image (e.g., a digital image captured by a digital image capture device, etc.). The user-specified condition vector may parameterize one or more desired values ​​for one or more defined characteristics of an output image. In some embodiments, the user-specified condition vector may be a real and continuously-valued vector. As an example, the vector may include values ​​(e.g., parameterized user desired values) that are both real and continuous (e.g., 0.5, 1, .2235, -5, etc.) that correspond to one or more desired characteristics of the output image.

[0111] At 706, method 700 may include generating an output image having one or more expected values ​​for one or more characteristics using one or more machine learning generator models. The defined characteristics of the output image may describe one or more features, style, content, or any other aspect of the output image. In some embodiments, the defined characteristics may include style characteristics describing one or more style transformations to be performed on the input image. As an example, a digital image generated by a digital camera may be transformed into an oil painting (e.g., an image-to-painting transformation). The machine learning generator model may be trained so that the style transformation can replicate various styles of known artists (e.g., Monet, Van Gogh, Picasso, etc.). For example, a digital input image generated by a digital camera may be transformed into an output image that replicates the style of Picasso. It should be noted that the above style transformation may also be performed in reverse by the machine learning generator model (e.g., transforming a Picasso painting into a "digital image" style). Both of the above style transformations may be applied to the input image to a certain degree. For example, the image-to-painting transformation may be applied to the input image with an expected value of 0.5, which may produce an output image that is partially transformed into the style of an oil painting. The style characteristics may describe any kind of subjective and / or objective transformation to be applied to an image (eg, painting style, drawing style, animation style, camera focal length / focus, etc.).

[0112] Additional disclosure

[0113] The technology discussed herein relates to servers, databases, software applications and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows many possible configurations, combinations, and the division of tasks and functions between and among components. For example, the process discussed herein can be implemented using a single device or component or multiple devices or components that work in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0114] Although the subject matter has been described in detail for various specific example embodiments of the subject matter, each example is provided for explanation rather than limitation of the present disclosure. Those skilled in the art, after obtaining an understanding of the foregoing, can easily produce variations, changes and equivalents to these embodiments. Therefore, the subject matter disclosure does not exclude such modifications, changes and / or additions to the subject matter that are readily apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such variations, changes and equivalents.

[0115] Specifically, although Figure 4-Figure 5The steps are depicted as being performed in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. Without departing from the scope of the present disclosure, Figure 4-Figure 5 The various steps shown in may be omitted, rearranged, combined and / or adapted in various ways.

Claims

1. A computing system for interpretable image-to-image translation, the computing system comprising: one or more processors; and One or more non-transitory computer-readable media collectively storing: One or more machine learning generator models, the machine learning generator model comprising an encoder portion and a decoder portion, and configured to: receiving an input image and a user-specified conditioning vector that parameterizes one or more desired values ​​for one or more defined characteristics of an output image, wherein the user-specified conditioning vector is provided directly to a decoder portion of the one or more machine learning generator models; and performing one or more transformations on the input image based at least in part on the user-specified condition vector to generate an output image having one or more desired values ​​for the one or more defined characteristics; One or more machine learning discriminator models, configured to: receiving the output image, the target image, and the user-specified condition vector; and generating a discriminator output that selects one of the output image and the target image as having one or more desired values ​​for the one or more defined characteristics; and Instructions, when executed by the one or more processors, cause the computing system to perform operations comprising: At least one of the one or more generator models is trained based at least in part on the discriminator output.

2. A computing system according to any preceding claim, wherein: The one or more machine learning generator models are further configured to: receiving the output image and the user-specified condition vector; and performing one or more reconstruction transforms on the output image to generate a reconstructed input image based at least in part on the output image and the user-specified condition vector, the one or more reconstruction transforms being configured to remove the one or more transforms applied to the input image; and Wherein, the one or more machine learning discriminator models are further configured to: receiving the input image, the reconstructed input image, and the user-specified condition vector; and generating a reconstruction discriminator output that selects one of the reconstructed input image and the input image; and The operation further includes: At least one of the one or more generator models is trained based at least in part on the reconstructed discriminator output.

3. A computing system according to any preceding claim, wherein: The user-specified condition vector comprises a real and continuously-valued vector.

4. The computing system according to claim 1 or 2, wherein: The one or more defined characteristics include one or more real-world characteristics.

5. The computing system according to claim 1 or 2, wherein: The user-specified condition vector describes the extent to which the one or more transformations are applied.

6. The computing system according to claim 1 or 2, wherein: The one or more defining characteristics include: light properties; at least one light source position characteristic; Environmental characteristics; Color characteristics; Stylistic characteristics; and Features characteristics.

7. The computing system according to claim 1 or 2, wherein: The user-specified condition vector describes one or more regions of the input image to which the one or more transformations are applied.

8. The computing system according to claim 1 or 2, wherein: At least one or more machine learning generator models include one or more layers that transform the user-specified conditioning vector into an alternative representation before performing the one or more transformations on the input image based at least in part on the alternative representation.

9. The computing system according to claim 1 or 2, wherein: At least one of the one or more machine learning generator models includes a residual neural network.

10. The computing system according to claim 1 or 2, wherein: The operations also include: At least one of the one or more discriminator models is trained based at least in part on a difference between the discriminator output and real data associated with the target image.

11. A computer-implemented method of performing image-to-image conversion, the method comprising: Obtaining, by a computing system including one or more computing devices, one or more machine learning generator models, the machine learning generator models including an encoder part and a decoder part, and configured to: receiving an input image and a user-specified conditioning vector that parameterizes one or more desired values ​​for one or more defined characteristics of an output image, wherein the user-specified conditioning vector is provided directly to a decoder portion of the one or more machine learning generator models; and performing one or more transformations on the input image based at least in part on the user-specified condition vector to generate an output image having one or more desired values ​​for the one or more defined characteristics; Receiving, by the computing system, the input image and the user-specified condition vector; and An output image having one or more desired values ​​for the one or more characteristics is generated by the computing system using the one or more machine learning generator models.

12. The computer-implemented method of claim 11, wherein: The one or more defined characteristics include one or more real-world characteristics.

13. The computer-implemented method of claim 11 or 12, wherein: The user-specified condition vector describes the extent to which the one or more transformations are applied.

14. The computer-implemented method of claim 11 or 12, wherein: The one or more defining characteristics include: light properties; at least one light source position characteristic; Environmental characteristics; Color characteristics; Stylistic characteristics; and Features characteristics.

15. The computer-implemented method of claim 11 or 12, further comprising: At least one of the one or more machine learning discriminator models is trained, by the computing system, based at least in part on a difference between the output image and real data associated with the output image.

16. The computer-implemented method of claim 11 or 12, wherein: The input image is a digital photo.

17. The computer-implemented method of claim 11 or 12, wherein: The output image includes an image style different from that of the input image.

18. The computer-implemented method of claim 11 or 12, wherein: The output image includes one or more modified features of the input image.

Images