Systems and methods for manipulating two-dimensional (2D) images of three-dimensional (3D) objects

Abstract

To provide an image processing system and method for operating a two-dimensional (2D) image of a three-dimensional (3D) object (e.g., a human face) of a predetermined class.SOLUTION: A method operates a physical characteristic of a 3D object as lighting that irradiates a 2D input image of a 3D object of a predetermined class to a 3D shape of a 3D object, an albedo of the 3D input object, a pose of the 3D input object, and the 3D input object, extracts a physical characteristic from the 2D input image by using a neural network to be trained so as to reconstruct the 2D input image, reconstructs the 2D input image by using a plurality of sub networks to separate the physical characteristic from pixels of the 2D input image, and combines the separated physical characteristic generated by the plurality of sub networks by using a differential renderer to be a 2D output image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to image processing, and more particularly to systems and methods for manipulating two-dimensional (2D) images of three-dimensional (3D) objects. [Background technology]

[0002] Today, image manipulation of two-dimensional (2D) images of human faces has gained popularity for a wide range of applications. These include changing lighting conditions to create compelling portraits, changing a person's identity to anonymize an image, and swapping hairstyles in virtual trial settings. In facial image manipulation, careful handling of facial image details is crucial to achieving photorealism in synthesized facial images. However, the human visual system can be sensitive to even very small artifacts in synthesized facial images. Typically, for a 2D image of an object to be photorealistic, it should be comparable to a 2D representation of a realistic 3D object (e.g., a 3D face whose shape and reflectance properties are within the range of typical human experience). Therefore, for a modified 2D facial image to remain photorealistic, it should correspond to realistic variations in the corresponding 3D object or its environment, such as variations in the apparent shape, expression, and appearance attributes of the corresponding 3D face. Existing prior art methods for photorealistic automatic image manipulation of 2D facial images are 2D in nature: they modify one 2D facial image to form another 2D facial image without explicitly defining the corresponding 3D face in either image. This limits the ability of such 2D methods to synthesize large variations in pose of the 2D images (e.g., from a frontal view to a profile view).

[0003] On the other hand, existing methods for image manipulation, which are 3D in nature and trained from 3D scans of human faces, can accurately perform 3D manipulation, but they are limited in their ability to generate photorealistic 2D images of the resulting 3D synthesized faces. For example, a 3D face model can be used to reconstruct a 3D face from 2D images of a human face. The 3D model can be obtained from a training 3D dataset of human faces. The training 3D dataset may have limited variation in the attributes of the 2D images, which can result in limited facial expression and pose control for the face. Furthermore, the 3D training dataset may not have sufficient detail or sufficiently accurate information about the face to render a photorealistic 2D image. Such limitations can cause a lack of realism in the reconstructed 2D face image. In some cases, high-quality 3D scans of human faces can be performed to augment the 3D data in the training 3D dataset. However, high-quality 3D scans can consume excessive amounts of time and storage resources.

[0004] Therefore, there is a need for a technical solution for reconstructing and manipulating 2D images with photorealism. More specifically, there is a need for manipulating 2D images of 3D objects to achieve photorealism in an efficient and feasible manner. Summary of the Invention

[0005] Therefore, an object of the present disclosure is to provide a system for manipulating a 2D image of a 3D object. The image of a 3D object is a 2D representation affected by one or more physical properties of the 3D object. The one or more physical properties of the 3D object include the 3D shape of the 3D object, the albedo of the 3D shape, the pose of the 3D object, and the lighting illuminating the 3D object. A change in the value of each of these physical properties affects the 2D image representation of the object. To this end, the 2D image representation of the object can be manipulated based on the manipulation of each of the physical properties. Additionally or alternatively, prior to the manipulation, the 2D image can be masked with a mask image. The mask image can be automatically generated for the 2D image using a segmentation method such as a semantic segmentation network, landmark localization, or face analysis. The mask image segments regions. For example, by masking the background region from the foreground region, the background region is segmented from the foreground region corresponding to the 3D object in the 2D image. For example, a human face in a 2D image may be segmented by masking background regions in the 2D image using a corresponding face mask image of the human face, and may be further segmented into hair regions using a corresponding hair mask image of the hair. Such mask images may enhance further processing of the 2D image representation of the 3D object.

[0006] Some embodiments are based on the recognition that one or more physical properties can be extracted from a 2D image representation of a 3D object. For example, some embodiments are based on the recognition that a neural network can be trained in one or a combination of supervised and unsupervised manners to extract one or more physical properties of a 3D object from training 2D images of the 3D object and reconstruct the original training image from the extracted one or more physical properties of the object. The one or more physical properties may include the 3D shape of the 3D input object, the albedo of the 3D shape of the 3D input object, the pose of the 3D input object, and the lighting illuminating the 3D input object. In some exemplary embodiments, the neural network is pre-trained in one or a combination of supervised and unsupervised manners based on a dataset including synthetically generated human facial images. Such a pre-training process may enable the neural network to capture one or more salient facial features from the synthetically generated human facial images.

[0007] Some embodiments are based on the recognition that manipulating these extracted physical properties of a 3D object prior to image reconstruction can be used for realistic image manipulation by manipulating the properties of the 3D object and then rendering the manipulated 3D object as a 2D image, as opposed to direct image manipulation in the 2D domain of the image. Some embodiments are based on another recognition that a neural network can include multiple and / or different sub-networks. In some exemplary embodiments, multiple sub-networks extract different physical properties of the 3D object from the 2D image. The extraction of the different physical properties by the sub-networks allows for the isolation of the different physical properties in an independent or separate manner. The independent isolation of each physical property allows for the representation of different variations of one or more corresponding physical properties, which in turn contributes to realism in reconstructing a 2D output image from a 2D input image of the 3D object. To that end, the isolated physical properties can be rendered into a 2D image.

[0008] Some embodiments are based on the recognition that the resulting rendered 2D image may not have a desired level of photorealism due to limitations in factors such as the rendering process, the representation of one or more physical properties, or the level and type of variation present in the training data. Some embodiments are based on the further recognition that the photorealism of the rendered 2D image may be improved by 2D refinement of the rendered 2D image, which in some embodiments is done using a 2D refiner network that may be trained using an adversarial approach such as that used in training a generative adversarial network (GAN).

[0009] In some exemplary embodiments, each of the one or more physical properties is separated from the pixels of the 2D input image by a trained neural network using multiple sub-networks. Each of the multiple sub-networks separates one or more of the one or more physical properties of the 3D object. The separated one or more physical properties generated by the multiple sub-networks are combined into a 2D output image using a differentiable renderer. For example, the differentiable renderer renders a reconstructed 2D image, such as the 2D output image, using the separated physical properties of the 3D object's lighting and pose. Such separation makes it possible to design different specialized sub-network architectures that can be adapted for different physical properties.

[0010] Some embodiments are based on the recognition that using a differentiable renderer rather than a more traditional computer graphics renderer allows a combined neural network including one or more of the sub-networks and the differentiable renderer to be trained exhaustively, so that errors in the rendered image can be propagated back up the combined neural network to train one or more of the sub-networks.

[0011] In one embodiment, the albedo sub-network can be trained to extract the albedo of a 3D object from pixels of a 2D input image. To that end, the albedo sub-network includes a style-based generative adversarial network (GAN) architecture, and the network can be adversarially trained using a GAN framework. The style-based GAN can be trained using a set of photographic images of objects from a predetermined class. In some embodiments, the 3D object can be selected from a predetermined class of human faces.

[0012] Style-based GAN architectures can be advantageous for generating realistic 2D images.

[0013] Some embodiments are based on the recognition that style-based GANs generate photorealistic 2D images of objects (e.g., faces) from a predetermined class after being trained on a large dataset of 2D photographic images of the objects from the predetermined class (e.g., a large database of facial images). Typically, style-based GANs are used to directly generate 2D images that are indistinguishable from photographic images from the training dataset, trained using adversarial training with a discriminator network. However, to generate a 3D object model by a style-based GAN using only 2D photorealistic image data, the output space of the style-based architecture may be a subset of the 3D model. The output space may not be the same space as the real-world dataset. Some embodiments are based on the further recognition that style-based GANs can be used to generate the albedo of a 3D object, e.g., a 2D albedo map. Such a 2D albedo map may be applied to the 3D shape of the 3D object, which may then be input to a renderer to generate a synthetic 2D output image of the 3D object. However, the generation of synthetic 2D output images may require high-quality albedo maps to train a style-based GAN by applying a discriminator to that output space.

[0014] Some embodiments are based on the additional recognition that a discriminator network can be used for adversarial training of a style-based GAN to generate photorealistic synthetic 2D output images without a high-quality albedo map. The discriminator network can be applied after using a differentiable renderer to render a 3D object model into a 2D synthetic image. The differentiable renderer that renders the 3D object model into a 2D synthetic image prevents the 2D albedo map from being directly applied to the style-based GAN. In this way, a style-based GAN architecture may be used to generate albedo maps for 3D objects, while the discriminator network is used in the space of 2D images for output at the end of the rendering process. The 2D image space can be used to obtain large amounts of real-world training data (e.g., photographic images of faces).

[0015] Additionally or alternatively, some embodiments recognize that neural networks can be trained in an unsupervised manner. An unsupervised neural network can extract distinct physical properties of a 3D object into a latent space. The latent space can be used to reconstruct a physical space for the corresponding physical properties. For example, an albedo subnetwork and a 3D shape subnetwork can have a GAN architecture that encodes pixels of a 2D image into corresponding latent spaces for the object's albedo and 3D shape, respectively. These latent spaces can be used to generate physical spaces for the object's albedo and 3D shape. The latent space and physical space enable a dual representation of the object. Such a dual representation allows for the manipulation of physical properties in either or both the latent space and the physical space to achieve photorealistic 2D images. In contrast, in some embodiments, the lighting and pose subnetworks are encoded from pixels of the 2D image into corresponding physical spaces. Unsupervised training of a neural network for the extraction of distinct physical properties into a latent space can eliminate the need for actual 3D data, which can improve the overall performance and storage efficiency of the image processing system.

[0016] Additionally or alternatively, the reconstructed 2D output image may include reconstructed hair of the human face in the 2D input image. In some embodiments, the reconstructed hair may be obtained from a hair model modeled by a hair manipulation algorithm. The hair manipulation algorithm manipulates the hair model independently of manipulating one or more physical characteristics of the human face. To that end, the hair manipulation algorithm modifies the representation of hair in the 2D output image in response to detecting manipulation of the pose of the human face from the 2D input image to the 2D output image. In some exemplary embodiments, the hair manipulation algorithm modifies the appearance of the hair to correspond to the modified pose of the human face in the 2D output image. Such a hair manipulation algorithm enables outputting a realistic synthetic image corresponding to the hair in the 2D input image, with the hair modified to correspond to one or more manipulated physical characteristics, such as the modified pose.

[0017] Additionally or alternatively, the image processing system may be used to generate multiple realistic synthetic images from 2D images of a 3D object. The multiple realistic synthetic images include variations in one or more physical characteristics of the 3D object, such as pose, facial expression, and lighting. The multiple realistic synthetic images may be used in different applications. The different applications may include an application for generating a cartoon or animated version of a user, an application for trying out different hairstyle looks for a user, a camera-enabled robotic system application, a facial reconstruction application, etc. In some exemplary embodiments, the multiple realistic synthetic images may be used as training data sample images for a different system (e.g., a robotic system, a facial reconstruction application) to generate realistic training images in an efficient manner while avoiding collecting sample images using a camera associated with the corresponding system. For example, an existing dataset for training a different system may include images of faces where variations in one or more physical characteristics, such as pose, facial expression, and lighting, are not the same as those in images where the different system is deployed. In such cases, the multiple realistic synthetic images may be used to augment or replace an existing training dataset with images that include more examples for the different system.

[0018] Additionally or alternatively, the image processing system may be used to generate anonymized versions of input images, which may be generated using anonymization techniques such as similarity-based metrics. This may be used in applications where images, such as facial images, should be made public, but users of the system do not have permission to share photos or videos of one or more people in the input images.

[0019] Accordingly, one embodiment discloses an image processing system for manipulating a two-dimensional (2D) image of a predetermined class of three-dimensional (3D) object. The image processing system includes an input interface, a memory, a processor, and an output interface. The input interface is configured to receive a 2D input image of a predetermined class of 3D input object and one or more manipulation instructions for manipulating one or more physical properties of the 3D input object. The one or more physical properties include a 3D shape of the 3D input object, an albedo of the 3D shape of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The memory is configured to store a neural network. The neural network is trained to reconstruct the 2D input image by isolating one or more physical properties of the 3D input object from pixels of the 2D input image using multiple sub-networks. The isolated one or more physical properties generated by the multiple sub-networks are combined into a 2D output image using a differentiable renderer. There is one subnetwork for each of one or more subsets of one or more physical properties of the 3D input object, and each subnetwork of the plurality of subnetworks is trained to extract the corresponding physical property from pixels of the 2D input image into one or a combination of a latent space and a physical space of the corresponding physical property. The albedo subnetwork trained to extract the albedo of the 3D input object from pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from a predetermined class. The processor is configured to present the 2D input image received from the input interface to the neural network and manipulate the latent space, the physical space, or both of the one or more physical properties of the 3D input object according to one or more manipulation instructions.An operation is performed on the separated one or more physical properties generated by the multiple sub-networks to modify the 3D input object before the differentiable renderer reconstructs the 2D output image. An output interface is configured to output the 2D output image.

[0020] Accordingly, another embodiment discloses a method for manipulating a two-dimensional (2D) image of a predetermined class of three-dimensional (3D) object. The method includes receiving a 2D input image of a predetermined class of 3D input object and one or more manipulation instructions for manipulating one or more physical properties of the 3D input object. The one or more physical properties include a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The method includes presenting the 2D input image to a neural network and manipulating a latent space, a physical space, or both of the one or more physical properties of the 3D input object in accordance with the one or more manipulation instructions, wherein the manipulation is performed on the isolated one or more physical properties generated by the multiple sub-networks to modify the 3D input object before a differentiable renderer reconstructs the 2D output image. The neural network is trained to reconstruct a 2D input image by separating physical properties of a 3D input object from pixels of the 2D input image using multiple subnetworks and combining the separated physical properties generated by the multiple subnetworks into a 2D output image using a differentiable renderer. There is one subnetwork for each subset of one or more of the physical properties of the 3D input object. Each subnetwork of the multiple subnetworks is trained to extract corresponding physical properties from pixels of the 2D input image into one or a combination of a latent space and a physical space of the corresponding physical property. The albedo subnetwork trained to extract the albedo of the 3D input object from pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from a predetermined class. The method further includes outputting the 2D output image.

[0021] Further features and advantages will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure is further described in the following detailed description with reference to the drawings, which are set forth as non-limiting examples of exemplary embodiments of the present disclosure. Like reference numerals represent like parts throughout the several views of the drawings. The drawings shown are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief explanation of the drawings]

[0023] [Figure 1] 1A-1C are diagrams illustrating exemplary representations of image manipulation of an image of a 3D object, according to some embodiments of the present disclosure. [Figure 2] 1 is a block diagram of a system for manipulating two-dimensional (2D) images of a predetermined class of three-dimensional (3D) objects according to an exemplary embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates a neural network framework for manipulating 2D images of a predetermined class of 3D objects, according to an exemplary embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating a process for manipulating 2D images of a predetermined class of 3D objects according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates a framework for a hair manipulation algorithm, according to an exemplary embodiment of the present disclosure. [Figure 6A] FIG. 1 illustrates a method for manipulating 2D images of a predetermined class of 3D objects according to an exemplary embodiment of the present disclosure. [Figure 6B] FIG. 1 illustrates a method for manipulating 2D images of a predetermined class of 3D objects according to an exemplary embodiment of the present disclosure. [Figure 7] 1 is a block diagram of a system for manipulating 2D images of a predetermined class of 3D objects according to an exemplary embodiment of the present disclosure. FIG. [Figure 8]FIG. 1 illustrates a use case scenario for manipulating 2D images of a predetermined class of 3D objects, according to an exemplary embodiment of the present disclosure. [Figure 9] FIG. 10 illustrates a use case scenario for manipulating 2D images of a predetermined class of 3D objects, according to another exemplary embodiment of the present disclosure. [Figure 10] FIG. 10 illustrates a use case scenario for manipulating 2D images of a predetermined class of 3D objects, according to another exemplary embodiment of the present disclosure. [Figure 11] FIG. 10 illustrates a use case scenario for manipulating 2D images of a predetermined class of 3D objects, according to yet another exemplary embodiment of the present disclosure.

[0024] While the above-described drawings set forth embodiments disclosed herein, other embodiments are contemplated, as noted in the description. This disclosure presents exemplary embodiments by way of representation and not limitation. Numerous other modifications and embodiments can be devised by those skilled in the art which fall within the scope and spirit of the principles of the embodiments disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0025] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0026] As used in this specification and claims, the terms "for example," "such as," and "e.g.," as well as the verbs "comprise," "have," and "include" and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended. That is, the list should not be considered as excluding other additional components or items. The term "based on" means based at least in part on. It should also be understood that the language and terminology employed herein are for purposes of description and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0027] Overview The proposed image processing system enables the reconstruction of a 2D output image from 2D input images of a predetermined class of 3D objects, such as a human face. The purpose of the image processing system is to manipulate the 2D input images for the reconstruction of a 2D output image. The 2D input images are manipulated by isolating one or more physical properties of the 2D input images, such as the 3D shape of the 3D object, the albedo of the 3D object, the pose of the 3D object, and the lighting illuminating the 3D object. This is further illustrated in Figure 1.

[0028] FIG. 1 illustrates an exemplary representation 100 illustrating image manipulation of an image 102 of a 3D object, according to some embodiments of the present disclosure. The image 102 is a 2D representation of a 3D object, such as a human face. The image 102 possesses one or more physical properties that can be extracted from the 2D representation of the human face. In some exemplary embodiments, the one or more physical properties can be extracted from pixel values ​​of the 2D image 102. The extracted one or more physical properties can be explicitly represented as an estimated 3D shape 104A of the 3D input object, an estimated albedo 104B of the 3D input object, an estimated pose 104C of the 3D input object, and an estimated lighting 104D illuminating the 3D input object. The explicit representation of each of the one or more physical properties, hereinafter collectively referred to as one or more physical properties 104A-104D, enables the manipulation of each of the one or more physical properties 104A-104D in an independent manner. The manipulation here corresponds to a change in the value (e.g., pose angle or lighting parameter) of each of the corresponding one or more physical properties 104A-104D. If there is a change in the value of the one or more physical properties, the 2D representation of the image 102 is manipulated. In some exemplary embodiments, such independent manipulation of one or more physical properties 104A-104D may be used to generate multiple realistic synthetic images from the 2D image of the 3D object. The multiple realistic synthetic images may correspond to variations in the one or more physical properties 104A-104D of the 3D object.

[0029] The one or more physical properties 104A-104D are combined to generate a 3D model of the object, which can be rendered to generate a 2D composite image 106A that resembles the input image 102. Manipulation of one or more of the one or more physical properties 104A-104D corresponds to a change in the 3D object model, which causes a corresponding manipulation in the 2D composite image 106A. Such manipulation prior to rendering the 3D model as a reconstructed composite 2D image 106A enables image manipulation of the image 102 in a realistic manner. In some embodiments, the reconstructed composite 2D face model 106A is combined with a hair model 106B to generate a final output image, such as output image 108. The output image 108 is a photorealistic composite 2D image that corresponds to the manipulated version of the input image 102.

[0030] Additionally, each variation of one or more physical properties 104A-104D generates different manipulations of lighting variation 110, facial expression variation 112, pose variation 114, shape variation 116, and texture variation 118 of image 102. The different manipulations of physical properties 104A-104D can be used to synthesize photorealistic 2D images that correspond to variations in properties such as pose, facial expression, and lighting conditions of input image 102.

[0031] Such image manipulation of the 2D representation of image 102 through isolation of each of one or more physical properties 104A-104D is performed by a system, which is further described with reference to FIG.

[0032] 2 illustrates a block diagram of a system 200 for manipulating 2D images of a predetermined class of 3D objects according to an exemplary embodiment of the present disclosure. The system 200 corresponds to an image processing system including an input interface 202, a memory 204, a processor 206, and an output interface 208. The input interface 202 is configured to receive a 2D input image, such as image 102, of a 3D input object, i.e., a human face. The input interface 202 is also configured to receive one or more manipulation instructions for manipulating one or more physical properties of the 3D input object, such as one or more physical properties 104A-104D, including the 3D shape of the 3D input object (e.g., shape 104A), the albedo of the 3D input object (e.g., albedo 104B), the pose of the 3D input object (e.g., pose 104C), and lighting properties illuminating the 3D input object (e.g., lighting 104D).

[0033] The memory 204 is configured to store a neural network 210. The neural network 210 is trained to reconstruct the image 102 by isolating one or more physical properties 104A-104D. The one or more physical properties 104A-104D are isolated from the pixels of the image 102. In some embodiments, multiple sub-networks are used to isolate the one or more physical properties 104A-104D from the pixels, one sub-network for each of one or more subsets of the one or more physical properties 104A-104D of the 3D object. Each sub-network of the one or more sub-networks is trained to extract a corresponding physical property from the pixels of the image 102 into one or a combination of the latent space and the physical space of the corresponding physical property. For example, an albedo sub-network is trained to extract the albedo 104B of the 3D shape 104A of the 3D input object from the pixels of the image 102. In some embodiments, the albedo sub-network may have a style-based generative adversarial network (GAN) architecture. In some exemplary embodiments, the style-based GAN may be trained using a set of photographic images of objects from a predetermined class.

[0034] Additionally, each of the individual separated physical properties are combined into a 2D output image. In some embodiments, the individual separated physical properties are combined using a differentiable renderer. The differentiable renderer renders a 2D output image of image 102 using the physical space of one or more physical properties 104A-104D.

[0035] The processor 206 is configured to present the image 102 received from the input interface 202 to the neural network 210 while manipulating the latent space, the physical space, or both, of one or more physical properties of the 3D input object according to one or more manipulation instructions. The processor 206 is configured to manipulate the physical properties separated by the multiple sub-networks to modify the 3D input object such that the differentiable renderer reconstructs a 2D output image. The 2D output image is output via the output interface 208.

[0036] The isolation of each of the one or more physical properties by a corresponding sub-network of neural network 210 is further described with reference to FIG.

[0037] 3 illustrates a framework 300 of a neural network 210 for manipulating 2D images of predetermined classes of 3D objects according to an exemplary embodiment of the present disclosure. When an image 102 is received, the processor 206 generates a mask image 302 for the image 102. In some exemplary embodiments, the processor 206 may estimate the mask image 302 using a semantic segmentation network. The semantic segmentation network may generate the mask image 302 including some pixels representing background (e.g., pixels having a value of zero) and the remaining pixels representing foreground (e.g., pixels having a non-zero value, such as a value of 1.0). The image 102 is masked with the mask image 302 such that the values ​​of pixels representing 3D objects in the image 102 are preserved in the masked image 304, while the values ​​of pixels representing the background and (in the case of a face) hair in the image 102 are masked with pixel values ​​that are zero. Masking of the pixel values ​​of the segmented background (and hair) produces a masked image 304 that includes pixel values ​​that represent a predetermined class of object (e.g., face) in image 102. This masked image 304 is presented to neural network 210.

[0038] The neural network 210 includes multiple subnetworks that isolate one or more physical properties 104A-104D of the image 102. In some exemplary embodiments, the multiple subnetworks correspond to an encoder architecture, including a shape encoder 306A, an albedo encoder 306B, a pose encoder 306C, and an illumination encoder 306D (hereinafter referred to as encoders 306A-306D), as shown in FIG. 3. Each of the multiple subnetworks, i.e., each of the encoders 306A-306D, processes the image 102 to extract a corresponding one or more physical properties 104A-104D from pixels of the masked image 304. For example, encoder 306A extracts shape 104A, encoder 306B extracts albedo 104B, encoder 306C extracts pose 104C, and encoder 306D extracts illumination 104D.

[0039] One or more physical properties 104A-104D extracted from image 102 may be represented in one or a combination of the latent space and physical space of the corresponding physical property. For example, shape encoder 306A extracts shape code 308A in shape latent space, and albedo encoder 306B extracts albedo code 308B in albedo latent space. In contrast, pose encoder 306C extracts pose representation 308C in pose physical space, and encoder 306D extracts lighting representation 308D in lighting physical space. Each of shape code 308A and albedo code 308B represents a compressed state of the corresponding physical property, i.e., shape 104A and albedo 104B.

[0040] Additionally, the shape code 308A and the albedo code 308B are provided as inputs to corresponding shape generators 310A and 310B, respectively, to generate a shape representation 312A in shape-physical space and an albedo representation 312B in albedo-physical space. In some exemplary embodiments, the shape generator 310A corresponds to a convolution generator that generates a 3D shape 312A of the image 304 from the shape latent space 308A. For example, the generated 3D shape 312A may be composed of three channels in a texture mapping space, such as UV space, that represent the 3D coordinates of the vertices of the 3D shape 312A. In some exemplary embodiments, the albedo representation 312B is represented as an albedo map, which may be used to determine the albedo at a specific or arbitrary location on the 3D shape 312A. For example, the albedo representation 312B may correspond to a red-green-blue (RGB) albedo map in UV space. In some embodiments, albedo representation 312B includes one or more channels for specular reflection. For example, albedo representation 312B may correspond to a red-green-blue specular albedo map having four channels (red diffuse, green diffuse, blue diffuse, and specular) in UV space. Different variations of albedo code 308B in albedo latent space may correspond to different variations of the albedo map of albedo representation 312B in albedo physical space to achieve photorealistic images in the final output image of the image manipulation. To that end, in some embodiments, albedo generator 310B corresponds to a style-based generative adversarial network (GAN).

[0041] In some exemplary embodiments, illumination physical space 308D may be represented using spherical harmonic parameterization, which enables the recovery of physical properties corresponding to illumination (such as illumination 104D) illuminating 3D objects in image 102, which may approximate any unknown lighting conditions. In some exemplary embodiments, pose physical space 308C may correspond to a 6-degree-of-freedom (6-DOF) pose vector, including three parameters for 3D rotation via an axis-angle representation and three parameters for 3D translation.

[0042] The generated 3D shape 312A, albedo map 312B, pose physical space 308C, and lighting physical space 308D are provided as inputs to a differentiable renderer 314. The differentiable renderer 314 generates a rendered image 316 of the 3D face model from the 3D shape 312A and albedo representation 312B using the physical space pose representation 308C and the physical space lighting representation 308D.

[0043] The process, starting from processing image 102 to produce masked image 304, extracting each of the corresponding one or more physical properties 104A-104D from pixels by encoders 306A-306D, generating shape representation 312A and albedo representation 312B, and outputting rendered image 316, corresponds to a face reconstruction pipeline 350 as shown in FIG. 3.

[0044] Additionally, the rendered image 316 of the 3D face model is masked using the mask image 302 to generate a reconstructed masked rendered face image 318. The reconstructed masked rendered face image 318 is combined with a reconstructed 2D masked hair image 330. In parallel, the masked hair image 330 is reconstructed from the image 102 using a hair manipulation algorithm. In some embodiments, the hair manipulation algorithm corresponds to an encoder / generator architecture that includes a hair encoder 320 and a hair generator 324. The encoder 320 extracts hair features 322 corresponding to hair from the image 102. The extracted hair features 322 are provided as input to the hair generator 324. The hair generator 324 generates a 2D hair image, such as hair image 326. The hair image 326 is masked with a hair mask 328 to generate the masked hair image 330.

[0045] Combining the masked rendered face image 318 and the masked hair image 330 generates a combined composite face image 332 from the face in image 102. In some embodiments, the reconstructed combined composite face image 332 is input to a refiner network 334 to generate a photorealistic refined composite face image 336 as the output of the image manipulation of image 102.

[0046]

number

[0047] In some exemplary embodiments, albedo generator 310B corresponds to a style-based generative adversarial network (style GAN) generator. To that end, system 200 may be adversarially trained using a GAN framework using a set of photographic images of human faces to generate photorealistic output images 336 that exhibit different variations of the human faces in image 102.

[0048] A style-based GAN generator can be used to generate realistic images of objects from a predetermined class. The realistic images can be generated by allowing inputs to modulate intermediate layers of a generator neural network (e.g., albedo generator 310B). The output space of the style-based GAN generator can be the same as the space of real-world training data, i.e., the space of 2D images of objects such as faces. Thus, a style-based GAN applied to an adversarial discriminator network in the space of 2D images of objects can train the style-based GAN neural network architecture to directly generate photorealistic 2D images.

[0049] In some embodiments, one or more discriminator networks may be used to train the style-based GAN architecture of albedo generator 310B and of other generators in framework 300, such as one or more of shape generator 310A and hair generator 324. The discriminator network may be applied at one or more stages of framework 300. In some embodiments, for example, the discriminator network is applied to an output or intermediate image of framework 300 after the differentiable renderer, such as one or more of rendered image 316, masked rendered face image 318, combined composite face image 332, and refined composite face image 336.

[0050] In some embodiments, part or all of the neural network 210, for example, the face reconstruction pipeline 350, may be pre-trained, as further described in FIG.

[0051] 4 shows a flowchart illustrating a process 400 for manipulating a 2D image of a predetermined class of 3D object, according to an exemplary embodiment of the present disclosure. Process 400 begins at step 402. The steps of process 400 may be performed by processor 206.

[0052] Step 404 corresponds to pre-training the face reconstruction pipeline 350 of the framework 300 of FIG. 3. The face reconstruction pipeline 350 is implemented by encoders and generators, such as the encoders 306A-306D and the shape and albedo generators 310A and 310B of the framework 300. To that end, in step 404, the encoders 306A-306D and the generators 310A and 310B of the neural network 210 are pre-trained using a pre-training dataset. For example, the face reconstruction pipeline 350 is trained to input images from the pre-training dataset as input images 304 and to output rendered images, such as rendered image 316, as reconstructions of the input images 304. In some embodiments, the pre-training input images 304 and the pre-training output images 316 are unmasked. The face reconstruction pipeline 350 may be pre-trained using a dataset of synthetically generated face images, such as face images generated by a linear 3D morphable model (3DMM). In such pre-training, the face reconstruction pipeline 350 learns to capture facial features using the synthetically generated face images. For example, 80,000 2D face images for pre-training may each be rendered by randomly sampling a 3D head model from a 3DMM, such as the Faces Learned with an Articulated Model and Expressions (FLAME) model, and rendering the sampled model into a 2D image. Before each sampled 3D model is rendered to generate training set images, it may be converted to 3D so that the resulting 2D face images in the training set correspond to the same 2D alignment as faces in the dataset of real face images. The dataset of real face images is also used during the training step 406. Additionally, ground truth values ​​for one or more isolated physical properties, such as ground truth values ​​corresponding to one or more physical properties 104A-104D, are used to generate each of the images in the pre-training set.

[0053]

number

[0054]

number

[0055] During training 406, the neural network 210 is trained based on real 2D images, such as images 102, to generalize the model pre-trained on synthetic facial images in step 404 to tackle real facial images. In some exemplary embodiments, the neural network 210 may be trained using a dataset of facial photographs and images. In some cases, the facial photographs and images may include faces with accessories, such as eyeglasses. In such cases, the accessories, such as eyeglasses, may be excluded from the dataset. The facial mask images 302 (M f ) is obtained automatically for each image in the dataset using a semantic segmentation network. Additionally or alternatively, a face mask image 302 may be obtained using different methods, such as face analysis, facial landmark localization, or manual human annotation. The masked 2D face image 304 is input to the neural network 210. In some embodiments, during the training process 406, the ground truth values ​​of one or more physical properties 104A-104D for the actual face image are unknown, and no reconstruction loss is applied to the one or more physical properties 104A-104D.

[0056] During training 406, the loss functions may include a pixel-wise reconstruction loss between the masked input face image 304 and the reconstructed face image 318, an identity loss that measures the identity similarity between the masked input face image 304 and the reconstructed masked rendered face image 318, a landmark loss that measures the distance between the image plane projection of the 3D facial landmark positions in the input image 102 and the corresponding positions in the 3D shape model 312A, and a non-saturating logistic GAN loss to improve realism in the masked rendered image 318.

[0057] The pixel-wise reconstruction loss between the masked input face image 304 and the reconstructed face image 318 is expressed by equation (8).

[0058]

number

[0059]

number

[0060]

number

[0061] In some cases, during training of neural network 210 for extraction of one or more physical properties 104A-104D, there may be ambiguity in the extracted one or more physical properties 104A-104D. The ambiguity may correspond to the relative contributions of color illumination intensity and surface albedo to the RGB appearance of a skin pixel in an image, such as image 102. To that end, the ambiguity may be overcome based on an albedo normalization loss corresponding to albedo map 312B.

[0062]

number

[0063] The albedo normalization loss (Equation 11) may minimize the projection of the albedo map 312B from a 3DMM (e.g., a FLAME model) trained on real face albedo scans into a principal component space for albedo. Additionally or alternatively, to address the same ambiguity,

[0064]

number

[0065] An illumination normalized loss may be used, which may be expressed as:

[0066]

number

[0067] Furthermore, the shape code 308A(α) and the albedo code 308B(β) are normalized. The shape code 308A(α) and the albedo code 308B(β) are normalized as follows:

[0068]

number

[0069] It is expressed as: In step 408, the masked rendered face image 318 is combined with the masked hair image 330 to obtain a combined composite face image 332. In some embodiments, the hair operations pipeline modifies the appearance of the hair to correspond to the modified pose of the face, as further described in FIG.

[0070] During refinement 410, the combined composite facial image 332 is further processed by a refiner neural network 334 to generate a refined composite facial image 336. The combined composite facial image 332 may not exhibit sufficient variation in areas such as the eye region and may lack certain details such as eyelashes, facial hair, and teeth. Furthermore, because face and hair are processed separately, some combined composite facial images, such as the composite face 332, may have face-hair fusion issues. To that end, the refiner network 334 generates a photorealistic refined composite facial image 336 that overcomes the realism gap between the combined composite facial images 332 and their corresponding original images 102 with minimal changes to the reconstruction. The refined composite image 336 is more photorealistic than the combined composite facial image 332, but is only minimally different from the combined composite facial image 332. In some exemplary embodiments, the refiner neural network 334 may correspond to a convolutional neural network architecture such as a U-net. The U-Net operates on the reconstructed combined composite image 332 and outputs a final image 336. The refiner neural network 334 may be trained using pairs of original and reconstructed face images 332 from a training dataset.

[0071] The refiner neural network 334 may be trained using a set of loss functions, which may include a perceptual loss that measures the perceptual similarity between the refined synthetic image 336 and the input image 102, an identity loss that measures the identity similarity between the refined synthetic image 336 and the input image 102, and a non-saturating logistic GAN loss to improve the photorealism of the refined synthetic image 336.

[0072] At step 412, a reconstructed refined composite image 336 is generated as output including a realistic-looking human facial image based on image 102. In some alternative embodiments, refined composite image 336 may be generated as an anonymized version of the facial image in input image 102. The anonymized version may be generated using an anonymization technique such as a similarity-based metric. The generated anonymized version may enable a user to share photos or videos of one or more people in an input image, such as input image 102, to a public forum that may restrict users from sharing their actual facial images. In such cases, a user may share the anonymized version corresponding to refined composite image 336 in a public forum while protecting the user's actual identity. At step 414, process 400 ends.

[0073] 5 illustrates a framework corresponding to a hair manipulation pipeline 500 according to an exemplary embodiment of the present disclosure. In some exemplary embodiments, the hair manipulation pipeline 500 corresponds to manipulating one or more of the hair image 326, the hair mask 328, and the masked hair image 330. The hair manipulation algorithm 500 may include a Multi-Input-Conditioned Hair Image GAN (MichiGAN), which processes hair attributes such as hair shape, structure, and appearance from the image 102 in a decoupled manner. The hair shape corresponds to a 2D binary mask of the hair region in the image 102. The structure represents a 2D hair orientation map. The appearance refers to the overall color and style of the hair, which may be encoded as a hair appearance code 504 in a hair appearance latent space.

[0074] In some exemplary embodiments, the hair manipulation pipeline 500 may be used to manipulate the shape and structure of hair from an input image 102 without changing the color and style of the hair from the original image 102. To that end, the hair manipulation algorithm 500 extracts hair signatures 504 in a hair appearance latent space from the input image 102.

[0075] In some exemplary embodiments, the hair operations pipeline 500 is coupled with the face reconstruction pipeline 350. The face reconstruction pipeline 350 is used to extract a 3D model 502 of a face in the image 102. The extracted 3D model 502 includes information about one or more physical properties 104A-104D, such as one or more of a shape signature 308A, an albedo signature 308B, a shape representation 312A, an albedo representation 312B, a pose representation 308C, and a lighting representation 308D.

[0076] In some exemplary embodiments, the hair manipulation pipeline 500 is implemented iteratively, for example, to achieve large pose manipulations by manipulating the pose through several intermediate steps that each achieve a smaller pose manipulation. The iterative approach for the hair manipulation pipeline 500 may improve the resulting photorealism of the hair image 326, especially for large pose manipulations. Once extracted from the input image 102, the 3D model 502 and hair signature 504 remain constant throughout all iterations of the hair manipulation pipeline 500.

[0077] In each iteration, the hair manipulation algorithm 500 may use a reference pose 504A from the previous iteration and a target pose 506A. In the first iteration, the reference pose 504A may be the pose 308C extracted from the input image 102. In the last iteration, the target pose 506A may be the final desired pose for the final output image. In each intermediate iteration (called iteration t), the reference pose 504A is equal to the target pose from the previous iteration (i.e., iteration t-1), and the target pose 506A is one step closer to the final desired pose for the final output image. In each iteration, the reference pose 504A and the target pose 506A are used to calculate a 2D distortion field 508. The 2D distortion field 508 represents how image pixels corresponding to the hair move in 2D as a result of the 3D head model 502 transitioning from the reference pose 504A to the target pose 506A. The 2D distortion field 508 may be calculated based on the projection of the 3D vertices of the 3D model 502 onto a 2D image plane (e.g., the 2D image plane of the input image 102) as the pose of the 3D model 502 changes from a reference pose 504A to a target pose 506A. For example, as shown in Figure 5, the 3D vertices of the 3D model 502 in the reference pose 504A may be represented by a face mesh 504B, and the 3D vertices of the 3D model 502 in the target pose 506A may be represented by a face mesh 506B. The 2D distortion field 508 may be obtained by extrapolating the distortion field at image locations that are part of the face to image locations that are not part of the face.

[0078] The reference image 510 input to each iteration is an image or rendering of the face at a reference pose 504A, while the target image 520 output from each iteration is a rendering of the face at a target pose 506A. In the first iteration, the reference image 510 may be the original input image 102. In each intermediate iteration, iteration t, the output image 520 from the previous iteration t-1 may be used as the input reference image 510 for iteration t. Alternatively, or in addition, the combined image 518 from the previous iteration t-1 may be used as the reference image 510 for iteration t. The distortion field 508 is used to warp the reference mask 512A and reference orientation map 512B of the reference image 510. The reference image 510, along with the reference mask 512A and reference orientation 512B, may be obtained from a previous iteration of the hair manipulation algorithm 500. The distortion field 508 warps the reference mask 512A into a distorted mask 514A and the reference orientation map 512B into a distorted orientation map 514B. The distorted mask 514A and the distorted orientation map 514B are each normalized to obtain a corresponding target mask 516A and target orientation 516B, respectively. The hair manipulation algorithm 500 combines the target mask 516A and the target orientation 516B with the hair appearance latent space 504 to generate a target image 518. The target image 518 is input to the refiner neural network 334 to generate a photorealistic refined target image 520. The refined target image 520 from iteration t of the hair manipulation algorithm may be used as the reference image 510 in the next iteration, iteration t+1. At the final iteration, the refined target image may correspond to the reconstructed 2D image 336.

[0079] Each of the reference pose 504A, hair appearance signature 504, face mesh 504B, face mesh 506B, 2D distortion field 508, reference image 510, reference mask 512A, reference orientation map 512B, distorted mask 514A, reference orientation map 512B, distorted orientation map 514B, target mask 516A and target orientation 516B, target image 518, and refined target image 520 is updated with every iteration.

[0080] 6A and 6B together illustrate a method 600 for manipulating 2D images of a predetermined class of 3D objects, according to an exemplary embodiment of the present disclosure. Method 600 begins with operation 602 and ends with operation 606, and is performed by system 200.

[0081] At operation 602, method 600 includes receiving a 2D input image (e.g., image 102) of a predetermined class of 3D input object and one or more manipulation instructions for manipulating one or more physical properties of the 3D input object (e.g., one or more physical properties 104A-104D). The one or more physical properties include a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. In some exemplary embodiments, the object is selected from a predetermined class of human faces.

[0082] At operation 604, method 600 includes presenting a 2D input image to a neural network (e.g., neural network 210) while manipulating a latent space, a physical space, or both, of one or more physical properties of the 3D input object according to one or more manipulation instructions. In some embodiments, the neural network is trained to reconstruct the 2D input image by using multiple sub-networks to isolate physical properties of the 3D input object from pixels of the 2D input image and combining the isolated physical properties produced by the multiple sub-networks into a 2D output image using a differentiable renderer. Each of the multiple sub-networks isolates one or more of the one or more physical properties of the 3D object. In some embodiments, there is one sub-network for each physical property of the 3D input object, and each sub-network of the multiple sub-networks is trained to extract the corresponding physical property from pixels of the 2D input image into one or a combination of the latent space and the physical space of the corresponding physical property. The albedo sub-network, trained to extract the albedo of a 3D input object from pixels of a 2D input image, has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from a predetermined class. Operations are performed on the isolated physical properties generated by the sub-networks to modify the 3D input object before the differentiable renderer reconstructs the 2D output image. In some exemplary embodiments, multiple realistic synthetic images can be generated from the 2D image of the 3D object. The multiple realistic synthetic images can include variations in one or more physical properties of the 3D object's pose, facial expression, and lighting illumination.

[0083] At operation 606, method 600 includes outputting a 2D output image. In some example embodiments, the 2D output image may be utilized in at least one of 3D animation of a 3D object from the 2D input image, hairstyle trials on a human face in the 2D input image, reconstruction of a photorealistic synthetic image of a human face from the 2D input image, and robotic assistance using the reconstructed 2D output image.

[0084] 7 illustrates a block diagram of a system 700 for manipulating 2D images of predetermined classes of 3D objects according to an exemplary embodiment of the present disclosure. System 700 corresponds to system 200 of FIG. 2. System 700 includes a processor 702 and a memory 704. Memory 704 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. Memory 704 is configured to store a neural network 706. Neural network 706 corresponds to neural network 210. Processor 702 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations.

[0085] System 700 also includes an input interface 720 configured to receive a 2D input image of a predetermined class of 3D input object and one or more manipulation instructions for manipulating one or more physical properties of the 3D input object. The one or more physical properties correspond to one or more physical properties 104A-104D, including the 3D shape of the 3D input object, the albedo of the 3D shape of the 3D input object, the pose of the 3D input object, and the lighting illuminating the 3D input object. Additionally or alternatively, input interface 720 may receive the 2D image from a camera, such as camera 724. Some examples of camera 724 may include an RGBD camera.

[0086] The processor 702 is configured to present the 2D images to a neural network 706 stored in a memory 704. The neural network 706 corresponds to the neural network 210.

[0087] In one implementation, a human machine interface (HMI) 714 in the system 700 connects the system 700 to a camera 724. Additionally or alternatively, a network interface controller (NIC) 716 may be adapted to connect the system 700 to a network 718 through a bus 712. In one implementation, 2D images may be accessed from an image dataset 710 via the network 718. The image dataset 710 may be used for pre-training and training of the neural network 706.

[0088] Additionally or alternatively, system 700 may include an output device 726 configured to output a final image, such as reconstructed 2D image 336. Output device 726 may be connected to system 700 via output interface 722.

[0089] Additionally or alternatively, system 700 may include storage 708 configured to store one or more physical properties, such as one or more physical properties 104A-104D of an image, a reconstructed image of the image (e.g., a reconstructed image after one or more physical properties 104A-104D have been manipulated), a latent space of one or more physical properties 104A-104D, etc.

[0090] The data stored in storage 708 may be accessed for further processing over network 718. For example, processor 702 may access storage 708 over network 718 to utilize the 2D output images in at least one of: 3D animation of a 3D object from the 2D input images, hairstyle trials on a human face in the 2D input images, reconstruction of a photorealistic synthetic image of a human face from the 2D input images, and robotic assistance using the reconstructed 2D output images.

[0091] FIG. 8 illustrates a use case scenario 800 for manipulating 2D images of a predetermined class of 3D objects, according to an exemplary embodiment of the present disclosure. Use case scenario 800 corresponds to an application for generating a cartoon or animated version of a user, such as user 802, using system 700. For the generation of such an animated version of user 802, system 700 may be trained using an animated training dataset. In some cases, the animated version of user 802 may correspond to an anonymized version of user 802. The application may be installed on user device 804. In one exemplary scenario, user 802 may capture a 2D image including a frontal view of user 802's face. The 2D image may be shared to system 700 over a network, such as network 718. System 700 processes the 2D image of user 802. After processing, system 700 generates one or more animated versions 806. The one or more animated versions 806 may include facial images of the user 802 in different poses and / or expressions, different animated photographs of the user 802, and the like.

[0092] FIG. 9 illustrates a use case scenario 900 for manipulating a 2D image of a predetermined class of 3D object according to another exemplary embodiment of the present disclosure. In the use case scenario 900, a 2D image of a user, such as user 902, can be used to realistically try on different hairstyle looks. In the exemplary scenario, a 2D facial image of user 902 is shared with electronic device 904. In some cases, electronic device 904 can be remotely connected to system 700. In other cases, system 700 can be embedded within electronic device 904. Electronic device 904 sends the 2D image of user 902 to system 700. System 700 generates different images of user 902 with corresponding different hairstyles in a photorealistic manner, such that the different images remain realistic as user 902 changes different head poses and corresponding different hairstyles. User 902 can select a desired hairstyle from the generated different images with different hairstyles.

[0093] FIG. 10 illustrates a use case scenario 1000 for manipulating 2D images of a predetermined class of 3D objects according to yet another exemplary embodiment of the present disclosure. The use case scenario 1000 corresponds to the reconstruction of a photorealistic synthetic image of a human face from a 2D input image. In one exemplary scenario, a surveillance camera may capture a 2D image of a person. The captured 2D image of the person may be a side view of the person's face. In such a scenario, the 2D image may be shared with the system 700. The system 700 may generate different images 1002 of the person from different angles of view, different poses, and / or facial expressions of the person. The different images 1002 may include images with a reconstructed frontal view of the person, which may be used by security personnel to track and capture suspects. For example, an image 1004 with a frontal view of the person may be selected from the different images 1002.

[0094] FIG. 11 illustrates a use case scenario 1100 for manipulating 2D images of a predetermined class of 3D objects according to yet another exemplary embodiment of the present disclosure. The use case scenario 1100 corresponds to a robotic system that uses a reconstructed 2D output image. Such a robotic system may be used in an industrial automation application for retrieving an object from a conveyor belt 1104. There may be times when lighting conditions for the object are poor, which may hinder the robotic system's ability to retrieve the object from the conveyor belt 1104. In such poor lighting conditions, the system 700 may generate a reconstructed image of the object, such as the object 1102, under the poor lighting conditions to assist the robotic system in retrieving the object 1102 from the conveyor belt 1104. To that end, a 2D image of a 3D object, such as the object 1102, may be captured using a device 1106. The device 1106 may include a camera. The 2D image of the object 1102 is shared from the device 1106 to the system 700 for further processing. The system 700 can reconstruct a 2D object with steered lighting illumination to assist a robotic system in retrieving the object 1102 from the conveyor belt 1104.

[0095] In this manner, system 700 can be used to manipulate 2D images of 3D objects using neural networks trained in an unsupervised manner. Unsupervised training of neural networks eliminates the need for 3D data, which may be difficult and / or expensive to obtain, improving overall processing and storage efficiency. Such manipulation of 2D images also helps to achieve photorealistic images in an efficient and feasible manner.

[0096] The foregoing description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the foregoing description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0097] Specific details are given in the above description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Also, the same reference numbers and names in the various drawings refer to the same elements.

[0098] Also, particular embodiments may be described as a process that is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While a flowchart may describe operations as a sequential process, many of the operations may occur in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not described or included in the drawings. Moreover, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the end of the function may correspond to the function returning to the calling function or main function.

[0099] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be performed or at least assisted through the use of a machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0100] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0101] The embodiments of the present disclosure may be embodied as methods, examples of which are provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, while an example embodiment shows actions as sequential, embodiments may be constructed in which actions are performed in a different order than shown, including performing some actions simultaneously. Additionally, the use of ordinal terms such as "first," "second," etc. in the claims to modify claim elements does not, in itself, imply any priority, precedence, or ordering of one claim element relative to another claim element, or any chronological order in which the actions of the method are performed, but is merely used as a label to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms).

[0102] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is, therefore, the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the present disclosure.

Claims

1. 1. An image processing system for manipulating two-dimensional (2D) images of a predetermined class of three-dimensional (3D) objects, the image processing system comprising: an input interface configured to receive a 2D input image of a 3D input object of the predetermined class and one or more manipulation instructions for manipulating one or more physical properties of the 3D input object, the one or more physical properties comprising a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object, the image processing system further comprising: the neural network is trained to reconstruct the 2D input image by isolating the one or more physical properties of the 3D input object from pixels of the 2D input image using a plurality of sub-networks and combining the isolated one or more physical properties produced by the plurality of sub-networks into a 2D output image using a differentiable renderer, each sub-network extracting one or more of the one or more physical properties of the 3D input object, each sub-network of the plurality of sub-networks being trained to extract a corresponding physical property from the pixels of the 2D input image into one or a combination of a latent space and a physical space of the corresponding physical property; an albedo sub-network trained to extract the albedo of the 3D input object from the pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from the predetermined class; and the image processing system further comprises: a processor configured to present the 2D input image received from the input interface to the neural network and to manipulate the latent space, the physical space, or both of the one or more physical properties of the 3D input object in accordance with the one or more manipulation instructions, manipulations being performed on the isolated one or more physical properties generated by the plurality of sub-networks to modify the 3D input object before the differentiable renderer reconstructs the 2D output image; and the image processing system further comprising: an image processing system including an output interface configured to output the 2D output image;

2. The image processing system of claim 1 , wherein the neural network further comprises a refiner neural network trained to improve the visual representation of the modified 3D input object in the 2D output image.

3. The image processing system of claim 1 , wherein the object is selected from a predetermined class of human faces.

4. 4. The image processing system of claim 3, wherein the processor is configured to execute a hair manipulation pipeline for manipulating hair on the human face separately from manipulating the one or more physical characteristics of the human face.

5. 5. The image processing system of claim 4, wherein the processor is configured to execute the hair manipulation pipeline to modify a representation of hair in the 2D output image in response to a manipulation of a pose of a human face in the 2D input image, the hair manipulation pipeline modifying an appearance of the hair to correspond to the modified pose of the human face.

6. The image processing system of claim 1 , wherein the neural network is pre-trained using a dataset comprising a plurality of synthetically generated human face images.

7. 2. The image processing system of claim 1, wherein the processor is configured to generate a plurality of realistic synthetic images from the 2D images of the 3D object, the plurality of realistic synthetic images including variations in the one or more physical characteristics of the 3D object's pose, facial expression, and lighting illumination.

8. The image processing system of claim 1 , wherein the processor is configured to generate a mask image for the 2D input image using a semantic segmentation network.

9. 2. The image processing system of claim 1, wherein the 2D output images are utilized in at least one of: 3D animation of the 3D object from the 2D input images; hairstyle trials on a human face in the 2D input images; reconstruction of a photorealistic synthetic image of the human face from the 2D input images; and a robotic system using the reconstructed 2D output images.

10. The image processing system of claim 1 , wherein the 2D output image is used as at least one of an anonymized version of the 2D input image and a training data sample image for a different system.

11. 1. A method for manipulating two-dimensional (2D) images of a predetermined class of three-dimensional (3D) objects, the method comprising: receiving a 2D input image of a 3D input object of the predetermined class and one or more manipulation instructions for manipulating one or more physical properties of the 3D input object, the one or more physical properties comprising a 3D shape of the 3D input object, an albedo of the 3D shape of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object, the method further comprising: presenting the 2D input image to a neural network and manipulating a latent space, a physical space, or both, of the one or more physical properties of the 3D input object in accordance with the one or more manipulation instructions, the manipulation being performed on the isolated one or more physical properties generated by the plurality of sub-networks to modify the 3D input object before a differentiable renderer reconstructs a 2D output image; the neural network is trained to reconstruct the 2D input image by isolating one or more of the physical properties of the 3D input object from pixels of the 2D input image using the plurality of sub-networks and combining the separated physical properties produced by the plurality of sub-networks into a 2D output image using the differentiable renderer, each sub-network extracting one or more of the physical properties of the 3D input object, each sub-network of the plurality of sub-networks being trained to extract a corresponding physical property from the pixels of the 2D input image into one or a combination of a latent space and a physical space of the corresponding physical property, an albedo sub-network trained to extract the albedo of the 3D shape of the 3D input object from the pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from the predetermined class, the method further comprising: outputting said 2D output image.

12. The method of claim 11 , further comprising using a refiner neural network of the neural network to improve the visual representation of the modified 3D input object in the 2D output image.

13. The method of claim 11 , wherein the objects correspond to a predetermined class of human faces.

14. The method of claim 13 , further comprising executing a hair manipulation pipeline for manipulating hair on the human face separately from manipulating the physical characteristics of the human face.

15. 15. The method of claim 14, further comprising: executing a hair manipulation algorithm to modify the representation of the hair in the 2D output image in response to a manipulation of a pose of an input human face in the 2D input image, the hair manipulation algorithm being trained to synchronize the appearance of the hair with the modified pose of the input human face.

16. The method of claim 11 , further comprising pre-training the neural network using a dataset containing synthetically generated human face images.

17. 12. The method of claim 11, further comprising generating a plurality of realistic synthetic images from the 2D images of the 3D object, the plurality of realistic synthetic images including variations in the one or more physical characteristics of pose, facial expression, and lighting illumination of the 3D object.

18. The method of claim 11 , further comprising generating a mask image for the 2D input image using a semantic segmentation network.

19. 12. The method of claim 11, further comprising utilizing the 2D output images in at least one of: 3D animation of the 3D object from the 2D input images; hairstyle trials on a human face in the 2D input images; reconstruction of a photorealistic synthetic image of the human face from the 2D input images; and a robotic system using the reconstructed 2D output images.

20. The method of claim 11 , further comprising utilizing the 2D output image as at least one of an anonymized version of the input image and a training data sample image for a different method.

Images