Systems and methods for generating two-dimensional (2D) images of objects in a scene

Abstract

An AI image processing system is provided. The system is configured to receive latent parameters defining a representation of attributes of an object in an image. The latent parameters comprise a plurality of subsets of latent parameters, and each subset of latent parameters corresponds to one attribute. The system is configured to generate a joint encoding of the latent parameters, and generate an implicit 3D model of the object based on the joint encoding. The implicit 3D model defines, for each 3D position in a scene, a volume representation relative to an input viewing direction. The volume representation defines signed distance values and / or optical densities, as well as color and / or feature vectors. The system is configured to render a 2D image of the scene including the object viewed from the input viewing direction based on the volume representations generated for different 3D positions in the scene.

Description

Technical Field

[0001] This disclosure generally relates to computer-generated images and computer vision, and more specifically, to a system and method for generating two-dimensional (2D) images of three-dimensional (3D) objects. Background Technology

[0002] Image generation is now widely used in the fields of vision and graphics. Three-dimensional deformable models (3DMMs) can be used to generate faces and other 3D objects. For example, the FLAME model and the Basel face model are linear 3DMMs that can be used to synthesize 3D models of faces, and 2D images can be rendered from these 3D models. Linear 3DMMs offer controllability and allow for the decoupled editing of certain features of the generated image. However, linear 3DMMs are limited by the correspondence assumption, which means that for any two faces modeled by a 3DMM, every point in one face corresponds exactly to a point in the other face, and vice versa. Furthermore, because linear 3DMMs are linear models based on principal component analysis (PCA), faces synthesized from these models lack fine details in shape and appearance and are generally not considered photorealistic.

[0003] To address this issue, nonlinear 3D face synthesis methods have been continuously developed. Compared to linear 3DMMs, nonlinear 3DMMs demonstrate stronger performance in 3D face generation, resulting in higher quality 2D facial images. However, the photorealism of images generated using nonlinear 3DMMs falls far short of that generated by Generative Adversarial Networks (GANs). 2D GANs demonstrate the ability to generate highly realistic images by training a generator network to produce data indistinguishable from real samples, while a discriminator network provides feedback to guide the training process. However, although GAN models can generate high-quality, photorealistic facial images, they are black-box models, with limited or no ability to independently model or modify specific attributes (such as texture, shape, lighting, hair, and other elements of the generated image).

[0004] In 3D implicit neural representations, a neural network encodes an implicit representation of a 3D scene that can be rendered using volumetric rendering. This neural network (hereinafter referred to as a deep implicit function neural network) can be trained unsupervised from a set of 2D images of an object to learn an implicit representation of the object's 3D structure, which can be used to render novel 2D views of the object.

[0005] Recently, 2D-styled GANs have been combined with 3D implicit neural representations to simultaneously achieve photorealism (due to the use of 2D-styled GANs) and the ability to learn detailed 3D object representations from 2D images (due to the use of 3D implicit neural representations). This paper refers to these combined models as Implicit 3D GANs.

[0006] Implicit 3D GANs can synthesize 3D structures and high-fidelity textures of objects such as faces. Unlike 3DMM, implicit 3D GANs can model structures that do not follow the correspondence assumption, including highly complex structures such as hair and facial parts that are not always visible (such as the inside of the mouth).

[0007] However, existing implicit 3D GANs do not allow for the controlled generation or editing of objects in 3D scenes (such as 3D faces). In some cases, inversion-based methods are used to edit the generated images to a certain extent. However, these methods are complex, cumbersome, and have limited effectiveness in achieving photorealistic editing of images.

[0008] Therefore, a technical solution is needed to generate and render 2D images of objects in a 3D scene in a photorealistic manner. More specifically, it is necessary to manipulate the properties of 3D objects in the 2D image so that the resulting (edited) 2D image of the object retains photorealism in an efficient and feasible way. Summary of the Invention

[0009] Therefore, the purpose of this disclosure is to provide a system for generating and rendering 2D images of 3D objects in various scenes, a system capable of editing and controlling multiple attributes of the objects. The image of the 3D object is a detailed 2D rendering of the 3D object in the scene.

[0010] In recent years, the development of high-fidelity 3D object generation technology and 2D image generation technology has accelerated significantly. Some embodiments of this disclosure are based on the understanding that methods for synthesizing photorealistic 2D images of objects (e.g., 2D GANs) struggle to independently modify individual properties of objects in an image. Often, modifying one property of a rendered image unintentionally modifies other properties. The inability of the properties of a generative model to change independently is known as entanglement. In 2D image generation techniques that do not employ 3D models (e.g., 2D GANs), pose is often entangled with other properties. For example, it is difficult to modify the latent code used to generate an image by modeling only the 3D pose of the drawn face without unintentionally modifying other facial properties in the image, such as lighting, shape, texture, hair, or background.

[0011] Some embodiments of this disclosure aim to provide a system and method for generating an editable 3D model of a face (e.g., a human face). For example, a face includes facial features such as shape, size, and albedo, as well as hair and / or parts of the head corresponding to the face. These parts of the head may, for example, correspond to the top of the head, the sides of the head, and the back of the head. For this purpose, an editable 3D model can be used to control or edit multiple facial features and hair and / or parts of the head. In some cases, the editable 3D model also allows control over the background of the face.

[0012] Some embodiments of this disclosure are based on the understanding that generating 3D objects and rendering them as images (i.e., AI-generated images) has numerous practical applications in fields including AR / VR, media, fashion, advertising, medical imaging, robotics, architectural design, marketing, avatar creation, and human-computer interaction.

[0013] Some implementations are based on the understanding that techniques based on implicit 3D GANs (such as StyleSDF) combine the advantages of different models. Specifically, implicit 3D GANs combine the advantages of 3D implicit neural representations such as Neural Radiation Field (NeRF) and Signed Distance Field (SDF) with the advantages of 2D style-based generative adversarial networks (style-based GANs) such as StyleGAN and StyleGAN2. In some cases, the implicit neural representations as part of implicit 3D GANs can use a three-plane frame, which can improve the computational efficiency of the 3D implicit neural representations and also improve the photorealistic quality of the 2D rendered images.

[0014] While implicit 3D GAN methods can directly manipulate 3D viewpoints, they cannot provide any explicit decoupled control over the attributes (characteristics) of the generated objects in the generated images. In other words, although implicit 3D GANs can generate highly realistic objects using 3D representations, they cannot easily and independently edit or control the attributes of the generated objects and images.

[0015] Examples of these attributes include the shape, albedo, lighting, pose, and background of the object in the generated image. When the object is a face, these attributes may also include shape related to identity, shape related to facial expression, and hair attributes. The techniques for generating images of 3D objects can be applied to faces, as well as other categories of objects such as cars, hands, human bodies, clothing, electronic devices, and other types of objects with different attributes.

[0016] Some embodiments of this disclosure are based on the understanding that existing methods capable of editing implicit 3D GANs cannot provide independent semantic control over specific facial attributes and may also lack photorealism.

[0017] Some embodiments of this disclosure are based on the understanding that linear and nonlinear 3D deformable models (3DMMs) offer improved quality and provide a degree of control over the properties of the generated images. However, linear and nonlinear 3DMMs still do not achieve the image quality achieved by GANs.

[0018] Some implementations are based on the understanding that the attributes of objects in the generated images cannot be controlled or edited. These shortcomings stem at least in part from the principles of generative AI, which generates 3D implicit neural representations from latent code vectors randomly sampled from a distribution. According to these principles, all attributes of the generated objects in an image depend on the latent code vector z (e.g., sampled from a multidimensional Gaussian distribution), but there is no explicit relationship between the elements of the latent code and the attributes of the generated objects. In one example, there is no explicit relationship between the latent code vector z and the attributes of the generated objects because the latent code vector z is optimized to match the generated distribution to the training data distribution, rather than for interpretable representations.

[0019] Some implementations are based on the understanding that in implicit 3D GANs, there is no explicit relationship between the latent code vector and the various properties of the image generated by the implicit 3D GAN.

[0020] Some implementations are based on the understanding that in implicit 3D GANs, the way latent code vectors encode different attributes of an image can be inherently entangled (e.g., the influence of the latent code on each attribute may not be separable into a linear subspace) because the network is not optimized for the separability of attributes in the latent space. However, some implementations are based on the understanding, demonstrated through modeling and experiments, that entangled latent code vectors can be replaced with multiple latent subvectors (i.e., latent parameters) dedicated to different attributes. By processing in this way, some attributes can be controlled without altering others by changing only the latent parameters corresponding to the attribute of interest, while reusing the latent parameters corresponding to the attributes of the object that are desired to be preserved. This editing and / or controlled generation is possible because the implicit 3D GAN can be trained in a decoupled manner using the relationship between the latent parameters and the corresponding attributes of the object learned offline.

[0021] Some implementations are based on the understanding that combining the photorealism of 2D GANs with the controllability of 3DMMs can have limitations. For example, some methods use 3DMMs to learn how to control the properties of style-based GANs, but their controllability is limited by the degree of 3D decoupling that 2D style-based GANs can learn. In these methods, for example, the inherent 2D nature of the style-based GANs they are based on hinders the decoupling of 3D pose from other attributes. One such method, the 3D Deformable Style GAN (MOST-GAN), is a non-linear 3DMM in which a style-based GAN (in this case, StyleGAN2) architecture is used to model texture maps. Because the 3DMM used in this method requires point-to-point correspondences between different individuals, MOST-GAN can model the 3D shape and texture of the face, but it cannot synthesize or modify hair regions in a fully 3D manner because there are no point-to-point correspondences between different objects in the hair region. Instead, MOST-GAN resorts to 2D methods for hair generation and editing, which leads to inconsistencies in hairstyle and hair texture during 3D pose changes.

[0022] Some embodiments of this disclosure aim to provide a method for generating and editing images of 3D objects, or a model that combines the advantages of controllable image attributes provided by a nonlinear 3DMM model with the high fidelity and photorealism offered by an implicit 3D GAN. Because this image generation and editing method can decouple image attributes from editing by controlling the latent code of the implicit 3D GAN model, it is referred to as Controllable Latent StyleSDF (or CoLa-SDF).

[0023] Some implementations aim to generate models with decoupled latent code vectors that can be used to generate 3D implicit neural representations based on a set of latent parameters. In one example, the latent code vectors can be divided into multiple subsets of latent subvectors or latent parameters, where each latent subvector or latent parameter subset is responsible for a specific attribute of an image rendered using the 3D implicit neural representation. For example, the model is trained to learn decoupled latent code vectors; these decoupled latent code vectors allow the image to be modified based on only a single or subset of latent parameters corresponding to one or more specific attributes to be changed, while preserving other attributes of the image.

[0024] Therefore, some implementations aim to modify certain properties of the generated image of a 3D object without using another randomly generated latent code vector to generate entirely new objects and images from scratch.

[0025] According to some implementations, predefined values ​​of multiple attributes of objects in an image are used to generate corresponding latent parameters in the image. In one example, the mapping between attributes and corresponding latent parameters is predefined and fixed. This predefined mapping may be based on a pre-trained encoder of a pre-trained 3DMM (which performs the mapping from the image to the corresponding latent parameter). In some implementations, the encoder may be an encoder of a pre-trained nonlinear 3DMM (e.g., MOST-GAN). Furthermore, the expected value of the attribute is directly mapped to the value of the latent parameter. In this way, the model can learn to modify or control the image based on attributes, using decoupled latent codes for each of the multiple attributes. However, this method can make generative AI models difficult to train because the sampling distribution (e.g., a Gaussian distribution) used for the latent parameter set is highly unlikely to match the overall distribution of attributes in the training dataset (e.g., the distribution of encoded attributes determined by applying the pre-trained encoder to images in the image training dataset). Therefore, the learned generative AI model may not accurately capture the variations present in the generated image, or may not be able to independently control the attributes of the generated image by changing the corresponding subset of the latent code of the generative AI model.

[0026] According to some implementations, the model is configured to learn the mapping between multiple attributes of an object and their corresponding latent parameters, such that another deep neural network instructing these mappings is learned during the training of the generative AI model. While this approach may be effective, it introduces additional parameters and an additional model that needs to be trained. This can lead to reduced training efficiency or getting stuck in poor local minima during training. It can also result in generating poor-quality, controllable generative models.

[0027] According to some implementations, the mapping between the latent parameter set of objects in an image and multiple attributes is based on statistical analysis of the distribution of image data in a dataset. These mappings aim to capture the relationship between the latent parameter set and multiple attributes.

[0028] According to some implementations, the mapping between the numerical values ​​of an object's attributes and the values ​​of latent parameters is determined such that the distribution of the latent parameter set throughout the dataset approximately matches a known distribution (e.g., a Gaussian distribution) of the numerical values ​​of these attributes, from which it is easy to sample the latent parameter set.

[0029] According to some implementations, principal component analysis (PCA) is applied to the numerical distribution of each attribute in the training dataset to achieve or determine the mapping between the numerical values ​​of multiple attributes of an object and the latent parameter values. In some implementations, these attribute values ​​across the entire training set are determined by applying a pre-trained encoder of a pre-trained 3DMM (e.g., a non-linear 3DMM MOST-GAN) to images in the dataset. Performing PCA on the distribution of these numerical attributes across the entire dataset yields a bidirectional mapping between the attribute values ​​in the population and a known multidimensional Gaussian distribution from which random values ​​can be easily sampled. In some implementations, PCA is applied to the values ​​of each attribute or subset of attributes separately. For example, performing PCA on the distribution of 3D shape attribute values ​​yields a mapping between a Gaussian random vector with a known covariance matrix and the distribution of 3D shape attribute values ​​across the entire training set. In some implementations, the Gaussian vector can be truncated, for example, by removing a subset of PCA basis vectors with the smallest eigenvalue. In some implementations, the covariance matrix is ​​made an identity matrix by multiplying each PCA basis vector by a constant determined according to its eigenvalue. Therefore, by sampling from a standard multidimensional Gaussian distribution N(0,I), random samples of the values ​​for each individual attribute (e.g., a 3D shape attribute vector) can be obtained, which approximately match the overall distribution of that attribute across the entire training dataset, where N(0,I) represents a normal distribution with a mean of zero and a covariance matrix equal to the identity matrix. This allows generative AI models (e.g., CoLa-SDF models) to be trained to reproduce the statistical distribution of multiple attributes across the entire dataset, while still allowing independent random sampling from the Gaussian latent parameter distribution for each attribute during both training and inference of the generative AI model.

[0030] In one aspect, this disclosure provides an artificial intelligence (AI) image processing system for generating and rendering images of a scene including objects viewed from one or more viewing directions. The AI ​​image processing system includes at least one processor and a memory storing instructions that cause the at least one processor of the AI ​​image processing system to receive a set of latent parameters defining representations of at least two attributes among the shape, albedo, illumination, pose, and background of an object in the image. When the object is a face, the set of attributes that can be defined by the latent parameters may include hair, which can be modeled as an independent hair attribute or combined with the background as a joint background / hair attribute. The latent parameter set includes multiple subsets of latent parameters. Furthermore, each subset of latent parameters corresponds to one of the at least two attributes of the object in the image. The at least one processor is also configured to submit the latent parameter set to a renderer mapping neural network trained to produce a joint encoding of the multiple parameters, and to submit the joint encoding to a deep latent function neural network trained to produce an implicit three-dimensional (3D) model of the object given the joint encoding. The implicit 3D model defines a volumetric representation relative to the input viewing direction for each 3D location in the scene including the object. The volume representation is defined by one or a combination of a signed distance value and optical density, and by one or a combination of a color and a feature vector. At least one processor is also configured to submit the volume representations generated for different 3D locations in the scene to an image renderer configured to render a 2D image of the scene including objects viewed from the input viewing direction.

[0031] According to some implementations, in order to modify the attribute of an object in a 2D image of a scene, the processor is configured to modify the latent parameter corresponding to that attribute of the object in the latent parameter set, and generate a modified image of the object based on the combination of the modified latent parameter and the remaining unmodified latent parameters in the latent parameter set.

[0032] According to some implementations, in order to modify the attribute of an object, the processor is configured to sample the probability distribution of the latent parameters corresponding to the attribute of the object to generate random values ​​for the modified latent parameters.

[0033] According to some implementations, the memory stores a mapping between the values ​​of the potential parameter set and the corresponding values ​​of the object's attribute.

[0034] According to some implementations, in order to modify the attribute of an object, the processor is configured to receive the expected value of the attribute to be modified in the image of the object, and use the stored mapping to determine the expected value of the modified potential parameter corresponding to the expected value of the attribute.

[0035] According to some implementations, this mapping is based on statistical analysis of the distribution of images in the dataset.

[0036] According to some implementations, the mapping is derived using principal component analysis (PCA) of attribute values ​​from a training image dataset of objects belonging to the same category (e.g., faces).

[0037] According to some implementations, the processor is configured to obtain modified latent parameters by using an object encoder to match the parameters of the input image. The object encoder (e.g., a pre-trained encoder of a pre-trained nonlinear 3DMM such as MOST-GAN) is configured to determine the numerical value or a set of numerical values ​​of the corresponding attribute in the input image.

[0038] According to some implementations, the processor is configured to use an object encoder to train a deep hidden function neural network, which determines the numerical values ​​of attributes based on training images of training objects.

[0039] According to some implementations, the processor is configured to use an object encoder to train a deep hidden function neural network based on a set of parameter consistency loss functions to enforce consistency between the received set of latent parameters used to generate a scene image and the parameters that encode the attributes of the rendered 2D image of the scene.

[0040] According to some implementations, the processor is configured to train a deep latent function neural network using an object encoder based on a set of image-based consistency loss functions, so as to minimize the correlation between latent parameters related to the background of the training object and the appearance of the training object, and to minimize the correlation between latent parameters related to the appearance of the training object and the appearance of the background of the training object.

[0041] According to some implementations, the object is the human head.

[0042] According to some implementations, the processor is configured to use an object encoder to train a deep latent function neural network based on a set of image-based consistency loss functions to minimize the correlation between latent parameters related to the background and hair of a person's head, or the set of parameters, and the facial appearance, and to minimize the correlation between latent parameters related to other attributes of a person's head, or the set of parameters, and the appearance of the background and hair of a person's head.

[0043] According to some implementations, the processor is configured to partition the multidimensional latent code of a scene into a set of latent parameters that define representations of at least two attributes of objects in an image, and to introduce an additional subset of latent parameters as part of the latent parameter set. This additional subset of latent parameters defines a variable representation of facial appearance factors of a human head.

[0044] According to some implementations, the processor is configured to execute modules of an AI image processing system. These modules may include a renderer mapping neural network to generate joint encodings based on a set of latent parameters representing at least two attributes of an object in an image. These modules may include a deep implicit neural network configured to generate one or more low-resolution images, low-resolution feature maps, and 3D shapes corresponding to object surfaces based on the input viewing direction and the joint encodings of the latent parameter set. These modules may include an image generator configured to render a high-resolution 2D image scene, including objects viewed from the input viewing direction, based on one or more low-resolution images, low-resolution feature maps, 3D shapes corresponding to object surfaces, and the joint encodings of the latent parameter set. The joint encodings of the latent parameter set may be transformed by the mapping neural network before being provided to the high-resolution image generator.

[0045] According to some implementations, a high-resolution image generator is configured to render a high-resolution 2D image of a scene, including objects viewed from the input viewing direction, instead of a 2D image of the scene rendered by an image renderer.

[0046] According to some implementations, shape attributes represent the three-dimensional (3D) shape of an object, and pose attributes represent the pose of the object in 3D space, or the 3D transformation between the object's pose and the camera's pose.

[0047] According to some implementations, lighting properties are represented using spherical harmonics.

[0048] According to some implementations, deep hidden function neural networks are signed distance function (SDF) neural networks.

[0049] According to some implementations, deep hidden function neural networks are neural radiation field (NeRF) neural networks.

[0050] On the other hand, a method is provided for generating and rendering an image of a scene including objects viewed from one or more viewing directions. The method includes receiving a set of latent parameters defining representations of at least two attributes of an object in the image: shape, albedo, illumination, pose, and background. When the object is a human head, the list of attributes may include hair, which may be modeled as a separate hair attribute or combined with the background as a joint background / hair attribute in an alternative list of background attributes. The latent parameter set includes multiple subsets of latent parameters. Furthermore, each subset of latent parameters corresponds to one of at least two attributes of the object in the image. The method also includes submitting the latent parameter set to a renderer mapping neural network trained to produce a joint encoding of the set of multiple parameters. The method further includes submitting the joint encoding to a deep latent function neural network trained to produce an implicit 3D model of the object given the joint encoding, the implicit 3D model defining a volume representation relative to an input viewing direction for each 3D location in the scene including the object. The volume representation defines one or a combination of a signed distance value and optical density, and one or a combination of color and feature vectors. The method also includes submitting volumetric representations generated for different 3D locations in the scene to an image renderer or image generator configured to render a 2D image of the scene including objects viewed from the input viewing direction, wherein the 2D image may be a low-resolution or high-resolution image.

[0051] In another aspect, a non-transitory computer-readable storage medium is provided on which a program executable by a processor is implemented to perform a method. The method includes receiving a set of latent parameters defining representations of at least two attributes of an object in an image: shape, albedo, illumination, pose, and background. When the object is a face, the list of attributes may include hair, which may be modeled as an independent hair attribute or combined with the background as a joint background / hair attribute. The latent parameter set includes multiple subsets of latent parameters. Furthermore, each subset of latent parameters corresponds to one of at least two attributes of the object in the image. The method also includes submitting the latent parameter set to a renderer mapping neural network trained to produce a joint encoding of the set of multiple parameters. The method further includes submitting the joint encoding to a deep latent function neural network trained to produce an implicit three-dimensional (3D) model of the object given the joint encoding, the implicit 3D model defining a volume representation relative to an input viewing direction for each 3D location in a scene including the object. The volume representation defines one or a combination of a signed distance value and optical density, and one or a combination of color and feature vectors. The method also includes submitting volumetric representations generated for different 3D locations in the scene to an image renderer configured to render a 2D image of the scene including objects viewed from the input viewing direction.

[0052] The image renderer can also be configured and trained to generate one or more of the following based on a joint encoding of the input viewing direction and the latent parameter set: a low-resolution image, a low-resolution feature map, and a 3D shape corresponding to the object's surface. These modules may include a high-resolution image generator configured to render a high-resolution 2D image of a scene, including objects viewed from the input viewing direction, based on a joint encoding of one or more low-resolution images, low-resolution feature maps, a 3D shape, and the latent parameter set. The joint encoding of the latent parameter set can be transformed by a mapping neural network before being provided to the high-resolution image generator.

[0053] Therefore, some implementations disclose an AI image processing system for generating and rendering images of a scene including objects viewed from one or more viewing directions. The AI ​​system generates an image of an object based on multiple latent parameters that respectively define representations of the object's shape, albedo, lighting, pose, and background. This functionality allows modification of a specific attribute of the generated image by altering the latent parameters representing that attribute. When the object is a face, the list of attributes may include hair, which may be modeled as a separate hair attribute or combined with the background as a joint background / hair attribute.

[0054] Some implementations are based on the understanding that deep hidden function neural networks can be trained in a way that makes specific latent parameters of the decoupled latent code vector correspond to specific attributes of the generated image.

[0055] The present disclosure will be further described below with reference to several accompanying drawings, through non-limiting examples of exemplary embodiments, wherein the same reference numerals denote similar parts in several views of the drawings. The drawings shown are not necessarily drawn to scale and generally focus on illustrating the principles of embodiments of the present disclosure. Attached Figure Description

[0056] [ Figure 1 ]

[0057] Figure 1 An exemplary environment is shown for implementing an AI-based image processing system for generating scene images according to one or more embodiments.

[0058] [ Figure 2 ]

[0059] Figure 2 An exemplary environment for extracting mapping relationships according to one or more implementations is shown.

[0060] [ Figure 3 ]

[0061] Figure 3 The mapping between potential parameter values ​​and attribute values ​​for a specific attribute in an image according to one or more implementation methods is shown.

[0062] [ Figure 4A ]

[0063] Figure 4A A framework for an object encoder for generating meaningful encodings of images, according to one or more embodiments, is shown.

[0064] [ Figure 4B ]

[0065] Figure 4B An example block diagram of an implicit 3D GAN according to one or more implementations is shown.

[0066] [ Figure 5 ]

[0067] Figure 5 An example block diagram of an image processing system according to one or more embodiments is shown.

[0068] [ Figure 6A ]

[0069] Figure 6A An example flowchart of a method for identifying a potential set of parameters according to one or more embodiments is shown.

[0070] [ Figure 6B ]

[0071] Figure 6B The training process for training a deep hidden function neural network is illustrated according to one or more embodiments.

[0072] [ Figure 7 ]

[0073] Figure 7 An example flowchart of a training method for an AI image processing system according to one or more embodiments is shown.

[0074] [ Figure 8 ]

[0075] Figure 8 An example flowchart of a method for training a system based on a consistency loss function, according to one or more embodiments, is shown.

[0076] [ Figure 9 ]

[0077] Figure 9 An example block diagram of an AI-based image processing system according to one or more embodiments is shown.

[0078] [ Figure 10A ]

[0079] Figure 10A An example flowchart of a method for modifying an image according to one or more embodiments is shown.

[0080] [ Figure 10B ]

[0081] Figure 10B Another example flowchart for modifying an image is shown according to one or more embodiments.

[0082] [ Figure 11A ]

[0083] Figure 11A A set of exemplary AI-modified images according to one or more implementation methods are shown.

[0084] [ Figure 11B ]

[0085] Figure 11B A set of exemplary AI-generated images according to one or more implementation methods are shown.

[0086] [ Figure 11C ]

[0087] Figure 11CA set of exemplary AI-modified images according to one or more implementation methods are shown.

[0088] [ Figure 12A ]

[0089] Figure 12A The illustration shows AI-modified images based on attributes from a source image, according to various implementation methods.

[0090] [ Figure 12B ]

[0091] Figure 12B The illustration shows AI-modified images based on attributes from a source image, according to various implementation methods.

[0092] [ Figure 12C ]

[0093] Figure 12C The illustration shows AI-modified images based on attributes from a source image, according to various implementation methods.

[0094] [ Figure 12D ]

[0095] Figure 12D The illustration shows AI-modified images based on attributes from a source image, according to various implementation methods.

[0096] [ Figure 13 ]

[0097] Figure 13 A block diagram of a computer- and AI-based image processing system for generating scene images, according to an exemplary embodiment, is shown.

[0098] Although the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated as noted in the discussion herein. This disclosure provides illustrative embodiments by way of illustration, not limitation. Those skilled in the art will be able to conceive of many other modifications and embodiments falling within the scope and spirit of the principles of the embodiments currently disclosed. Detailed Implementation

[0099] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown only in block diagram form to avoid obscuring this disclosure.

[0100] As used in this specification and claims, the terms “for example,” “in other words,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used in conjunction with a list of one or more components or other items, shall be construed as open-ended, meaning that the list should not be considered as excluding other additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal effect or limitation.

[0101] Example Architecture Overview of Artificial Intelligence (AI)-Based Image Processing Systems

[0102] The proposed AI image processing system includes a generative AI-based model capable of modifying 2D output images by altering or controlling certain properties of the 2D output image while preserving other properties. Embodiments of this disclosure disclose a model called the CoLa-SDF model, which combines the controllability of a nonlinear 3D deformable model (3DMM) with the high fidelity of an implicit 3D GAN. Specifically, the CoLa-SDF model enforces a mapping between the latent space of object attributes in the 2D output image and the interpretable physical latent parameters used to generate or edit the 2D output image.

[0103] This disclosure enables high-fidelity image synthesis or generation and subsequent manipulation of 3D images, with complete control over the decoupled latent parameters. The 2D output image is manipulated by decoupling the latent parameter code vector into a set of 2D image latent parameters corresponding to specific attributes of the 3D object shown in the image. These specific attributes include, for example, the object's 3D shape, albedo, pose, texture, lighting, and background of the object in the image. When the object is a face, these attributes may also include hair. The following describes the process in conjunction with... Figures 1 to 13 Further explanation of the technology disclosed herein.

[0104] Figure 1 An exemplary environment 100 is shown for an artificial intelligence-based image processing system 102 (hereinafter referred to as system 102) that implements an image 110 for generating a scene, according to an exemplary embodiment. Image 110 is a 2D representation of a scene. 2D image 110 includes 2D visual representations of objects, scenes, or patterns. According to this disclosure, 2D image 110 may include 2D representations of 3D objects in a scene. Examples of 3D objects may include, but are not limited to, a human head, a car, one or more natural objects (e.g., trees, forests, bodies of water, etc.), clothing, architectural elements (e.g., houses, bridges, buildings, etc.), food, etc.

[0105] According to this disclosure, system 102 is configured to generate new or synthetic images using artificial intelligence algorithms. In another example, system 102 is configured to create images with predefined desired properties. Furthermore, system 102 allows any specific properties of the generated 2D image 110 to be changed or modified to generate the desired image.

[0106] The 2D image 110 is composed of a grid of image elements called pixels. Furthermore, each pixel contains information relating to certain properties of the 3D objects in the image 110. In some cases, the 2D image 110 can be represented as a matrix, where each element (or pixel) in the matrix corresponds to a specific location in the image 110 and stores a value representing its properties.

[0107] Image 110 represents multiple physical properties of a 3D object, which may need to be modified to obtain a desired image of the 3D object and / or scene. In some exemplary embodiments, the multiple physical properties of the object may include, but are not limited to, shape, albedo, texture, pose, lighting or illumination, and background. In one example, the pixels of image 110 may indicate the numerical value of each property of the 3D object at a corresponding location in 2D image 110.

[0108] Some implementations are based on the understanding that a 3D scene contains one or more 3D objects. Furthermore, the purpose of this disclosure is to generate high-resolution synthetic or realistic images of 3D objects within a scene. In this regard, implicit 3D models can represent objects and scenes using parameters of a neural network. In one example, an implicit 3D model can use a neural radiation field (NeRF) to represent objects, querying the density and radiance of 3D points via a network and rendering them as an image using volume rendering. In another example, an implicit 3D model can use a signed distance field instead of a density field to represent object surfaces. However, NeRF and SDF models may only fit a given scene and are not inherently generative. Furthermore, while these methods are capable of generating images that can be rendered as visual images... Figure 1 It can generate 3D scenes for images, but the high computational cost may prevent it from generating high-resolution images in a cost-effective or time-efficient manner.

[0109] In some cases, 3D deformable models (3DMMs) can be used to generate images of 3D objects. A 3D deformable model is a highly controllable model capable of generating images with controllable properties. In one example, the 3DMM for an object (such as a face) is a parametric 3D model that allows explicit control over semantic properties such as the object's shape and albedo. 3DMMs can also be used to control the pose of 3D objects. However, images generated by 3DMMs lack photorealism, fine detail, and expressiveness.

[0110] Nonlinear 3DMMs can provide enhanced expressiveness and photorealism. However, the generative quality of 3DMM models is inferior to that of generative adversarial networks, and their modeling of certain aspects of 3D objects is limited. For example, when the 3D object is a face or a human head, 3DMMs have limited modeling capabilities for features such as hair and teeth because features like hair and teeth lack pointwise correspondences between different objects and are not part of the underlying 3DMM model.

[0111] Alternatively, implicit 3D generative adversarial network (GAN) models can be used to generate images with high-quality photorealism. Implicit 3D GAN models combine 3D implicit neural representation models (which learn detailed 3D object representations from 2D images) with style-based GAN models (which generate images of novel synthetic objects of a particular category with high photorealism). However, existing implicit 3D GAN models do not allow control over specific object attributes for image editing and / or image generation.

[0112] The embodiments of this disclosure are based on the understanding that existing implicit 3D GAN models cannot support the controllability of specific attributes because the latent code vectors used to generate images are entangled. Implicit 3D GANs are trained to produce synthetic data with a distribution similar to the training set, but the dataset used during training is not partitioned according to attribute values ​​(typically unlabeled), and no attribute values ​​are involved during training. Because the relationship between the image attributes and the latent code vectors is not specified during training, this relationship is entangled and difficult or impossible to interpret.

[0113] Some implementations are also based on the understanding that existing implicit 3D GANs and related methods cannot define the relationship between image attributes and the latent code vectors used to generate 3D objects. Therefore, the generative AI technology of this disclosure allows for the generation of controllable images of various 3D objects (e.g., faces, cars, buildings, landscapes, etc.), wherein different parts of the latent code vectors are used to control different attributes of the object.

[0114] Furthermore, some embodiments of this disclosure are based on the understanding that even controllable 3DMMs offer only limited controllability for modeling or editing image attributes used for image generation and / or editing. For example, some 3DMMs may allow modeling or control of the values ​​of shape and texture attributes. However, they cannot allow accurate modeling of the background of 3D objects in a scene. Specifically, when the 3D object is a face, the 3DMM cannot allow modeling of the hair region in a fully 3D manner because there is no point-to-point correspondence between attributes in the hair region.

[0115] This disclosure provides the following technology: combining the ability of implicit 3D GAN models to learn 3D model generation from 2D images with the ability of 3DMM to provide decoupled controllability for image generation and / or image editing.

[0116] In this regard, some implementations are based on the understanding that specific attributes of a 3D object can be independently controlled for image generation and / or image editing by identifying a mapping relationship 112 between different attributes of the 3D object and the Gaussian distribution of attributes in the latent space. For this purpose, for the generation of image 110, the mapping relationship 112 indicates the mapping between the attributes of the 3D object and the latent parameter set 104. Specifically, based on the mapping relationship 112, each of the multiple attributes is explicitly represented as a subset of latent parameters in the latent parameter set 104, making it possible to independently manipulate each attribute of the 3D object to generate image 110.

[0117] In one example, the latent parameter set 104 defines the representation of the properties of a 3D object. In this respect, latent parameters... This can correspond to the shape attributes of image 110. latent parameters This can correspond to the lighting or illumination properties of image 110. latent parameters This can correspond to the albedo or texture properties of image 110. latent parameters This can correspond to the background attribute h of image 110. When the object is a human head, attribute h can represent hair, or it can represent the combination of hair and the background behind the head.

[0118] In one example, during the training phase, a differentiable nonlinear 3DMM module is used to supervise the training of the implicit 3D GAN model, enabling the deep hidden function neural network 106 of system 102 to learn to generate images using decoupled representations of different attributes as a latent parameter set 104. The following... Figure 6A , Figure 6B , Figure 7 and Figure 8 The text further describes the details of the training phase.

[0119] Furthermore, during the exemplary inference phase, random values ​​can be assigned to each subset of latent parameters in the latent parameter set 104. Additionally, a desired input viewing direction 108 is received or determined. The random values ​​of the latent parameter set 104 are provided to the deep hidden function neural network 106 along with the input viewing direction 108. In one example, a perspective view of a 3D object in a 2D image 110 is based on the input viewing direction 108. Furthermore, the deep hidden function neural network 106 is configured to generate a high-fidelity 2D image 110 of the 3D object. For example, the 2D image 110 could be a synthetic image generated based on random values ​​from the latent parameter set 104.

[0120] In one example, the latent parameter 104 of this decoupling could indicate a set of decoupled representations that capture changing factors or variable attributes in a manner independent of each other. This means that each representation should capture only one attribute of the image data, and changing one representation should not affect the others. In one example, this set of decoupled representations has partitioned or decoupled the set of attributes of the object into individual attributes, and encoded each attribute as a separate dimension.

[0121] During another exemplary inference phase, the attributes used to generate image 110 (e.g., , , The specified values ​​of h) are provided to the deep hidden function neural network 106 along with the input viewing direction 108. Furthermore, based on the mapping relationship 112, the latent parameter set 104 can be determined for the specified values. , , and After determining the potential parameter set 104, an image 110 with the desired attributes can be generated.

[0122] It should be noted that in some cases, only specific attribute values ​​can be defined or assigned. In this case, system 102 is configured to determine the values ​​of latent parameters corresponding to the specific attribute based on the assigned values, and to use random values ​​for other latent parameters for which no attribute values ​​have been assigned. In this way, the generated image 110 is partially controlled to have some desired attributes, while the image can exhibit other variations that are not within the scope of the controlled attributes. For this reason, image 110 can be controlled because implicit 3D GANs can directly manipulate 3D viewpoints.

[0123] In one example, the system can be provided with the assigned values ​​of specific attributes by providing an input image and specifying which of its multiple attributes will be used to generate image 110. For instance, if the shape of a 3D object in the input image is to be used as a reference for generating another image, the input image can be encoded to determine the shape of the input image. Value. Additionally, using Value generation latent parameters At the same time, random values ​​can be used for other potential parameters to generate image 110.

[0124] In one example, the properties in the potential parameter set 104 used for corresponding attributes (e.g., ...) can be changed independently or in various combinations. , , The various latent parameters of (and h) (e.g.) , , and This allows for the editing of 3D objects in image 110. To this end, modified or edited images can be generated based on changes to any one or a combination of latent parameters 104 and the mapping 112 between the latent parameter set 104 and the attributes of image 110. Therefore, embodiments of this disclosure use multiple latent sub-vectors, i.e., latent parameters, specifically for different attributes (e.g., shape, albedo, lighting, and hair and / or background). , , and , the latent code vector that replaces entanglement.

[0125] According to some implementations, the deep implicit function neural network 106 is implemented using a neural network based on the signed distance function (SDF). Alternatively, the deep implicit function neural network 106 can be implemented using a neural network based on the neural radiation field (NeRF). In yet another example, the deep implicit function neural network 106 is implemented using a Gaussian splash neural network. These examples of the deep implicit function neural network 106 are merely illustrative and should not be construed as limiting. In other implementations, other generative AI models for image generation can be utilized as the deep neural network 106 to generate the image 110.

[0126] In one example, an implicit 3D GAN, StyleSDF, is generated by merging an SDF-based 3D representation model and a volume renderer with a style-based 2D generator. The SDF-based model and volume renderer render low-resolution feature maps of the 3D model of the 3D object, and the style-based generator generates a view based on these low-resolution feature maps. Figure 1 To the image, i.e., image 110. It is worth noting that SDF-based 3D modeling defines detailed 3D objects, thereby achieving consistent volume rendering.

[0127] In one example, decoupling the latent parameters into latent subvectors allows control over a single desired attribute while keeping other values ​​constant. Specifically, this decoupling control is achieved using StyleSDF or another implicit 3D GAN trained implicitly based on the relationship between latent parameters and their corresponding attributes.

[0128] In addition, volumetric representations generated for different 3D locations in the scene are submitted to an image renderer configured to render a 2D image of the scene that includes objects viewed from the input viewing direction.

[0129] A brief overview of training deep hidden function neural networks

[0130] Reference Figure 2 The extraction of mapping relationship 202 is described according to some implementation methods. In one example, mapping relationship 202 can be identified, determined, or extracted before training a deep hidden function neural network 106 for generating images.

[0131] It is understood that system 102 may include different networks for different functions. For example, object encoder 204 and mapping neural network 206 may be used to extract mapping 202. In one example, mapping 202 may indicate the mapping between attribute representations in the latent space and the numerical values ​​of different attributes in the dataset of the object or 3D image 208.

[0132] According to one implementation, 2D images 208 in a dataset can be analyzed to extract numerical values ​​of attributes from the corresponding images 208. In one example, an object encoder 204 is configured to generate a set of codes for each image based on the attributes of 3D objects in the images 208. For example, the object encoder 204 is configured to determine numerical values ​​of object attributes based on each image 208. Specifically, the object encoder 204 can generate numerical values ​​of attributes of 3D objects (not shown) based on the images 208. In one example, the object encoder 204 can be a conventional encoder or a neural network. More specifically, in one example, the object encoder 204 can be implemented using an encoder of a non-linear 3DMM (e.g., an encoder of a trained MOST-GAN model). In another example, the object encoder 204 can be a linear 3DMM. In yet another example, the object encoder 204 can be implemented using a convolutional neural network (CNN). In yet another example, the object encoder 204 can consist of multiple non-linear regression functions, each trained to obtain values ​​of different attributes of the object based on the image of the object.

[0133] For example, the intensity or color value at each point or location in an image can be used to generate the image's attributes. , , The values ​​of h and are used. In this way, a set of attribute values ​​are extracted as encodings for each image 208 in the training image set of the object image. The distribution of these attribute values ​​across the entire training set of images can be approximated by a Gaussian distribution to reduce dimensionality and determine the mapping relationship. In this regard, the values ​​of these attributes can be transformed into multiple subsets of the latent parameter set 104 (i.e., , , and The value of ).

[0134] In one example, the numerical values ​​of attributes in image 208 can have physical meaning and can be high-dimensional. To reduce the dimensionality of these values ​​and use latent parameters with a Gaussian distribution, it may be necessary to transform the numerical values ​​of these attributes into values ​​of latent parameter set 104. In one example, the image attributes ( , , The values ​​of (and h) are transformed into a vector of latent parameter sets, where the vector has , , The decoupled representations of the values ​​of h, each contained as a separate subvector. , , and .

[0135] For example, for a specific attribute in each of the multiple 2D images 208 (e.g. The value of ) can determine the subvector corresponding to each image. In addition, it can be based on attributes. The value is determined by the change in the value in the set of images 208. and The mapping relationship between them. In one example, it can be used... Gaussian distribution to determine and The mapping relationship.

[0136] In one example, the decoupled set of latent parameters 104 encodes different aspects or attributes of the data (i.e., the image data of training image 208) as individual latent variables or parameters. Therefore, a consistent meaning is provided within the mapping relation 202. In one example, the mapping neural network 206 is configured to be based on... , , The values ​​of h are expressed as , , and To generate mapping relation 202. For example, mapping relation 202 can be a statistical relation, a Gaussian relation, a deterministic relation, or a probabilistic relation. For example, mapping relation 202 or mapping is based on a Gaussian distribution of the attributes of the training images 208 in the dataset (specifically, the latent parameters of the attributes). , , and Statistical analysis of ).

[0137] To learn the mapping relationship 202 of the decoupled latent parameters 104, the dimension of the latent code vector Z used to represent the data in the latent space is partitioned into disjoint subsets. In other words, the dimension of the latent code vector Z is partitioned into latent parameters. , , and The dimensions. In one example, the latent code vector Z can have 256 dimensions. In this case, 128 dimensions can be partitioned among the latent parameter set 104 for various attributes of image 112. For example, these 128 dimensions can be partitioned into , , and In some implementations, 64 unassigned dimensions may be retained to cover any remaining details inside and around the 3D objects in training image 208 that are not covered by these four attributes. For example, when the 3D object is a human head, The dimension can be used to examine fine details in an image, undefined by... The background elements or image aspects not consistently encoded by the object encoder are encoded. In one example, a mapping 202 is stored between the values ​​of the latent parameter set 104 and the corresponding numerical values ​​of the attributes of the 3D objects in image 208 for future inference.

[0138] In one example, to determine which property to assign , , and The dimension of each attribute in the matrix can be used to determine the corresponding data covariance matrix. , , and Perform eigenvalue decomposition, and these data covariance matrices measure the attributes respectively. , , The values ​​of h are the covariance across the entire training image set of 208.

[0139] Furthermore, in one example, the mapping 202 is derived by approximating the distribution of each attribute value across the entire training image 208 as a Gaussian distribution of each subset of the latent parameters 104 using Principal Component Analysis (PCA). Specifically, the mapping 202 is derived by analyzing the embeddings of the values ​​of the latent parameter set 104 of the attributes in the latent space (e.g., the PCA space). In one example, the PCA latent space of the values ​​of each latent parameter in the latent parameter set 104 can be sampled to utilize parameter consistency loss and image-based consistency loss in the embeddings of these values. This is used to extract the mapping 202 between attribute values ​​and latent parameter values ​​in the training image 208.

[0140] In other implementations, the mapping relation 202 can be determined using methods such as t-distributed random neighborhood embedding (t-SNE), linear discriminant analysis (LDA), uniform manifold approximation and projection (UMAP). The following... Figure 3 The details of the PCA mapping are further described in the text.

[0141] Overview of using PCA to generate maps

[0142] Figure 3 This illustrates a mapping 300 between latent parameter values ​​and attribute values ​​for a specific attribute in an image, according to an exemplary implementation. According to this example, a statistical method, Principal Component Analysis (PCA), is used to generate the mapping between latent parameter values ​​and attribute values ​​in the image. It should be noted that using PCA to map attribute values ​​to latent parameters is merely exemplary and should not be construed as limiting. Some implementations are based on the understanding that decoupled latent parameters can be obtained using decoupled parameters of a 3DMM sampled in PCA space, thereby allowing for independent, accurate, and efficient control of the attribute.

[0143] It is understood that an object encoder 204 is used to train a deep hidden function neural network 106, which determines the numerical values ​​of the attributes of the training object based on image 208. For example, the deep hidden function neural network 106 uses the object encoder 204 to learn the mapping between the numerical values ​​of the attributes (hereinafter referred to as attribute values) and the latent parameter values ​​of the corresponding attributes in the PCA space (hereinafter referred to as latent parameter values). Specifically, the object encoder 204 can generate attribute values ​​of a 3D training object (not shown) based on image 208.

[0144] In one example, object encoder 204 is configured to generate attribute values ​​for an image (e.g., image 208 during pre-training or an input image during inference). Multidimensional distribution 302 depicts the distribution of attribute values ​​for attribute w of the image. In one example, attribute values ​​are generated by object encoder 204. The attribute values ​​include attribute values ​​306, 310, and 314, represented as samples from multidimensional distribution 302. Furthermore, a normalized Gaussian distribution 304, depicting latent parameter values ​​(308, 312, and 316), is generated based on a linear transformation of the attribute values ​​306, 310, and 314 of attribute w in multidimensional distribution 302. The latent parameter values ​​308, 312, and 316 in the latent PCA space can represent latent parameters corresponding to attribute w. Potential parameters The latent parameter values ​​308, 312, and 316 are approximately represented by a normalized Gaussian distribution 304. In this way, based on the linear transformation of PCA, the attribute values ​​306, 310, and 314 of attribute w are projected as latent parameter values ​​308, 312, and 316 in the latent space or PCA space onto the normalized Gaussian distribution 304.

[0145] In this respect, the projection vector is limited by the variance of the attribute values ​​across the entire training set. Dimensionality reduction and latent parameter values ​​are obtained by restricting the dimension of the attribute values ​​in the PCA space to a certain number of components that cover most of the variance in the attribute values. Based on the projection vector values ​​of attribute values ​​306, 310, and 314 projected onto a normalized Gaussian distribution 304, latent parameter values ​​308, 312, and 316 corresponding to the attributes in the image are generated or determined.

[0146] To this end, the PCA method is used to transform the values ​​of multiple relevant attributes of the image at each 3D location within the image into a set of linearly uncorrelated latent parameters 104. The transformed values ​​of the attribute or latent parameter set 104 are called the principal components of the attribute. Various patterns are identified by reducing the variance or total variability of the values ​​of multiple attributes at multiple 3D locations in the image. These patterns can correspond to mappings 202 or 110 between attribute values ​​in the image and the corresponding latent parameter values ​​in the PCA space. For this purpose, the latent parameter values ​​308, 312, and 316 in the PCA space correspond to the projections of attribute values ​​306, 310, and 314 in the high-dimensional space of the multidimensional distribution 302 onto the lower-dimensional space of the normalized Gaussian distribution 304.

[0147] In one example, the variance or total variability of the values ​​of each attribute is used to divide the categories and reduce the dimensionality of the corresponding latent parameter values ​​in the PCA space.

[0148] To generate a set of latent parameters 104 corresponding to the attribute values ​​and with reduced dimensionality, an object encoder 204 is used to determine the value of each attribute. Based on the distribution of the values ​​of each attribute throughout the training image set, a covariance matrix can be computed for each attribute w. Subsequently, based on the resulting covariance matrix for each attribute... The eigenvalues ​​and eigenvectors of the covariance matrix determine the dimensions to be retained in the values ​​of each attribute. For example, the eigenvectors corresponding to the larger eigenvalues ​​of the covariance matrix can indicate the directions or dimensions with higher information content in the attribute space. Each covariance matrix corresponding to attribute w... These eigenvectors and eigenvalues ​​are used to compute new values ​​for the latent parameter set. For this purpose, each subset of latent parameters in the latent parameter set 104, i.e. , , and They are independent of each other. For example, the latent parameter values ​​308, 312, and 316 for each latent parameter correspond to the PCA coefficients embedded in the PCA space of the latent parameter values ​​308, 312, and 316.

[0149] According to this example, attribute value 306 is mapped to latent parameter value 308, attribute value 310 is mapped to latent parameter value 312, and attribute value 314 is mapped to latent parameter value 316. Furthermore, for example, since for ,expect Therefore, eigenvalue decomposition of latent parameter values ​​is used to identify the mapping or mapping relationship between the parameter codes of attribute values ​​generated by object encoder 204 and the corresponding latent code vectors or latent parameter values. In one example, the PCA mapping is represented as: (1) in, and They are The top eigenvectors and eigenvalues, and It is the data mean.

[0150] Object Encoder Overview

[0151] Figure 4A A framework 400A for an object encoder 402 for generating meaningful encodings of an image 404, according to an exemplary embodiment of the present disclosure, is shown. In one example, image 404 may be a training image or an inference image. During training, the image encoding is used to identify and extract attributes of the image to generate a mapping 202. Furthermore, during inference, the image encoding is used to identify and extract attributes of the image to generate updated values ​​for one or more latent parameters that need to be changed based on image 404. When image 404 is received, image 404 is submitted to object encoder 402.

[0152] In one example, the object encoder 402 is instantiated as a neural network or ensemble of neural networks that outputs a numerical vector encoding shape, albedo, illumination, pose, and background when given an image as input. In one example, the object encoder 402 is implemented using the encoder portion of a pre-trained (pre-existing) 3D deformable StyleGAN (MOST-GAN) model, which is a non-linear 3DMM comprising a set of encoders. Therefore, using a MOST-GAN neural network as the object encoder 402 is merely exemplary and should not be construed as limiting. In other implementations, the object encoder 402 can be implemented using different regression or analytical models, such as DECA (Detailed Expression Capture and Animation), MICA (Metric Face), or other models or combinations of models capable of providing separate numerical encodings for each of two or more attributes of an object in an image.

[0153] In one example, only a pre-trained MOST-GAN neural network encoder is used to numerically encode image attributes and further generate values ​​for a latent parameter set 104 corresponding to these attributes. The MOST-GAN object encoder 402 includes multiple sub-networks for decoupling multiple physical attributes of the image 404. In some exemplary implementations, such as... Figure 4A As shown, multiple sub-networks correspond to including a shape encoder. 406A, Albedo Encoder 406B, Attitude Encoder 406C, Light or Illumination Encoder 406D and background encoder The encoder architecture of 406E (hereinafter referred to as encoders 406A to 406E). Each of the multiple sub-networks, namely each encoder in encoders 406A to 406E, processes image 404 to extract each physical attribute from the pixels of image 404.

[0154] For example, the shape encoder 406A extracts shape attributes. Albedo encoder 406B extracts albedo or texture attributes. 406C attitude encoder extracts 3D attitude attributes The 406D lighting encoder extracts illumination or lighting attributes. Alternatively, spherical harmonic lighting parameters, and the background encoder 406E extracts the background attribute h.

[0155] Multiple physical attributes extracted from image 404 can be represented in one or a combination of the latent space and physical space corresponding to the physical property. For example, shape encoder 406A extracts shape code 408A in the shape latent space, albedo encoder 406B extracts albedo code 408B in the albedo latent space, and background encoder 406E extracts background feature or hair feature 408E in the background latent space. In contrast, pose encoder 406C extracts 3D pose representation 408C in the physical space, and illumination encoder 406D extracts illumination representation 408D in the physical space. Each of shape code 408A, albedo code 408B, and hair / background feature 408E represents a compressed state of the corresponding physical property, i.e., the shape attribute and the albedo attribute.

[0156] In some exemplary embodiments, the lighting physical space 408C or lighting attributes can be represented using spherical harmonic parameterization. Spherical harmonic parameterization enables the recovery of physical properties corresponding to the lighting (e.g., lighting 104D) illuminating the 3D object in image 404, which can approximate any unknown lighting conditions. In some exemplary embodiments, the pose latent space 408D can correspond to a 6-DOF pose vector, which includes three parameters for 3D rotation represented by axis angles and three parameters for 3D translation. Therefore, pose attributes represent the pose of the 3D object in 3D space based on the pose vector.

[0157] In one example, when the 3D object is a human head or face, the background encoder 406E can use the encoder part of the hair manipulation algorithm to extract hair features 408E.

[0158] In one example, shape code 408A, albedo code 408B, pose code 408C, lighting code 408D, and background code 408E are used to generate values ​​for latent parameter set 414, i.e. , , , and The values ​​of the latent parameter set 414 are generated based on separate PCA performed on the values ​​of each attribute (i.e., shape 410A, lighting 410D, albedo 410B, pose 410C, and background or hair 410E).

[0159] In one example, encoder 402 can be used to generate attribute encodings for image 404 of objects of a predetermined category. A realistic image can be generated by a generator (e.g., an implicit 3D GAN 420) that uses values ​​of a latent parameter set 414 to generate a 3D volumetric representation of the 3D object and renders the 3D volumetric representation as output image 416. Specifically, volumetric representations are generated for different 3D locations in the scene. Furthermore, the 3D volumetric representation is submitted or transmitted to an image renderer configured to render a 2D image as output image 416 of the scene, including the object viewed from the input viewing direction. The image renderer can render the 2D output image 416 on a display device or store the 2D output image 416 for future rendering and / or analysis. For example, the display device could be a computing device, monitor, screen, projector, television, etc.

[0160] In some implementations, one or more attributes of image 404 may not be encoded by an encoder. In one example, the object background (and / or hair) is not encoded. In this case, there is no background encoder 406E, background code 408E, and background PCA 410E. In this case, it may be impossible to use the encoder to train the implicit 3D GAN 420 so that a specific subset of the latent parameters controls the appearance of the background (and / or hair). However, alternative methods for training the implicit 3D GAN 420 may exist so that a specific subset of the latent parameters controls the background (and / or hair). An exemplary alternative method is described below.

[0161] Overview of the architecture of implicit 3D GAN

[0162] Figure 4B An example architecture 400B of an implicit 3D GAN 420 according to some embodiments of the present disclosure is shown. The implicit 3D GAN 420 is configured to form an image generation model using, for example, a renderer mapping neural network 424, a deep implicit function neural network 106, an image renderer 428, a generator mapping neural network 440, and a high-resolution image generator 438.

[0163] In one example, the input to the implicit 3D GAN 420 is the latent code vector Z 422. For example, the implicit 3D GAN architecture is based on the StyleSDF architecture. In previous uses of implicit 3D GAN architectures, including StyleSDF, the latent code vector is an entangled representation of all attributes of the 3D object to be generated and its 2D image.

[0164] However, in embodiments of the present invention, the latent code vector Z 422 is the set of decoupled latent parameters 104 and 414 divided into subsets of latent parameters, where each subset represents a different attribute or subset of attributes. In some embodiments, the latent code vector Z 422 can be extracted from a Gaussian distribution (e.g., Gaussian distribution 304).

[0165] The renderer mapping neural network 424 transforms the decoupled latent code vector Z 422 into a joint representation W 426, which may be entangled but has a form usable by the deep latent function neural network 106. A differentiable image renderer 428 is used to render the output of the 3D object implicitly represented by the deep latent function neural network 106 based on the joint representation W 426. In some embodiments, the image renderer 428 outputs a low-resolution 2D image 430 of the object, and in some embodiments, it also outputs the corresponding 3D shape 432 of the object.

[0166] While a low-resolution image 430 of the object may be preferable, in some implementations a high-resolution image 434 of the object is desired. To achieve this, the image renderer 428 also outputs a low-resolution feature map F 436, which is then fed as input to the high-resolution image generator 438.

[0167] Using a deep hidden function neural network 106, an image renderer 428 generates a low-resolution image 430 and a low-resolution feature map 436 through volume rendering. In some implementations, this is accomplished as follows: Based on the values ​​of the elements of the joint representation W 426, the deep hidden function network 106 receives a 3D query point x and a viewing direction v, and maps them to an SDF value d(x,W), a radiance c(x,v,W), and a feature vector f(x,v,W). The image renderer 428 uses these outputs of the deep hidden function neural network to generate a low-resolution image C 430 and a corresponding low-resolution feature map F 436 through volume rendering. In one example, each pixel in the low-resolution image C 430 and the low-resolution feature map F 436 is computed by querying the point indexed by t along the ray r using r(t) = o + tp, where o is the virtual camera position from which the ray is emitted, and p is a vector corresponding to the image pixel position of the ray to be rendered. In this way, each pixel in C430 and the low-resolution feature map F436 is calculated using the following formula:

[0168] in, This represents the visibility of each point along the ray, and tn and tf represent the t values ​​corresponding to the nearest and farthest points along the ray that can be included in the rendered view, respectively. Here, according to... Density field obtained from SDF d(x) ,in, These are the parameters for learning.

[0169] In one example, the high-resolution image generator 438 is implemented as a neural network with a style-based GAN architecture (e.g., the StyleGAN2 architecture). A low-resolution feature map F 436 is passed to the high-resolution image generator 438. The style-based image generator 438 is configured to map the feature map F 436 to a high-resolution image I 434, conditioned on a style code g(W). The style code is generated by passing the joint representation W 426 to the generator mapping neural network g 440. The generator mapping neural network 440 transforms the joint representation W 426 into a form that can be used to modulate the style codes of the layers of the style-based GAN high-resolution image generator 438.

[0170] In one example, the high-resolution image generator 438 is configured to render a high-resolution 2D image 434 of a scene including objects. The 2D image 434 can be viewed from the input viewing direction v. In some cases, the image renderer 428 can also be configured to render a low-resolution 2D image 430 of the scene. One or both of images 430 and 434 can be rendered and stored for later use, for example, stored in a computer file on computer memory or analog or digital storage media, and / or rendered on a display device. The display device may be, for example, the screen of a computing device.

[0171] To enable the implicit 3D GAN 420 to take the decoupled latent parameters as its input Z 422, the implicit 3D GAN 420 is trained using a loss function that prompts or forces the input representation Z 422 to decouple according to the attributes of objects in the image. (See reference...) Figure 6A and Figure 6B Describe this decoupled training.

[0172] Overview of the architecture of an AI-based image processing system

[0173] Figure 5 An example block diagram 500 of an image processing system 102 according to some embodiments is shown. System 102 includes an input interface 502, a memory 504, a processor 506, and an output interface 508. The input interface 502 is configured to receive a latent parameter set 104. The latent parameter set 104 may define the representation of attribute values ​​of 3D objects in an image within a latent space. In one example, the latent parameter set 104 may include, but is not limited to, […]. , , , , and Therefore, the properties corresponding to the latent parameter set 104 include, but are not limited to, shape. albedo Lighting or illumination Background h, posture and other or remaining attributes For example, attribute values ​​can be generated by object encoder 402 or by different encoders or a collection of encoders.

[0174] For example, the latent parameter set 104 may also include a subset of latent parameters corresponding to hair. For example, when the 3D object is a human head, This can correspond to hair and / or background. Additionally, elements derived from... The additional latent parameter subset is represented as part of latent parameter set 104. Introducing the additional latent parameter subset. This enables the system to represent scene appearance variations not covered by other latent parameters. Specifically, the model (e.g., a deep hidden function neural network 106) can use variable latent parameters. This is used to assign any appearance factors that may not yet be interpreted by other latent parameters in the received latent parameter set 104. For example, such additional latent parameters may correspond to one or more attributes that have not been identified or encoded by the object encoder. For example, if Corresponding to hair, The representation of the scene background in an image can be encoded, either entirely or partially. This can be achieved by... Other attributes represented include fine appearance details, mirror lighting effects, and object shape or appearance details not encoded by encoder 402.

[0175] In one example, the latent parameter set 104 includes multiple subsets of latent parameters. It is understood that embodiments of this disclosure are intended to use decoupled latent code (interchangeably referred to as latent code vectors) for a 3D implicit model configured to generate a 3D object. In this regard, the multidimensional latent code of a scene is partitioned into a latent parameter set 104 representing multiple attributes of 3D objects in an image. The latent parameter set includes multiple subsets of latent parameters. For example, multiple subsets of latent parameters refer to those corresponding to attributes... , , The various latent parameters associated with h , , and Therefore, each subset of potential parameters (e.g.) ) corresponds to one of the multiple properties of a 3D object (e.g. ).

[0176] The memory 504 is configured to store a renderer mapping neural network 424, a deep hidden function neural network 106, an image renderer 428, a generator mapping neural network 440, and a high-resolution image generator 438.

[0177] In operation, the renderer mapping neural network 424 is trained to produce a joint encoding of the latent parameter set 104. For example, the joint encoding can represent the values ​​of the latent parameter set 104 used for processing in the intermediate latent space.

[0178] Furthermore, the deep implicit function neural network 106 is trained to generate implicit 3D models of 3D objects in an image. For example, the implicit 3D model can define a volume representation relative to the input viewing direction (e.g., input viewing direction 108) for each 3D location in an image scene including 3D objects. In one example, the volume representation is based on a signed distance field (SDF), a neural radiation field (NeRF), or a Gaussian splash. For example, the volume representation defines a signed distance value and / or optical density, as well as color and / or feature vectors, for each 3D location and viewing direction in the scene. In one example, the deep implicit function neural network 106 can utilize the signed distance function (SDF) to generate the volume representation. In this regard, the deep implicit function neural network 106 can receive a 3D query point x and an input viewing direction v as input. Furthermore, conditioned on a latent parameter set z, the deep implicit function neural network 106 outputs an SDF value d(x,z), a view-dependent color value c(x,v,z), and a feature vector f(x,v,z). The SDF value indicates the distance of the queried point from the surface boundary, and its sign indicates whether the point is inside or outside the surface. In another example, the deep hidden function neural network 106 can utilize NeRF to reconstruct a 3D scene as a combination of neural radiation fields and density fields using volumetric representation.

[0179] In one example, SDF is used to generate the volume representation, such that the volume representation is constrained by a signed distance value or color. In another example, NeRF is used to generate the volume representation, such that the volume representation is constrained by optical density or a feature vector.

[0180] Subsequently, the image renderer 428 uses an implicit 3D model generated by the deep implicit function neural network 106 to generate a new image or generate a modified image 110.

[0181] For example, the deep latent function neural network 106 is trained to employ a set of parameter consistency loss functions, thereby enforcing consistency between the values ​​414 of the latent parameter set 414 of the implicit 3D model used to generate the 3D object and the attribute values ​​or attribute values ​​of the 3D object extracted from the image by the object encoder 204 or 402. Furthermore, to generate images of 3D objects based on human heads, the deep latent function neural network 106 is also trained to employ a set of image-based consistency loss functions. The image-based consistency loss functions are employed to ensure consistency between the latent parameters 414 of the implicit 3D model used to generate the 3D object and the attribute values ​​or attribute values ​​of the 3D object extracted from the image by the object encoder 204 or 402. The goal is to minimize the impact of changes on the facial appearance of the human head. For example, potential parameters for facial appearance may include... , , and Alternatively, an image-based consistency loss is employed to minimize the impact of changes in face-specific latent parameters on the appearance of hair and background.

[0182] Furthermore, each latent parameter in the independent decoupled latent parameters is combined to produce a 2D output image 110. In some embodiments, an image renderer 428 is used to produce the output image 110. In other embodiments, a high-resolution image generator 438 is used to produce the output image 110.

[0183] Processor 506 is configured to submit data (e.g., values ​​414 of latent parameter set 104, input viewing direction 108, and in some cases, input images received from input interface 502) to deep hidden function neural network 106 for image generation and / or image editing or manipulation. Processor 506 is configured to manipulate decoupled latent parameters of physical properties using multiple subnetworks to modify the properties of a 3D model of a 3D object, thereby enabling the reconstruction of a 2D output image. The 2D output image is output via output interface 508.

[0184] Overview of Training in AI-Based Image Processing Systems

[0185] Figure 6A An example flowchart 600 of a method for identifying a latent parameter set 104 according to some embodiments is shown. In one example, an image processing system 102 is configured to generate or identify the latent parameter set 104 during a training phase, specifically a mapping or mapping relationship 202 between the latent parameter set and corresponding values ​​of attributes. In this case, the attribute values ​​of the training image 112 are used to generate values ​​for each subset of the latent parameter set 104.

[0186] At position 602, the 256-dimensional latent code Z is partitioned into disjoint subsets of the latent parameter set 104. In one example, 128 dimensions are assigned to latent parameters corresponding to the attributes of 3D objects in the scene, for example... , , and .

[0187] In addition, 64 dimensions were assigned to latent parameters corresponding to the appearance of the hair and background. And assign 64 dimensions to the latent parameters. This is to cover the interior of the face of a 3D object or a person's head and any other details around it.

[0188] In one example, to determine the dimension to assign to each attribute, the corresponding data covariance matrix is ​​analyzed. , and Performing eigenvalue decomposition, these data covariance matrices are obtained by using a shape... albedo and lighting The pre-trained encoder corresponding to the attribute is obtained by the object encoder 402 encoding the image 112 in the training dataset. Based on feature decomposition, for example... Assigned dimension And for Assigned dimension These dimensions cover well over 95% of the variance in their respective distributions. Furthermore, to enable complete explicit control over the 27 spherical harmonic lighting properties, for example, by increasing the dimensions... Assigned to .

[0189] At position 604, the attribute values ​​of the training image 112 are generated by the object encoder 402. Details of the object encoder's operation are as follows... Figure 4A and Figure 4B As described in the text.

[0190] At position 606, based on each attribute The values ​​(called attribute values) and the corresponding subsets assigned to the potential parameter set 104 (e.g.) , , and The dimension of the latent parameter set 104 is used to determine the value of each subset (called the latent parameter value). In one example, this is done by considering each attribute... The value was determined by performing PCA analysis. , , and The value of .

[0191] In one example, the eigenvectors of each matrix Z corresponding to these properties, along with the coefficients of those eigenvectors, are used to compute new values ​​for a subset of the latent parameter set 104. For this purpose, each subset of the latent parameters, i.e. , , , and It is computed separately from other subsets in the potential parameter set 104.

[0192] Subsequently, at point 608, the mapping between latent parameter values ​​and attribute values ​​is identified. In other words, system 102 is configured to utilize an attribute (e.g., ...) at different 3D locations of the 3D object in training image 112. The Z-matrix of the attributes is generated from the values ​​of ) , i.e. .based on and The value of determines the mapping between the two. For example, the Z matrix is ​​generated using methods for reducing the dimensionality of high-dimensional values ​​of attributes or representing them as low-dimensional values ​​of latent parameters in the latent space (e.g., using PCA).

[0193] In addition, in order to obtain Feature decomposition is used to create a mapping between attribute-encoded values ​​and corresponding latent codes from an image. Mapping for a single attribute... It is defined by equation (1).

[0194] To this end, the mapping between attribute values ​​and latent parameter values ​​for each image in training image 122 is identified or extracted. Subsequently, the mapping relationship between attributes and corresponding parameters is identified and stored as mapping relationships 202 and 112.

[0195] The following is combined Figure 6B Further details describe the training of the AI-based image processing system 102.

[0196] Figure 6B The training process of an implicit 3D GAN 420 trained using decoupled latent vectors, according to some implementations, is illustrated. In one example, the implicit 3D GAN architecture 420 includes a renderer mapping neural network 424, a deep latent function neural network 106, an image renderer 428, a generator mapping neural network 440, and a high-resolution image generator network 438 implemented as a style-based GAN. The entire system may be referred to as Controllable Latent StyleSDF (CoLa-SDF).

[0197] At 610, the numerical values ​​of the attributes of the training images 112 used for training are determined. For example, the attributes of each image are determined by processing each image in the set of training images 112 with object encoder 402 to generate an encoding (e.g., The numerical value of ). In some cases, the encoded value is represented in the latent space as the value of the latent parameter set.

[0198] Furthermore, at 612, an implicit 3D GAN 420 is trained based on attribute values ​​in the training images of the training objects. Specifically, the attribute values ​​or values ​​encoded in the latent space, i.e., the values ​​of the latent parameter set 104 of the training image 112, are used to train the deep implicit function neural network 106, the renderer mapping neural network 424, the generator mapping neural network 440, and the high-resolution image generator 438. Additionally, the deep implicit function neural network 106 is trained based on the training objects in the training image 112 to generate a 3D volumetric representation of the training objects based on the values ​​of the latent parameter set 104 of the training image 112.

[0199] In one example, by forcing the training of image 112's attributes (such as shape) albedo and lighting The implicit 3D GAN420 is trained by establishing a correspondence between the latent code of the training image and the parameters of a 3DMM model with built-in decoupled latent code. Furthermore, pose control is enforced using 3D volume rendering and view-dependent features of the implicit 3D GAN. Additionally, to ensure that some latent codes or parameters of the training image 112 correspond to hair and background, an image-based photometric consistency loss on the hair and background regions of the image ensures that different faces generated using the same hair and background codes have a consistent hair and background appearance.

[0200] In this way, an implicit 3D GAN 420 is trained using an object encoder 402, specifically encoding the attribute values ​​of the training images. For example, the object encoder 402 determines the numerical values ​​of attributes based on the training image 112. Based on the attribute values ​​of the training image 112 and their representation in the latent space as a latent parameter set 104, a mapping relationship between the attributes and the latent parameter set is determined. For example, this mapping indicates how the attribute values ​​are transformed into corresponding latent parameter values. Furthermore, the identified mapping is used to generate images during the inference phase.

[0201] In one example, an implicit 3D GAN model 420 based on the StyleSDF architecture is used to implement the image generation model because it has high rendering quality and explicitly models the 3D shape attribute values ​​of the training image 112 in the form of a signed distance field (SDF). In this way, embodiments of this disclosure can modify and enhance StyleSDF to achieve decoupled control of physical properties by modifying disjoint segments of its latent code.

[0202] Some implementations are based on the understanding that decoupling the latent space of the implicit 3D GAN model from the 3DMM requires the 3DMM encoder to be differentiable and have strong expressive power. Therefore, an encoder of a nonlinear 3DMM model (such as MOST-GAN) is used in the object encoder 402 to generate attribute codes or values ​​of the latent parameter set 104.

[0203] Training of the implicit 3D GAN (comprising various sub-networks that make up the AI-based image processing system 102) is performed based on loss functions that enforce consistency between the rendered images and the input data during the training phase. The following section combines... Figure 7 This will be described in detail.

[0204] Figure 7 An example flowchart 700 illustrates a training method for the AI ​​image processing system 102. During the training phase, a training image 112 is used, comprising a low-resolution image 702 and a high-resolution image 704. In one example, the low-resolution image 702 may have significantly fewer pixels than the high-resolution image. For example, the low-resolution image may be 64×64 pixels in size, and the high-resolution image may be 256×256 pixels in size. In one example, the training image 112 may include both the low-resolution image 702 and the high-resolution image 704. In another example, the low-resolution image 702 may be generated, for example, by downsampling from the high-resolution image 704.

[0205] In one example, the AI ​​image processing system 102 includes a deep hidden function neural network 106, a renderer mapping neural network 424, a high-resolution image generator 438, and a generator mapping neural network 440. For example, the deep hidden function neural network 104 is trained to generate a 3D volumetric representation of a 3D object based on joint encoding. The image renderer 428 (which does not require training in a preferred embodiment) can use the trained deep hidden function neural network 106 to render a low-resolution 2D image of the 3D object and a corresponding low-resolution feature map 436. Furthermore, the high-resolution image generator 438 is trained to render a corresponding high-resolution 2D image of the 3D object. For this purpose, the AI ​​image processing system 102 is trained in two stages.

[0206] In the first phase of training, the deep hidden function neural network 106 and the renderer mapping neural network 424 are trained, while the generator mapping neural network 440 and the high-resolution image generator 438 remain untrained and unused. At the end of the first phase of training, the weights of the deep hidden function neural network 106 and the renderer mapping neural network 424 are frozen.

[0207] In the second phase of training, the high-resolution image generator 438 and the generator mapping neural network 440 are trained. For example, the deep hidden function neural network 106 and the image renderer 428 include an SDF volume renderer, and the high-resolution image generator is based on a 2D style generative adversarial network (GAN). In addition, in addition to the StyleSDF-based loss, a consistency loss is introduced at each stage to enforce the relationship between the properties in the neural network and their latent parameters.

[0208] In the first phase of training, the deep hidden function neural network 106 and the renderer mapping neural network 424 are trained. In this case, low-resolution images 702 can be used for training. For example, the low-resolution discriminator 706 can attempt to distinguish between real data and data created by the generator. In other words, the low-resolution discriminator 706 attempts to distinguish between low-resolution images of real photographs belonging to the training set and low-resolution 2D images 430 of synthetic 3D objects, which are generated by the deep hidden function neural network 106 using random samples of the latent parameter set 104 and rendered by the image renderer 428. In some cases, the low-resolution discriminator 706 can also receive high-resolution images 704, convert them to low resolution, and then attempt to distinguish between these low-resolution real images and low-resolution images 430 of synthetic 3D objects, which are generated by the deep hidden function neural network 106 using random samples of the latent parameter set 104 and rendered by the image renderer 428.

[0209] Based on the distinction made by the low-resolution discriminator 706, a loss is introduced. In one example, the loss It may include nonsaturated GAN loss with R1 regularization Attitude alignment loss loss of process letter and minimum surface loss .

[0210] In one example, pose alignment loss Designed to ensure all generated objects are globally aligned. In addition to predicting whether an image is real or fake, the discriminator also attempts to predict the angles of the two input cameras (…). ,θ). Attitude alignment loss The volume renderer is used to generate the pose of the image. (θ) and the pose predicted by the low-resolution discriminator Smoothing L1 loss between the two. Loss It has two practically identical terms, one for θ and the other for For the sake of brevity, only the terms used for θ are shown here: .

[0211] In one example, the procedural loss Physical validity of forced signed distance fields: .

[0212] In one example, minimum surface loss Penalize SDF values ​​close to zero to avoid the formation of pseudo-zero crossovers and invisible surfaces: .

[0213] In the second phase of training, the high-resolution image generator 438 and the generator mapping neural network 440 are trained. In this case, the high-resolution discriminator 708 attempts to distinguish between real high-resolution images belonging to the training set and high-resolution images 434 of synthetic 3D objects, which are generated by passing random sampled values ​​of the latent parameter set 104 through the implicit 3D GAN pipeline 400B.

[0214] Based on the differentiation of the high-resolution discriminator 708, a loss is introduced. In one example, the loss Including path regularization loss and non-saturated GAN loss with R1 regularization For example, the training loss in the first phase. Second-stage training loss It can be limited to: (2) (3) in, , , and These are constant weights that can be set empirically to achieve an effective balance among the various losses. In one example, , , and .

[0215] Some implementations are based on the understanding that, in order to decouple attributes such as shape, albedo, and illumination, it is necessary to force the sampled values ​​of the latent parameter set 104 to be separated from the attribute values ​​(e.g., Parameter consistency between the encodings of facial parameters is required. Furthermore, to decouple the hair / background, facial parameters also need to be consistent. The values ​​are resampled and image-based consistency is enforced on the hair and background, then... The values ​​are resampled and image-based consistency is enforced on the facial region.

[0216] Except for the StyleSDF loss in the first phase of training StyleSDF loss in the second phase of training In addition, some implementations include encoder-based parameter consistency loss and image-based hair / background consistency loss in both the first and second training phases. These consistency losses promote the decoupling of object or facial attributes in the model. In the first training phase, the consistency loss is applied to low-resolution images. In the second training phase, the consistency loss is applied to high-resolution images. The following combines... Figure 8 Explain the consistency loss function and the aforementioned losses. and The combined use of .

[0217] Figure 8 An example flowchart 800 of one iteration of training system 102 using a consistency loss function according to some implementations is shown. This training flowchart applies to phases 1 and 2 of the training.

[0218] In each iteration of training, for example, a set of values ​​for the latent parameter 104 is sampled randomly 802. These values ​​are then passed through the implicit 3D GAN 400B to generate an image I 830 of a 3D object. The image represented by I 830 in the figure depends on the training phase the system is undergoing. In the first phase of training, image I 830 represents a low-resolution image 430 output by image renderer 428, and the aforementioned loss 820 used for this image is... However, in the second phase of training, image I 830 represents the high-resolution image 434 output by the high-resolution image generator 438, and the aforementioned loss 820 used for this image is... .

[0219] Using the mapping relationship between the sampled latent parameter set 104 determined in 608 and the corresponding attribute value 804 (e.g., equation (1)), the attribute value 804 corresponding to the latent parameter value 104 can be determined. To cause the generated image I 830 to present the corresponding visual attribute, the attribute encoder 406 from the object encoder 402 is applied to the generated image I 830 to determine the attribute value 408 presented in the image, i.e., the image attribute 810. For example, the shape encoded from image I 830 can be... albedo and lighting The values ​​810 are written as follows: , and .

[0220] Parameter consistency loss function 822 is used during training to promote consistency between the generated image I 830 (as described by its encoded image attribute 810) and the values ​​of the latent parameter set 104 sampled 802 for generating that image. Parameter consistency loss. 822 is limited to: (4) Among them, the shape consistency loss function Limited to:

[0221] For example, Force the shape encoding of the generated image I 830 produced by the object encoder 402 The sampled shape attribute value 804 obtained according to equation (1) same.

[0222] Constrain the albedo consistency loss function in a similar manner Lighting consistency loss function .

[0223] In addition, consistency loss 822 includes the attitude consistency loss function. To force modification of non-attitude latent parameters (e.g.) The value of ) should not change the pose of the generated object. This is enforced by comparing the pose encoding of I with the pose encoding of the image generated during the iterative resampling phase, as described below. In each iteration, the pose consistency loss function Given by one of the following two:

[0224] or

[0225] The generated image depends on the iterative resampling stage. 832 or 834.

[0226] In equation (4), , , and These are constant weights that can be chosen empirically to produce good results, for example, , , =60 and .

[0227] It should be noted that when the 3D object is a human head, the image of the human head can include facial and hair regions. However, existing 3DMM-based methods do not model hair and background. One non-linear 3DMM method, MOST-GAN, does include a hair encoder. However, the MOST-GAN hair encoder is not fully three-dimensional (3D), which may make it less suitable for defining hair properties in a 3D system. However, an attribute encoder is not necessary to learn decoupled latent codes for controlling specific attributes. In some implementations, semantic segmentation of the image can be used to achieve decoupling of one or more attributes. In some implementations, latent parameters of hair and background are decoupled from other latent parameters in this way.

[0228] According to embodiments of this disclosure, in order to decouple hair / background from other physical properties, a subset of latent parameters for hair / background is used. This method is only applicable to modeling hair and background. In other implementations, hair and background attributes can be divided into two subsets of parameters, one for hair and the other for background. In implementations where the object is not a human head, a similarly trained subset of latent parameters can be learned to be used only for the background, or for object attributes that are not available to the encoder.

[0229] To this end, some implementations utilize semantic segmentation algorithms that can be used to divide an image into semantically meaningful regions. For example, a face parser refers to a semantic segmentation algorithm capable of segmenting an image containing a face into semantically meaningful regions, including face regions, hair regions, and background regions. The output of such a face parser may include binary masks Mface, Mhair, and Mbackground, where Mface indicates pixels located in the face region of the image, Mhair indicates pixels located in the hair region of the image, and Mbackground indicates pixels located in the background region of the image (i.e., regions that contain neither a face nor hair). Furthermore, these binary masks can be combined to indicate combined regions. For example, mask Mhairbg = Mhair. Mbackground indicates pixels in an image that belong to the hair or the background. For example, the notation Mhairbg(I) is a binary mask indicating pixels in image I that belong to the hair or the background. In other implementations, M may be a real-valued mask instead of a binary mask.

[0230] In each iteration, an iterative resampling phase can be performed after the initial sampling 802 of the latent parameters and the generation of the corresponding image I 830. In this resampling phase, some latent parameters are resampled, and an image consistency loss ensures that attributes not resampled remain the same in the image generated from these resampled parameter values. Which latent parameters are resampled in each iteration can be selected alternately. In some implementations, in even-numbered iterations, [the following is a partial translation of the first part, but the full context is missing]. , and Resampling is performed 812 times, while in odd-numbered iterations, for Resample 814.

[0231] In one example, when the 3D object is a human head and it's necessary to decouple facial attributes from hair / background attributes, image-based consistency loss functions Lh and Lface are used. In this regard, the implicit 3D GAN 400B is trained based on two sets of image-based consistency loss functions: the facial consistency loss Lface828, which minimizes the correlation between facial appearance and the latent parameters Zh involving the background and hair of the human head; and the hair and background consistency loss Lh824, which minimizes the correlation between the appearance of the background and hair of the human head and the latent parameters involving facial appearance. , and Minimize the correlation between them.

[0232] In another example, when 3D objects in the scene may have backgrounds not encoded by encoder 402, a similar set of image-based consistency loss functions is used. In this case, Lh is used as the background consistency loss, and Lface is used as the object consistency loss.

[0233] In even-numbered iterations, for , and Perform resampling 812, then use the included , and The latent code of the new values ​​is used to generate a new image Ih 832. Since the resampled latent code is expected to control facial attributes but not hair and background, the regenerated image Ih 832 is used, through a hair / background consistency loss function. 824 is used to force hair / background consistency.

[0234] In the resampling phase of odd-numbered iterations, face-related latent parameters (e.g.) , and The value of ) remains unchanged, and only for the hair / background latent parameters Perform resampling 814. Then, use the included... The latent code of the new value is used to generate a new image Iface 834. Due to the expectation The facial region is not affected, so the regenerated image Iface 834 is used, and the facial consistency loss function is applied. 828 is used to force facial consistency.

[0235] In one example, image-based consistency loss, specifically the hair / background consistency loss function. and facial consistency loss function Limited to: (5) (6) in, It is the union of the hair / background masks from the two generated images I830 and Ih832, and It is the union of the facial masks from the two generated images I 830 and Iface 834. In this regard, a pre-trained face parser 710 is used to parse the rendered facial image into a segmentation mask for the face and a segmentation mask for the hair and background. Furthermore, the masking photometric loss function... Limited to:

[0236] in, There are two images, m is a binary mask, and ⊙ is the element-wise product operator. Similarly, masking-perceptual loss... Limited to:

[0237] in, There are two images, m is a binary mask, ⊙ is the element-wise multiplication operator, and the function... This represents a feature map extracted from one or more hidden layers of an image processing neural network (such as a VGG network).

[0238] Therefore, the overall loss for the first phase of training is given by a weighted combination of losses 820, 822, 824, and 828: (7) Similarly, the loss and loss function for the second phase of training are calculated. The overall loss for the second phase of training is given by the following formula: (8) in, and These are weights that indicate the relative weights of the loss function and can also be used to indicate in which iterations each loss is used. For example, for equations (7) and (8), in even-numbered iterations... However, in the odd number of iterations, it equals 0, and in the odd number of iterations... =5, but =0 in even iterations.

[0239] In one example, in order to obtain meaningful encodings from the object encoder 402 and meaningful facial parsings from the face parser 710, the generated images from the AI-based image processing system 102 need to resemble 3D objects such as faces. Therefore, each stage uses only StyleSDF loss. and Training is performed for initialization, for example, 3000 to 7000 iterations, after which... , and .

[0240] For example, during training, the model or network in the AI-based image processing system 102 is trained on a dataset. For instance, to train system 102 to generate human heads, this dataset could be the Flickr-Faces-HQ (FFHQ) dataset, which includes high-quality images of human faces. For example, the FFHQ dataset could include a large number (e.g., over 50,000) of high-resolution images of human faces with different ages, ethnicities, and image conditions. After training, the model or network in the AI-based image processing system 102 is evaluated during the testing phase.

[0241] Some implementations are based on the understanding that the photorealism provided by the StyleSDF architecture is maintained by employing a non-linear 3DMM solely through loss functions. During training of the StyleSDF architecture or the deep latent function neural network 106, control is enforced through a set of parameter consistency loss functions that utilize the image decoder of the object encoder 402 for reverse rendering, thereby employing the decoupled latent parameters or properties of the object encoder 402.

[0242] For example, facial generation evaluation of the model involves generating a 2D image of a 3D object based on provided and / or random attribute values, while subsequent editing evaluation involves modifying a rendered image based on the input image or input attribute values. To numerically evaluate the generation quality, the ability of system 102 to generate photorealistic images is compared with existing methods in terms of Fraser Initial Distance (FID). Furthermore, to evaluate image editing, the ability of system 102 to generate decoupled latent parameters for shape, albedo, pose, lighting, and hair / background in latent space and to explicitly edit these properties is demonstrated and evaluated.

[0243] Once system 102, specifically the network or model within system 102, has been trained, tested, and evaluated, system 102 can be deployed for synthetic image generation and / or image or synthetic editing applications.

[0244] Overview of Example Implementations of Artificial Intelligence (AI) Image Processing Systems

[0245] Figure 9 An example block diagram 900 of an AI-based image processing system 102 according to some exemplary embodiments is shown. The purpose of system 102 is to combine the controllability of nonlinear 3DMM with the photorealism of implicit 3D GAN.

[0246] According to this example, a nonlinear 3DMM (i.e., MOST-GAN) encoder is combined with an architecture of an SDF-based implicit 3D GAN model (i.e., StyleSDF) to form system 102. Furthermore, the implicit 3D GAN model in system 102 includes, for example... Figure 5 The components described herein include a deep implicit function neural network 106, an image renderer 428, and a high-resolution image generator 438. The deep implicit function neural network 106 and the image renderer 428 include a volume renderer based on a signed distance function (SDF), and the high-resolution image generator 438 is based on a 2D-style generative adversarial network (GAN). In one example, the deep implicit function neural network 106 is configured to generate an implicit 3D model of the volumetric representation of a 3D object. Furthermore, the image renderer 428 is configured to render a 2D image of the 3D object based on the implicit 3D model generated by the deep implicit function neural network 106.

[0247] System 102 also includes a renderer mapping neural network 424 and a generator mapping neural network 440, which are configured to generate encodings of potential parameters.

[0248] During operation, in the inference phase, the system receives a latent parameter set 104. In one example, the latent parameter set can be received or obtained from a database or the object encoder 402. For example, the object encoder 402 can generate the latent parameter set 104 for different attributes of 3D objects in different images. For this purpose, the mapping relationship between attributes and latent parameter sets is known.

[0249] Furthermore, the received set of latent parameters 104 can define the shape of the 3D object in the reference image. albedo ,illumination ,attitude And a representation of at least two attributes in the background h. In one example, the latent parameter set 104 includes multiple subsets of latent parameters depicted as 906A, 906B, 906C, 906D, and 906E. For example, each subset of latent parameters corresponds to an attribute of an object in the image. For example, latent parameter subsets Corresponding to shape attributes latent parameter subset Corresponding to albedo attribute and the subset of latent parameters Corresponding to lighting or illumination attributes In this way, multiple subsets of potential parameters can exist that correspond to other attributes such as pose, hair, and background.

[0250] Subsequently, the renderer mapping neural network 424 is configured to generate or produce a joint encoding W426 of the latent parameter set. In one example, the joint encoding W426 is a joint encoding of latent parameters 104, which may be entangled but may be better suited for processing by an implicit 3D GAN. In one example, the joint encoding W is an intermediate latent space used to represent the learned attribute values. In one example, the renderer mapping neural network 424 is configured to generate a reparameterized latent code to a new code that better suits the characteristics desired by the generator mapping neural network 440. In one example, the renderer mapping neural network 424 is a nonlinear transformation network that allows decoupled representations of latent parameters to be mapped to entangled representations corresponding to the StyleSDF representation. The joint encoding is generated based on the mapping from the decoupled representation to the entangled representation of StyleSDF.

[0251] In one implementation, the renderer mapping neural network 424 comprises a 3-layer MLP with LeakyReLU activations and is configured to map the input latent code Z into the space of W, and then generate frequency modulations for each layer of the deep latent function neural network 106. and phase shift .

[0252] Subsequently, the joint encoding W of the latent parameter set is fed into the deep hidden function neural network 106. In one example, the deep hidden function neural network 106 also receives the 3D query point x and the input viewing direction v as input and maps them into SDF values. Radiance c and eigenvectors Based on the mapping relationships identified during training, the process maps the input data (i.e., query point, viewing direction, and the set of latent code parameters corresponding to different attributes) to feature vectors.

[0253] In one example, based on joint encoding, a deep implicit function neural network 106 can generate an implicit three-dimensional (3D) model of an object. For example, the implicit 3D model can characterize the entire embedding space on the object by classifying each 3D location as located inside, outside, or on the surface of the object. In one example, the volume representation 908 is defined by a signed distance value and / or optical density, as well as color and / or feature vectors.

[0254] In one example, the image renderer 428 can use a deep hidden function neural network 106 and input data such as joint encoding and input viewing direction 108 to generate a low-resolution image C 430 and a low-resolution feature map F436 through volume rendering. For example, the low-resolution image is Image. In one example, each pixel of image C is computed by querying 3D points along a ray r emanating from camera position o and passing through the pixel position corresponding to vector p, where the ray is... It is expressed as follows: (9) Furthermore, each pixel of the feature map is constrained as follows: (10) in, This indicates the visibility of each point along the ray. According to... From SDF Obtain density field , where α is the learned property, i.e. the shape property used to generate the density field of the 3D shape 916.

[0255] Furthermore, the volume representation 908 is transmitted to the high-resolution image generator 438. This style-based generative adversarial network (GAN) 438 is configured to, in accordance with style codes... Under certain conditions, the low-resolution feature map F 436 is mapped to a high-resolution image I 434. Style codes are generated by passing the joint representation W 426 to the generator mapping network g 440. The role of the generator mapping neural network 440 is to transform the joint representation W 426 into style codes that can be used as modulation of the layers of the style-based GAN high-resolution image generator 438. The generator mapping neural network 440 generates style codes and feeds them into the image generator 438. In one example, each component may have an accompanying generator mapping neural network 440 to map the input latent vector into a modulation signal for each layer.

[0256] Therefore, an SDF-based loss function is used. To train a deep hidden function neural network 106 and a renderer mapping neural network 424, and to use the loss function alone. To train the stylized image generator 438 and the generator mapping neural network.

[0257] Overview of Modifying Rendered Images Using AI-Based Image Processing Systems

[0258] Figure 10A An example flowchart 1000 is shown for a method of modifying an image according to some embodiments.

[0259] In this regard, at 1002, the expected value of the attribute to be modified in an image of the object or 3D object is received. In one example, the expected value of the attribute can be received as the expected numerical value, letter value, or alphanumeric value of the attribute. In another example, an expected image can be received, from which the expected value of the attribute can be determined. In some cases, the expected value of the attribute can correspond to a combination of attributes, such as shape and pose, or hair and shape.

[0260] At position 1004, the stored mapping is used to determine the expected values ​​of the modified latent parameters based on the expected values ​​of the attributes. For example, the stored mapping could indicate the relationship between an attribute and its latent parameters in the latent space. This mapping could be stored in the system's memory. Specifically, the mapping could indicate the relationship between the values ​​of the latent parameter set 104 and the corresponding values ​​of the object's attributes. For this purpose, based on an attribute of the image to be modified (e.g., shape)... ), can be based on objects and The mapping between stored values ​​is used to modify their underlying parameters, i.e. .

[0261] In one example, the mappings between corresponding attributes and their latent parameters of different 3D objects can have different properties. Therefore, during training, the mappings between attribute values ​​and latent parameter values ​​in multiple training images can be analyzed to extract these mappings.

[0262] Subsequently, at point 1006, a modified image of the object is generated based on the expected values ​​of the modified latent parameters. For example, the modified latent parameters for that attribute, along with previously used or unmodified latent parameters for the remaining attributes, or the remaining latent parameters in latent parameter set 104, are used to generate the modified image. Since the change in the image involves only one attribute, the other attributes of the image are unaffected or minimally affected by the change.

[0263] Figure 10B Another example flowchart 1010 is shown, illustrating a method for modifying an image according to some embodiments.

[0264] In this regard, at position 1012, the probability distribution of the latent parameters corresponding to the attribute is sampled. For example, if the attribute needs to be modified to produce a modified image, then the attribute values ​​of objects in the currently rendered image are sampled.

[0265] Furthermore, at point 1014, the latent parameters corresponding to the object's properties in the latent parameter set are modified. In one example, the probability distribution of the latent parameter values ​​for that property is modified to change the latent parameter values ​​based on the expected values.

[0266] Subsequently, at position 1016, a modified image of the object is generated based on the combination of the modified latent parameters and the remaining unmodified latent parameters in the latent parameter set.

[0267] While this example discloses the use of System 102 to edit or modify an image, in other examples, System 102 can be used to generate new images, such as synthetic images. In this regard, System 102, the deep hidden function neural network 106, or the object encoder 402 can limit random values ​​of attributes or latent parameters based on the type of object to be generated. New images are then generated based on these random values.

[0268] In some other cases, the input image can be fed to system 102 where it may be necessary to mimic certain properties of the input image in the new image. In this case, the object encoder 402 can generate attribute codes for the input image. These attribute codes or values ​​can be retrieved based on the attributes to be used and used to generate corresponding latent parameters. For example, if the shape of the input image is to be mimicked, the shape attribute values ​​of the objects in the input image can be preserved. This preserved value can be used to generate latent parameters for the shape. Furthermore, random values ​​are used for other attributes such as pose, lighting, hair / background, etc. A new image is generated based on predefined values ​​of the latent parameters for the shape and random values ​​of the latent parameters for other attributes.

[0269] Once a new image is generated, it can be gradually controlled in one or more iterations to modify one or more different attributes.

[0270] Overview of the effects of generating or modifying images using AI-based image processing systems

[0271] Figure 11A A set of exemplary AI-modified images 1110 according to some implementation methods are shown.

[0272] The controllable latent StyleSDF neural network disclosed in this disclosure combines the controllability of physical properties provided by 3DMM-based methods with the high-quality generation capabilities of implicit 3D GANs. The set of images 1100 can be manipulated independently in terms of different shapes, expressions, albedo, lighting conditions, hairstyles, and backgrounds.

[0273] For example, multiple modified images 1102 and 1104 with different facial shapes are shown. Starting with the latent parameters used to generate the initial generated image 1101, the latent parameters of the image's shape are modified at different values. This is used to generate a modified image 1102. For example, the 3D shape in modified image 1102 is modified so that the object (or face) image appears to belong to a different person. On the other hand, the shape parameters of image 1101 are modified at different values. To obtain a modified image 1104, so that the object (or face) appears to have different facial expressions.

[0274] Furthermore, by changing the illumination parameters of image 1101 at different values... To generate a modified image 1106, and by changing the albedo parameter of image 1101 at different values. This is used to generate the modified image 1108. Furthermore, the underlying parameters for hair and background in image 1101 are changed at different values. To generate and modify image 1110.

[0275] Figure 11B A set of exemplary AI-generated images 1112 are shown according to some implementation methods.

[0276] This set of images 1112 includes images from multiple viewpoints. For example, from viewpoint... Figure 1 Image 1112 is synthesized using a method that starts from the maximum azimuth angle. The maximum arc and elevation angle is Two randomly generated faces, 1114A and 1114B, are rendered from different viewpoints with varying curvature. To demonstrate the quality of the underlying 3D surfaces, moving cube meshes 1116A and 1116B obtained from a signed distance field are shown. Furthermore, for each example, the latent code of the shape is... Mapped to property values ​​generated by the object encoder And this mapping is based on equation (1). In this way, the MOST-GAN decoder is used to generate shape attribute values. Corresponding 3D meshes 1118A and 1118B. For this purpose, the multi-view images of faces 1114A and 1114B each use latent codes of shape. The corresponding values ​​are fixed values ​​for other object parameters, but are generated by changing the values ​​of the pose parameters.

[0277] Figure 11C A set of exemplary AI-modified images 1120 according to some implementation methods is shown. According to this example, lighting properties are edited using spherical harmonic functions. In this paper, for three randomly generated faces (described as faces 1122A, 1122B, and 1122C), the lighting is altered or changed by directly modifying the spherical harmonic coefficients.

[0278] The level of global (ambient) illumination is controlled in image 1124 by changing the first spherical harmonic coefficient. Furthermore, the horizontal directionality of the illumination is controlled in image 1126 by changing the second spherical harmonic coefficient.

[0279] Figure 12A , Figure 12B , Figure 12C and Figure 12D This illustrates AI-modified images based on attributes from a source image, according to various implementation methods. Figure 12A In, shape latent parameters are used. The shape from source image 1202A Transmit to target image 1204A.

[0280] Reference Figure 12B Using albedo potential parameters Albedo from source image 1202B Transmitted to target image 1204B. Additionally, in Figure 12C In the middle, the albedo potential parameter is used. Light or illumination from source image 1202C Transmitted to target image 1204C. Figure 12D In the middle, using background latent parameters The hair or background h from source image 1202D is transmitted to target image 1204D. To this end, physical properties are transmitted from source images 1202A, 1202B, 1202C, and 1202D to target images 1204A, 1204B, 1204C, and 1204D, indicating that the latent code or latent parameters of the neural network are decoupled.

[0281] Exemplary Implementation

[0282] Figure 13 A block diagram 1300 of a computer and AI-based image processing system 102 (hereinafter referred to as the System) for generating an image 110 of a scene according to an exemplary embodiment is shown. The System 102 includes multiple interfaces for connecting the System 102 to other systems and devices. The System 102 includes an input interface 1302 configured to accept attributes of 3D objects and a potential parameter set 104 corresponding to the generation of the image.

[0283] In some embodiments, system 102 includes a network interface controller (NIC) 1306 configured to obtain a potential parameter set 104 via a network 1308, which may be one or a combination of a wired network and a wireless network. The NIC 1306 is adapted to connect system 102 to the network 908 via a bus 910, which connects system 102 to input device 1304. Input device 1304 may correspond to a database or user equipment storing the potential parameter set 104 or image 404 to be provided to system 102 to generate and render a 3D volumetric representation as an output image.

[0284] Additionally or alternatively, system 102 may include a human-machine interface (HMI) 1312. The HMI 1312 within system 102 connects system 102 to a keyboard 1314 and a pointing device 1316, wherein the pointing device 1316 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc.

[0285] System 102 includes a processor 1318 configured to execute stored instructions 1320 and a memory 1322 storing instructions executable by the processor 1318. The processor 1318 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 1322 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 1318 may be connected to one or more input and output devices via a bus 1310.

[0286] According to some implementations, instruction 1320 can implement a method for generating and rendering an image of a scene including objects viewed from one or more viewing directions. For this purpose, computer memory 1322 stores a renderer-mapped neural network 424, a deep implicit function neural network 106, an image renderer 428, a generator-mapped neural network 440, and a high-resolution image generator 438.

[0287] A renderer mapping neural network 424 is configured to generate a joint encoding 426 based on a latent parameter set 104 representing at least two attributes 810 of objects in a defined image 830. A deep hidden function neural network 106 and an image renderer 428 are configured to generate one or more low-resolution images 430, low-resolution feature maps 436, and one or more low-resolution signed distance function (SDF) surfaces based on the input viewing direction v and the joint encoding W 426 of the latent parameter set 104. Furthermore, a generator mapping neural network 440 is configured to convert the joint representation W 426 into style codes that modulate the high-resolution image generator 438. The high-resolution image generator 438 is configured to render a high-resolution 2D image 434 of a scene including objects viewed from the input viewing direction.

[0288] In some implementations, output interface 1324 may be configured to render output on display device 1326, namely, a high-resolution 2D image 434 of a scene including objects viewed from the input viewing direction. Examples of display devices 1326 include computer monitors, televisions, projectors, or mobile devices. System 102 may also be connected to application interface 1328, which is adapted to connect system 102 to external device 1330 to perform various tasks.

[0289] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the above description of exemplary embodiments will provide those skilled in the art with a sufficient description for implementing one or more exemplary embodiments. It is contemplated that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the subject matter set forth in the appended claims.

[0290] Specific details have been set forth in the foregoing description to provide a thorough understanding of these embodiments. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other cases, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. Furthermore, the same reference numerals and designations in the various figures denote the same elements.

[0291] Furthermore, various implementations can be described as processes depicted using flowcharts, schematic diagrams, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process may terminate upon completion of its operations, but may have additional steps not discussed or included in the accompanying drawings. Moreover, not all operations in any specifically described process will occur in all implementations. A process can correspond to a method, function, program, subroutine, subroutines, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or the main function.

[0292] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. Manual or automatic implementation can be performed, or at least assisted in by using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. One or more processors can perform the necessary tasks.

[0293] The various methods or processes outlined herein can be encoded as software capable of executing on one or more processors employing any of a variety of operating systems or platforms. Furthermore, such software can be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, in various implementations, the functionality of program modules can be combined or distributed as needed.

[0294] The embodiments of this disclosure can be implemented as methods, an example of which has been provided herein. The actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those shown, which may include performing some actions simultaneously, although in the illustrative embodiments these actions are shown as sequential actions. Furthermore, the use of ordinal terms such as “first,” “second,” etc., to modify claim elements in the claims does not indicate any priority, order, or sequence of one claim element relative to another, nor does it indicate the temporal order of performing method actions; rather, it serves only as a label to distinguish one claim element having a certain name from another element having the same name (but using ordinal terms), thereby differentiating these claim elements.

[0295] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications may be made within the spirit and scope of this disclosure. Therefore, the appended claims are intended to cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. An artificial intelligence (AI) image processing system for generating and rendering images of a scene including objects viewed from one or more viewing directions, the AI ​​image processing system comprising at least one processor and a memory storing instructions that cause the at least one processor of the AI ​​image processing system to: Receive a latent parameter set, which defines a representation of at least two attributes of the object in the image: shape, albedo, illumination, pose, and background, wherein, The latent parameter set includes multiple subsets of latent parameters, wherein each subset of latent parameters corresponds to one of the at least two attributes of the object in the image; The latent parameter set is submitted to a renderer mapping neural network, which is trained to generate joint encodings of the latent parameter set. The joint encoding is submitted to a deep implicit function neural network, which is trained to generate an implicit 3D model of the object given the joint encoding. This implicit 3D model defines a volume representation relative to the input viewing direction for each 3D location in the scene including the object, wherein the volume representation defines one or a combination of a signed distance value and optical density, and one or a combination of color and feature vectors; and The volumetric representations generated for different 3D locations in the scene are submitted to an image renderer configured to render a 2D image of the scene including the objects viewed from the input viewing direction.

2. The AI ​​image processing system according to claim 1, wherein, In order to modify the properties of the object in the 2D image of the scene, the processor is configured to: Modify the latent parameters in the latent parameter set that correspond to the attribute of the object; as well as A modified image of the object is generated based on a combination of the modified latent parameters and the remaining unmodified latent parameters in the latent parameter set.

3. The AI ​​image processing system according to claim 2, wherein, In order to modify the attribute of the object, the processor is configured to: The probability distribution of the latent parameters corresponding to the attribute of the object is sampled to generate the modified latent parameters.

4. The AI ​​image processing system according to claim 2, wherein, The memory stores a mapping between the values ​​of the potential parameter set and the corresponding values ​​of the object's attributes, and wherein, in order to modify the object's attributes, the processor is configured to: Receive the desired value of the attribute to be modified in the image of the object; and The stored mapping is used to determine the expected value of the modified potential parameter corresponding to the expected value of the attribute.

5. The AI ​​image processing system according to claim 4, wherein, The mapping is based on statistical analysis of the image distribution in the dataset.

6. The AI ​​image processing system according to claim 5, wherein, The mapping is derived using principal component analysis (PCA) of the attribute values ​​from the training image dataset of the object.

7. The AI ​​image processing system according to claim 2, wherein, The processor is configured to: The modified latent parameters are obtained by using an object encoder to match the parameters of the input image, wherein the object encoder is configured to determine the numerical values ​​of corresponding attributes in the input image.

8. The AI ​​image processing system according to claim 2, wherein, The processor is configured to: The deep hidden function neural network is trained using an object encoder, which determines the values ​​of attributes based on training images of training objects.

9. The AI ​​image processing system according to claim 8, wherein, The processor is configured to: The object encoder is used to train the deep hidden function neural network based on a set of parameter consistency loss functions to enforce consistency between the received latent parameter set of the image used to generate the scene and the parameters that encode the attributes of the rendered 2D image of the scene.

10. The AI ​​image processing system according to claim 8, wherein, The processor is configured to: The object encoder is used to train the deep hidden function neural network based on a set of image-based consistency loss functions to minimize the correlation between latent parameters related to the background of the training object and the appearance of the training object, and to minimize the correlation between latent parameters related to the appearance of the training object and the appearance of the background of the training object.

11. The AI ​​image processing system according to claim 8, wherein, The object in question is a human head.

12. The AI ​​image processing system according to claim 11, wherein, The processor is configured to: The object encoder is used to train the deep hidden function neural network based on a set of image-based consistency loss functions to minimize the correlation between latent parameters or the set of latent parameters related to the hair and background of the person's head and the facial appearance, and to minimize the correlation between latent parameters or the set of latent parameters related to other attributes of the person's head and the appearance of the hair and background of the person's head.

13. The AI ​​image processing system according to claim 11, wherein, The processor is configured to: The multidimensional latent code of the scene is divided into a set of latent parameters that define representations of at least two attributes of the objects in the image; and Additional latent parameters are introduced as part of the set of latent parameters, which define a variable representation of the facial appearance factors of the person's head.

14. The AI ​​image processing system according to claim 1, wherein, The processor is configured to execute a module of the AI ​​image processing system, the module comprising: The renderer mapping neural network is used to generate the joint encoding based on the latent parameter set that defines the representations of at least two attributes of the object in the image; The deep implicit neural network and the image renderer are configured to generate one or more of the following based on the joint encoding of the input viewing direction and the latent parameter set: a low-resolution image, a low-resolution feature map, and one or more low-resolution signed distance function (SDF) surfaces. A generator-mapped neural network, configured to convert the joint representation into style codes for a modulated high-resolution image generator; and The high-resolution image generator is configured to render a high-resolution 2D image of the scene including the object viewed from the input viewing direction.

15. The AI ​​image processing system according to claim 14, wherein, The high-resolution image generator is configured to render a high-resolution 2D image of the scene, including the objects viewed from the input viewing direction, instead of the 2D image of the scene rendered by the image renderer or other than the 2D image.

16. The AI ​​image processing system according to claim 1, wherein, The shape attribute represents the three-dimensional 3D shape of the object, and the pose attribute represents the pose of the object in 3D space.

17. The AI ​​image processing system according to claim 1, wherein, Lighting properties are represented using spherical harmonics.

18. The AI ​​image processing system according to claim 1, wherein, The deep hidden function neural network is one of the following: a signed distance function (SDF) neural network, a neural radiation field (NeRF) neural network, or a Gaussian splash neural network.

19. A method for generating and rendering an image of a scene including objects viewed from one or more viewing directions, the method comprising: Receive a set of latent parameters, the set of latent parameters defining a representation of at least two attributes of the object in the image, including shape, albedo, illumination, pose, and background, wherein the set of latent parameters includes a plurality of subsets of latent parameters, wherein each subset of latent parameters corresponds to one of the at least two attributes of the object in the image; The latent parameter set is submitted to a renderer mapping neural network, which is trained to generate joint encodings of the latent parameter set. The joint encoding is submitted to a deep implicit function neural network, which is trained to generate an implicit 3D model of the object given the joint encoding. This implicit 3D model defines a volume representation relative to the input viewing direction for each 3D location in the scene including the object, wherein the volume representation defines one or a combination of a signed distance value and optical density, and one or a combination of color and feature vectors; and The volumetric representations generated for different 3D locations in the scene are submitted to an image renderer configured to render a 2D image of the scene, including the objects viewed from the input viewing direction, on a display device.

20. A non-transitory computer-readable storage medium having a program executed by a processor to perform a method, the method comprising: Receive a latent parameter set, the latent parameter set defining a representation of at least two attributes of an object in an image, including shape, albedo, illumination, pose, and background, wherein the latent parameter set includes multiple subsets of latent parameters, wherein each subset of latent parameters corresponds to one of the at least two attributes of the object in the image; The latent parameter set is submitted to a renderer mapping neural network, which is trained to generate joint encodings of the latent parameter set. The joint encoding is submitted to a deep implicit function neural network, which is trained to generate an implicit 3D model of the object given the joint encoding. This implicit 3D model defines a volume representation relative to the input viewing direction for each 3D location in the scene including the object, wherein the volume representation defines one or a combination of a signed distance value and optical density, and one or a combination of color and feature vectors; and The volumetric representations generated for different 3D locations in the scene are submitted to an image renderer configured to render a 2D image of the scene, including the objects viewed from the input viewing direction, on a display device.

Images