Method and system for generating image of person

Abstract

Camera parameters describing the perspective and gesture parameters describing the shape and gesture of the parameterized human body model are processed to generate geometry information characterizing the 3D geometry of the person and appearance information characterizing the RGB appearance of the human body. These information is then processed to generate an image of the person. In the image, the person is depicted as viewed from a perspective, and wherein the body of the person has a shape and posture described by posture parameters.

Description

[0001] Cross-references

[0002] This application claims priority to Singapore application number 10202250421B filed on July 8, 2022, entitled “Method and system for generating images of persons”, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present application relates to methods and systems for generating images of people using neural networks. Background Art

[0004] Virtual humans (avatars) with full control over their poses and appearance are used in various applications such as immersive photographic visualization, virtual try-on, VR / AR, and creative image editing. Known solutions for creating images of virtual humans rely on classical graphics modeling and rendering techniques. While they can provide high-quality images, they usually require pre-captured templates, multi-camera systems, controlled studios, and long-term work by artists.

[0005] Neural networks offer the advantage of low-cost image synthesis. However, known approaches for 3D-aware image synthesis using neural networks are either limited to rigid object modeling or learning a clear person representation for a single subject. The former limits the quality and controllability of more challenging person generation, while the latter is not generative and thus cannot synthesize new identities and appearances. Summary of the invention

[0006] The present invention aims to provide a novel and useful method for generating images of the human body, in particular images showing the human body in a desired posture. That is, the image includes at least a portion of the human torso and at least a portion of one or more (usually all) of the human limbs.

[0007] Broadly speaking, the present invention proposes a generator neural network suitable for synthesizing images of a person, wherein the shape and pose of the person depicted in the image is controlled by external inputs received by the generator neural network. One way to achieve this is to have the generator neural network generate an intermediate 3D representation of the person in a predefined pose. In order to generate a feature image of the person in a desired pose, the intermediate 3D representation is sampled at multiple spatial points to obtain feature data, and the obtained feature data is assigned to spatial points in the feature image according to a mapping that takes into account the desired pose and a predetermined pose (i.e., a single fixed pose used during network training; for example, the predetermined pose may be a pose with all limbs of a person extending straight out from the person's torso). A decoder generates a final image from the feature image.

[0008] In one aspect, the present invention proposes processing camera parameters describing the viewpoint and posture parameters describing the shape and posture of a parameterized human model to generate geometry information (which characterizes the 3D geometry of the person) and appearance information (which characterizes the RGB appearance of the person). This information is then processed to generate an image of the person. In the image, a person is depicted observing the person from the viewpoint, and the person's body has the shape and posture described by the posture parameters.

[0009] In an implementation, the camera parameters are used to generate a representation of a first 3D space containing a person in a predetermined pose. The representation is sampled at one or more index positions (typically multiple index positions) obtained based on the camera parameters and the pose parameters. The representation can be conveniently obtained based on a latent vector of, for example, random numbers. In a computationally efficient form, the representation can be a three-plane representation based on three feature planes.

[0010] An index position may be created by selecting a spatial position in a first image of a parameterized human body model arranged in a pose described by the pose parameters (a "first spatial position") and transforming the first spatial position to a corresponding second spatial position in a second image showing the parameterized body model in a predetermined pose. A neural network that performs such a mapping based on the pose parameters may be readily obtained using known techniques.

[0011] Then, the index position can be obtained based on the second spatial position. Optionally, the second spatial position can be deformed by a deformation network model to generate the index position.

[0012] The appearance information at each index position may be processed (eg, by another adaptive unit such as a multilayer perceptron) to generate appearance information.

[0013] In principle, the geometric shape information can be obtained in the same manner using another adaptive unit. However, more preferably, the geometric shape information is obtained by generating 3D mesh data of a parameterized human body model having a shape described by a posture parameter arranged in a predefined posture; sampling the 3D mesh data of the parameterized human body model at an index position to obtain a first distance value; obtaining a second distance value using the first distance value and the sample of the representation at the index position; and providing a signed distance value obtained by modifying the first distance value using a signed distance value as the geometric shape information. It has been found that using signed distance values ​​can achieve superior learned geometry.

[0014] A second aspect of the invention relates to a method of training the above neural network. This can be achieved by treating it as a generator neural network and training it jointly with a discriminator neural network (i.e., continuous updating of the generator network is interleaved or performed simultaneously with continuous updating of the discriminator neural network). The discriminator neural network is updated to enhance its ability to distinguish between images generated by the generator neural network (fake images) and images from a training database. The images in the training database may be images captured by one or more corresponding cameras.

[0015] The present invention may be expressed as a method, or as a system comprising one or more computers and one or more storage devices storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the method. It may also be expressed as a computer program product, such as downloadable program instructions (software) or one or more non-transitory computer storage media storing instructions. When executed by one or more computers, the program instructions cause the one or more computers to perform the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Embodiments of the present invention will now be described by way of example only with reference to the following drawings, in which:

[0017] Figure 1 is a schematic diagram of an example generator neural network as an embodiment of the present invention;

[0018] Figure 2 It is used for training Figure 1 A schematic diagram of an example training system for a generator neural network of FIG.

[0019] Figure 3 is a flow chart of an example process for generating an image of a person;

[0020] Figure 4 It is used for training Figure 1 A flowchart of an example process of a generator neural network of ; and

[0021] Figure 5 shows that it can be used to perform Figure 4 and 5 Methods and implementations Figure 1 The generator neural network and Figure 2 A computer system for training systems. DETAILED DESCRIPTION

[0022] Figure 1 An example generator neural network 10 is shown for generating images of people. Generator neural network 10 may be suitable for generating images of people with various appearance styles and clothing in arbitrary poses. Generator neural network 10 is also suitable for generating images of animated human avatars.

[0023] The generator neural network 10 is configured to receive camera parameters describing a viewing angle. As explained below, the generator neural network 10 is configured so that the camera parameters control the viewing angle used in drawing an image of a person. In other words, the image generated by the generator neural network 10 depicts the person as viewed from that viewing angle.

[0024] The generator neural network 10 is also configured to receive posture parameters describing the shape and posture of the parameterized human body model. When set according to the posture parameters, the parameterized human body model represents a 3D human body model having a body shape according to the shape described by the posture parameters and being in a body posture according to the posture described by the posture parameters. The shape described by the posture parameters can, for example, indicate the height and / or obesity level (e.g., body mass index) of the parameterized human body model. The posture described by the posture parameters can control the way the parameterized human body model stands and can be a natural human posture. An example parameterized human body model (referred to as an SPML model) is described in more detail in "SMPL: a skinned multi-person linear model" by Matthew Loper et al. in ACM Trans.On Graphics in 2015. The posture parameters can be parameters of the SPML model. That is, the posture parameters can be SPML parameters, or include SPML parameters.

[0025] As explained below, the generator neural network 10 is further configured so that the posture parameters control the body shape and posture of the person depicted in the generated image. In other words, the image generated by the generator neural network 10 depicts a person in a posture described by the posture parameters.

[0026] The generator neural network 10 is also configured to receive a first random number array, referred to as a canonical code or a first latent vector. Alternatively, the generator neural network 10 can be configured to generate the canonical code instead of receiving it, for example using a random number generator unit of the generator neural network 10.

[0027] The generator neural network 10 is also configured to receive a second array of random numbers, referred to as a geometry code or a second latent vector. Alternatively, the generator neural network 10 may be configured to generate a geometry code instead of receiving it, for example using a random number generator.

[0028] The generator neural network 10 includes a canonical mapping module 11 configured to receive camera parameters and a canonical code and generate a first conditional feature vector. The canonical mapping module can be implemented by using a multi-layer perceptron (such as an 8-layer multi-layer perceptron).

[0029] The generator neural network 10 further includes a geometry mapping module 13, which is configured to receive the posture parameters and the geometry code and generate a second conditional feature vector. The geometry mapping module 13 can be implemented using another layer of perceptrons, such as a multi-layer perceptron with 8 layers.

[0030] The generator neural network 10 includes an encoder 12. The encoder is configured to receive the first conditional feature vector and generate a "3D representation" that is a representation of a first 3D space (or "canonical space") that includes a person in a predetermined pose (e.g., a pose in which all of the person's limbs are extended straight from the person's torso). That is, each spatial point in the space is associated with a set of features that indicate a person in a predetermined pose.

[0031] Any spatial point in the 3D space ("index position") may be sampled using a sampler unit 16. The sampler unit 16 receives the index position and samples the 3D representation to retrieve a feature data set associated with the index position. The 3D representation may be an explicit representation, such as based on a voxel grid. Alternatively, the 3D representation may be an implicit representation that represents the first 3D space as a continuous function.

[0032] Preferably, the 3D representation is an explicit-implicit hybrid representation, such as the three-plane representation described in detail by Eric R. Chan et al. in “Efficient Geometry-aware 3D Generative Adversarial Networks” (arxiv.org, arXiv:2112.07945, 2012).

[0033] In one example, the encoder 12 is configured to generate three feature planes as a 3D representation of the first 3D space. Each feature plane is an NxNxC array, where N is the spatial resolution and C is the number of channels. N can be 256 and C can be 32. The three feature planes can be considered as axis-aligned orthogonal planes. Feature data of any 3D point can be obtained by searching on the three planes. In other words, any 3D position in the first 3D space can be sampled by projecting any 3D position in the first 3D space onto each of the three feature planes, retrieving the corresponding feature vector by bilinear interpolation, and aggregating the three features by summing.

[0034] In other words, the three-plane representation is based on three feature planes spanned by coordinates (xy), (yz), and (zx), respectively. Each feature plane is an NxN array, and each pixel of each array is associated with a corresponding feature data set (C eigenvalues). Given an index position (x, y, z), the sampler 16 samples the three-plane representation by interpolating between the C eigenvalues ​​of each pixel adjacent to the position (x, y) on the first plane to obtain C eigenvalues; interpolating between the C eigenvalues ​​of each pixel adjacent to the position (y, z) on the second plane to obtain C eigenvalues; and interpolating between the C eigenvalues ​​of each pixel adjacent to the position (z, x) on the third plane to obtain C eigenvalues. The three sets of C eigenvalues ​​are then aggregated.

[0035] Note that in an implementation, the encoder 12 and sampler unit 16 may be implemented as a single unit that receives a first condition vector and a data set specifying an index position and generates a feature data set associated with the corresponding spatial point at the index position (i.e., without generating feature data for other points).

[0036] The generator neural network 10 includes a transformation module 14 configured to receive camera parameters and pose parameters. For each of a plurality of values ​​of the integer index i, the transformation module 14 is configured to generate a first plurality of coordinates Pi indicating the positions of corresponding spatial points in a first image of a parameterized human model arranged in a pose described by the pose parameters and viewed from a perspective described by the camera parameters. The space depicted by the first image is referred to as the "observation space".

[0037] The transformation module 14 is further configured to apply a mapping transformation to the first plurality of coordinates P corresponding to each spatial point. i , to generate the corresponding second plurality of coordinates P i '. The second plurality of coordinates corresponding to each spatial point indicates the position of the corresponding second spatial point in the second image of the parameterized human body model in the predetermined posture. The mapping transformation is based on the posture described by the posture parameters and the predetermined posture. In other words, the mapping transformation maps the spatial point in the observation space to the corresponding second spatial point in the canonical space.

[0038] In one example, the parameterized human body model is an SPML model, and the pose parameters are given by the vector p=(θ,β), where θ and β are SPML pose and shape parameters, respectively. In this example, the first image depicts an SPML human body arranged according to the vector p=(θ,β) and viewed from the perspective described by the camera parameters.

[0039] In order to convert any point P in the first plurality of coordinates i Mapped to the corresponding point P in the canonical space i', the SMPL model can be used to guide the transformation performed by the transformation module 14. SMPL defines a vertex-based skinned human model (V, W), where v∈V is a vertex and w∈W is a skinned weight assigned to the vertex, so that the ∑ j w j =1,w j ≥ 0. Therefore, the inverse skinned transform can be used to map a SMPL mesh in observation space with an SPML pose θ into a canonical space with a predetermined pose:

[0040] T SMPL (v,w,θ)=∑ j w j (R j v+t j ) where R j and t j is the rotation and translation at each joint j derived from the SMPL model with the SPML pose θ. The predefined pose can be expressed as

[0041] Where R j and t j is the rotation and translation at each joint j derived from the SMPL model with the SPML pose θ. The predefined pose can be expressed as

[0042] Those skilled in the art will appreciate that this formulation can be extended to any spatial point in the observation space by taking the same transformation from the nearest point on the SMPL grid surface. Formally, for each spatial point P i , the closest point v* on the SMPL grid surface is v* = arg min v∈V (||Pi-v||). Pi is then transformed to Pi' in the canonical space using the corresponding skinned weight w* as follows:

[0043] P′ i =T o (P i |θ)=T SMPL (v,w * ,θ).

[0044] The generator neural network 10 includes a deformation network 15, which is configured to receive a spatial point Pi' in the canonical space from the transformation module 14 and a second conditional feature vector from the geometric shape mapping module 13. The deformation network 15 is also configured to process the spatial point Pi' and the second conditional feature vector to generate a spatial point P in the canonical space. i 'Deformation ΔP i . Deformation ΔP i It can be expressed as ΔP i =TΔ (P′ i |g,β), where g is the second conditional feature vector. In one example, the deformation network 15 performs a fine-grained geometric transformation from the observation space to the canonical space and compensates for the inaccuracy of the mapping transformation applied by the transformation module 14. In one example, the deformation network 15 compensates for the inaccuracy of the inverse skinning transformation. In one example, the deformation network 15 provides pose-dependent deformations to improve the modeling of non-rigid dynamics. In one example, the deformation ΔP provided by the deformation network 15 i The quality of depicting cloth wrinkles in the final image is improved. Note that in some embodiments, the geometry mapping module 13 and the deformation network 15 can be omitted, which means that the geometry code is not required. However, this will result in the generated image being less realistic.

[0045] Another multi-layer perceptron can be used to implement the transformation module 14. In this case, the spatial point P i 'Processed by another multi-layer perceptron to generate an embedding space point P i ', which is then concatenated with the second conditional feature vector g and the SMPL shape β and input into this other multilayer perceptron:

[0046] ΔP i =MLPs(Concat(Embed(P′ i ),g,β).

[0047] In general, the combination of the transformation module 14 and the deformation network 15 aims to provide a final mapping transformation T from the observation space to the canonical space o→c (P i ), which can be expressed as:

[0048] T o→c (P i ) = P′ i +ΔP i =(P i |θ)+T Δ (P′ i |g,β).

[0049] In other words, the deformation network 15 modifies the second plurality of coordinates generated by the transformation module 14. The modified second plurality of coordinates (including the mapped spatial point P′) i +ΔP i ) is called an index position. Each value of i has an index position.

[0050] As described above, the generator neural network 10 includes a sampler unit 16 configured to sample the 3D representation at each index position to receive feature data f i(also called “feature value”). The sampler unit samples the feature data f i The predictor module 17 is configured to process the received feature data f i To generate geometry information and appearance information for each index position.

[0051] The predictor module 17 may include a first multilayer perceptron configured to process the sampled feature data for each index position to generate corresponding appearance information. The appearance information may describe a color associated with the corresponding index position. In an example where the number of channels of the feature planes included in the 3D representation is C, the appearance information generated by the first multilayer perceptron for each index position is a vector containing C values. For example, the number of channels of the feature planes may be 32, and the appearance information generated by the first multilayer perceptron for each index position may be a vector containing 32 values.

[0052] The predictor module 17 may include a second multi-layer perceptron configured to process the sampled feature data for each index position to generate corresponding geometric shape information. The geometric shape information may be a signed distance value or a density value associated with the corresponding index position.

[0053] In the example where the geometric shape information is a signed distance value, the predictor module 17 is further configured to generate 3D mesh data of a parameterized human body model arranged in a predefined pose and having a shape described by the pose parameters. The mesh can be represented as M=SMPL(θ can ,β). In this case, the predictor module 17 is also configured to calculate at each index position P′ i +ΔP i Sample the 3D mesh data of the parameterized human body model to obtain the corresponding coarse signed distance values The predictor module 17 is further configured to, for each index position, sample feature data f from the 3D representation i and the corresponding coarse signed distance value Connect and feed into the second multilayer perceptron to generate the residual signed distance value Δd i The predictor module 17 is further configured to provide a residual signed distance value Δd by using i Modify the coarse signed distance value The signed distance value d is obtained i As geometric shape information. For example, In this example, providing signed distance values ​​rather than density values ​​can introduce direct geometry adjustments and guidance into the generator neural network 10.

[0054] The generator neural network 10 includes a volume rendering network 18, which is configured to process the geometric shape information and the appearance information to generate a feature image and an RGB image. The volume rendering network also receives the pose parameters. Both the feature image and the RGB image depict a person viewed from a perspective described by the camera parameters, and wherein the human body has a shape and pose described by the pose parameters. In one example, the volume rendering network 18 is configured to convert the signed distance value di of each point Pi along the ray r provided by the predictor module 17 as the geometric shape information into a density value σ i ,Right now Where α>0 is a learnable parameter that controls the tightness of the density around the surface boundary. By integrating along the ray r, the corresponding pixel features can be obtained as follows

[0055]

[0056] where δ i =‖P i -P i-1 ‖, N is the number of spatial points in the observation space, c i To describe the o→c (P i ) = P′ i +ΔP i . Feature data f sampled from the 3D representation i The volume rendering network 18 is further configured to integrate the plurality of rays to generate a feature image. The feature image may have a channel depth and resolution of 32x128x128. The RGB image may have a channel depth and resolution of 3x128x128.

[0057] The generator neural network 10 includes a decoder 19, which is configured to process the feature image and the RGB image to generate an image of a person. The decoder 19 can also be configured to also process the first conditional feature vector generated by the canonical mapping module 11. In one example, the decoder 19 is configured to generate an image of a person with a resolution higher than the resolution of the RGB image. An example architecture of the decoder 19 is described in detail in "Analyzing and Improving the Image Quality of StyleGAN" by Tero Karras et al. (CVPR, 2020).

[0058] The generator neural network 10 can be trained as part of a generative adversarial network comprising the generator neural network 10 and a discriminator. Figure 2 An example training system 20 is shown for training the generator neural network 10. To avoid unnecessary repetition, like reference numerals will be used to denote like features.

[0059] The training system 20 includes a generator neural network 10, a discriminator neural network 22, and a parameter update system 24. The generator neural network 10 refers to Figure 1 Description.

[0060] The training system 20 is configured to receive as input camera parameters, pose parameters, specification code, and geometry code.

[0061] The discriminator neural network 22 is configured to receive a training image of a person generated by the generator neural network 10 and generate a prediction of whether the training image of the person is an image of a real person or an image of a fake person (i.e., an image generated by the generator neural network 22). In one example, the discriminator neural network 22 is configured to receive an image generated by the decoder 19 and an RGB image generated by the volume rendering network 18. In this example, the consistency of the generated multi-view images can be improved. Additionally or alternatively, the discriminator neural network 22 can also be configured to receive camera and pose parameters. In this example, the image generated by the trained generator neural network may show improved consistency with the camera and pose parameters.

[0062] The parameter update system 24 is configured to process the prediction generated by the discriminator neural network 22 and modify one or more network parameters of the generator neural network 10 and the discriminator neural network 22 based on the prediction. By modifying one or more network parameters, the parameter update system 24 trains i) the generator neural network 10 to generate photo-realistic images of people, and ii) the discriminator neural network 22 to reliably distinguish between images of real people and fake people. For example, when the discriminator neural network 22 cannot recognize that the generated training image it receives from the generator neural network 10 is an image of a fake person, the parameter update system 24 updates the discriminator neural network 22, and when the discriminator neural network 22 recognizes that the generated training image it receives from the generator neural network 10 is an image of a fake person, the parameter update system 24 updates the generator neural network 10.

[0063] The discriminator neural network 22 can be further trained using a training dataset containing labeled images of real people. This process is performed by updating steps interleaved with the above-mentioned updates. Specifically, when the discriminator neural network 22 cannot recognize that the generated training images it receives from the training dataset are images of real people, the parameter update system 24 updates the discriminator neural network 22.

[0064] The parameter updating system 24 may be configured to be based on the L Reg The non-saturated GAN loss L GANIn the example where the geometry information is provided as a signed distance value, the parameter update system 24 may be further configured to modify the one or more network parameters based on an eikonal loss defined as:

[0065]

[0066] Additionally, in this example, the parameter update system 24 may be further configured to modify one or more network parameters also based on the minimum surface loss to encourage the generator neural network 10 to represent human geometry with a minimal amount of zero crossings:

[0067]

[0068] In this example, the parameter updating system 24 can be further configured to also be based on the loss L SMPL Modify one or more network parameters so that the generated surface is close to a given SMPL mesh and consistent with a given SMPL pose. To this end, the vertex v∈V is subjected to random noise δ on the SMPL mesh. v Perturbation of , and sampling the 3D representation at the corresponding points:

[0069] L SMPL =∑ v∈V ‖MLP d (F(v+v v ))‖.

[0070] In this example, the overall loss can be expressed as:

[0071] L total =L GAN +λ Reg L Reg +λ Eik L Eik +λ Minsurf L Minsurf +λ SMPL L SMPL ,

[0072] where λ * is the corresponding loss weight.

[0073] refer to Figure 3 , explains the use of reference Figure 1 A method 30 of generating images of persons using a generator neural network 10 is described.

[0074] In step 31, camera parameters describing the viewing angle are received. The camera parameters control the viewing angle used when drawing an image of a person. Also in step 31, the generator neural network 10 receives or generates a canonical code. Further in step 31, a canonical mapping module receives the camera parameters and the canonical code and generates a first conditional feature vector.

[0075] In step 32, posture parameters describing the shape and posture of the parameterized human body model are received. The posture parameters control the body shape and body posture of the person depicted in the generated image. Also in step 32, the generator neural network 10 receives or generates the geometric shape code. Further in step 32, the geometric shape mapping module receives the posture parameters and the geometric shape code and generates a second conditional feature vector.

[0076] In step 33, the camera parameters and the pose parameters are processed to generate appearance information. To this end, the encoder receives the first conditional feature vector and generates a 3D representation of a first 3D space (or "canonical space") including a person in a predetermined pose. In one example, the encoder 12 generates three feature planes as the 3D representation of the first 3D space in step 33.

[0077] Also in step 33, the transformation module 14 generates a first plurality of coordinates indicating N spatial points in a first image of the parametric human body model arranged in a pose described by the pose parameters and viewed from a perspective described by the camera parameters. The transformation module 14 further applies a mapping transformation to the first plurality of coordinates to generate a second plurality of coordinates indicating corresponding second spatial points in a second image of the parametric human body model in a predetermined pose, wherein the mapping transformation is based on the pose described by the pose parameters and the predetermined pose.

[0078] When the parameterized human body model is an SPML model and the posture parameters are given by the vector p=(θ, β), the transformation module 14 applies the mapping transformation P′ i =T o (P i |θ)=T SMPL (v,w * ,θ) to map each spatial point Pi in the first plurality of coordinates to a corresponding point P in the second plurality of coordinates i '.

[0079] Further in step 33, the deformation network 15 processes the spatial point Pi' and the second conditional feature vector to generate a deformation ΔP of the spatial point Pi' in the canonical space i In fact, the transformation module 14 and the deformation network 15 transform T according to the final mapping o→c (P i ) = P′ i +ΔP iMaps the first plurality of coordinates in observation space together to index locations in canonical space.

[0080] Still in step 33, the predictor module 17 samples the 3D representation at each index position to receive feature data f i Then, the predictor module 17 generates appearance information by processing the received feature data. The predictor module 17 may include a first multilayer perceptron that processes each set of sampled feature data f i To generate corresponding appearance information. The appearance information may describe the color associated with the corresponding index position.

[0081] In step 34, the predictor module 17 generates geometric shape information by processing the received feature data. The predictor module 17 may include a second multilayer perceptron that processes each sampled feature data to generate corresponding geometric shape information. The geometric shape information may be a signed distance value. In this case, the predictor module 17 generates 3D mesh data of a parameterized human body model arranged in a predefined posture and having a shape described by the posture parameters. In addition, in this case, the predictor module 17 then samples the 3D mesh data of the parameterized human body model at each index position to obtain a coarse signed distance value. The predictor module 17 connects the feature data sampled from the 3D representation with the corresponding coarse signed distance value for each index position and feeds it to the second multilayer perceptron to generate a residual signed distance value. Then, the predictor module 17 provides the signed distance value obtained by modifying the coarse signed distance value using the residual signed distance value as the geometric shape information.

[0082] In step 35, the volume rendering network 18 processes the geometry information, appearance information and pose parameters to generate a feature image and an RGB image. Both the feature image and the RGB image depict a human body viewed from a perspective described by the camera parameters, and the human body has a shape and pose described by the pose parameters.

[0083] Further in step 35, the decoder 19 processes the feature image and the RGB image to generate an image of the person. In one example, the decoder 19 generates the image of the person at a resolution higher than the resolution of the RGB image.

[0084] refer to Figure 4 , explains the reference Figure 2 A method 40 for training a generator neural network 10 and a discriminator neural network 22 is described. In general, training images are generated by the generator neural network 10 and then classified as images of real people or images of fake people by the discriminator neural network 22. Depending on whether the classification is correct or incorrect, one or more network parameters of the generator neural network 10 or the discriminator neural network 22 are modified.

[0085] In step 41, with reference Figure 3 Similar to step 31 described above, camera parameters describing the viewing angle are received. The camera parameters control the viewing angle when drawing the training image of the person. Also in step 41, the generator neural network 10 receives or generates the canonical code. Further in step 41, the canonical mapping module receives the camera parameters and the canonical code and generates a first conditional feature vector.

[0086] In step 42, with reference Figure 3 Similar to step 32 described above, posture parameters describing the shape and posture of the parameterized human body model are received. The posture parameters control the body shape and body posture of the person depicted in the generated training image. Also in step 42, the generator neural network 10 receives or generates the geometric shape code. Further in step 42, the geometric shape mapping module receives the posture parameters and the geometric shape code and generates a second conditional feature vector.

[0087] In step 43, with reference Figure 3 Similar to step 33 described above, camera parameters and pose parameters are processed to generate appearance information. To this end, the encoder receives the first conditional feature vector and generates a 3D representation of a first 3D space (or "canonical space") including a person in a predetermined pose. In one example, the encoder 12 generates three feature planes as a 3D representation of the first 3D space in step 43.

[0088] Also in step 43, the transformation module 14 generates a first plurality of coordinates indicating N spatial points in a first image of the parametric human body model arranged in a pose described by the pose parameters and viewed from a perspective described by the camera parameters. The transformation module 14 further applies a mapping transformation to the first plurality of coordinates to generate a second plurality of coordinates indicating corresponding second spatial points in a second image of the parametric human body model in a predetermined pose, wherein the mapping transformation is based on the pose described by the pose parameters and the predetermined pose.

[0089] When the parameterized human body model is an SPML model and the posture parameters are given by the vector p=(θ, β), the transformation module 14 applies the mapping transformation P′ i =T o (P i |θ)=T SMPL (v,w * ,θ) each spatial point P in the first plurality of coordinates i The corresponding point P′ mapped to the second plurality of coordinates i .

[0090] Further in step 43, the deformation network 15 processes the spatial point P i ' and the second conditional eigenvector to generate a spatial point P in the canonical space i 'Deformation ΔPi In fact, the transformation module 14 and the deformation network 15 transform T according to the final mapping o→c (P i ) = P′ i +ΔP i Maps the first plurality of coordinates in observation space together to index locations in canonical space.

[0091] Still in step 43, the predictor module 17 samples the 3D representation at each index position to receive feature data f i Then, the predictor module 17 generates appearance information by processing the received feature data. The predictor module 17 may include a first multilayer perceptron that processes each set of sampled feature data f i To generate corresponding appearance information. The appearance information may describe the color associated with the corresponding index position.

[0092] In step 44, with reference Figure 3 Similar to step 34 described above, the predictor module 17 generates geometric shape information by processing the received feature data. The predictor module 17 may include a second multilayer perceptron that processes each sampled feature data to generate corresponding geometric shape information. The geometric shape information may be a signed distance value. In this case, the predictor module 17 generates 3D mesh data of a parameterized human body model having a shape described by the posture parameters and set in a predefined posture. In addition, in this case, the predictor module 17 then samples the 3D mesh data of the parameterized human body model at each index position to obtain a coarse signed distance value. The predictor module 17 connects the feature data sampled from the 3D representation with the corresponding coarse signed distance value for each index position and feeds it into the second multilayer perceptron to generate a residual signed distance value. The predictor module 17 then provides the signed distance value obtained by modifying the coarse signed distance value using the residual signed distance value as the geometric shape information.

[0093] In step 45, with reference Figure 3 Similar to step 35 described above, the volume rendering network 18 processes the geometry information, appearance information and pose parameters to generate a feature image and an RGB image. Both the feature image and the RGB image depict a person from the perspective described by the camera parameters, and the person's body has a shape and pose described by the pose parameters.

[0094] Further in step 45, the decoder 19 processes the feature image and the RGB image to generate a training image of a person. In one example, the decoder 19 generates a training image of a person having a resolution higher than the resolution of the RGB image.

[0095] In step 46, the discriminator neural network 22 receives the training image of the person generated by the generator neural network 10, and generates a prediction of whether the training image of the person is a real person image or a fake person image (i.e., an image generated by the generator neural network 22). In one example, the discriminator neural network 22 receives the image generated by the decoder 19 and the RGB image generated by the volume rendering network 18. In this example, the consistency of the generated multi-view image can be improved. Additionally or alternatively, the discriminator neural network 22 can also receive camera and pose parameters. In this example, the image generated by the trained generator neural network can show that its consistency with the camera and pose parameters is improved.

[0096] In step 47, the parameter update system 24 processes the prediction generated by the discriminator neural network 22 and modifies one or more network parameters of the generator neural network 10 and the discriminator neural network 22 based on the prediction. By modifying the one or more network parameters, the parameter update system 24 trains i) the generator neural network 10 to generate realistic images of people, and ii) the discriminator neural network 22 to reliably distinguish between images of real people and fake people. For example, when the discriminator neural network 22 cannot recognize that the generated training image it receives from the generator neural network 10 is an image of a fake person, the parameter update system 24 updates the discriminator neural network 22, and when the discriminator neural network 22 recognizes that the generated training image it receives from the generator neural network 10 is an image of a fake person, the parameter update system 24 updates the generator neural network 10.

[0097] The parameter updating system 24 can be based on L with R1 regularization Reg The non-saturated GAN loss L GAN Modify one or more network parameters. In the example where the geometry information is provided as a signed distance value, the parameter update system 24 may further modify one or more network parameters based on the Icona loss. In addition, in this example, the parameter update system 24 may further modify one or more network parameters based on the minimum surface loss to encourage the generator neural network 10 to represent the human geometry with the minimum amount of zero crossings. Further in this example, the parameter update system 24 may also modify one or more network parameters based on the loss L SMPL One or more network parameters are further modified so that the generated surface is close to a given SMPL mesh and consistent with a given SMPL pose. In this example, the overall loss can be L expressed as above total .

[0098] Figure 5 is a block diagram showing a technical architecture 500 of a server that can execute Figure 3 or Figure 4The technology architecture includes a processor 522 (which may be referred to as a central processor unit or CPU) that communicates with memory devices including secondary storage 524 (e.g., a disk drive), a read-only memory (ROM) 526, and a random access memory (RAM) 528. The processor 522 may be implemented as one or more CPU chips. The technology architecture may also include input / output (I / O) devices 530 and network connection devices 532.

[0099] Secondary storage 524 is typically comprised of one or more disk drives or tape drives and is used for non-volatile data storage and as an overflow data storage device if RAM 528 is insufficient to hold all working data. Secondary storage 524 may be used to store programs that are loaded into RAM 528 when such programs are selected for execution.

[0100] In this embodiment, the secondary storage device 524 has a sequential processing component 524a, which includes non-transitory instructions operated by the processor 522 to perform various operations of the disclosed method. ROM 526 is used to store instructions and possible data read during program execution. Secondary storage device 524, RAM 528 and / or ROM 526 may be referred to as computer-readable storage media and / or non-transitory computer-readable media in some cases.

[0101] I / O devices 530 may include a printer, video monitor, liquid crystal display (LCD), plasma display, touch screen display, keyboard, keypad, switches, dials, mouse, trackball, voice recognizer, card reader, paper tape reader, or other well-known input devices.

[0102] The processor 522 executes instructions, codes, computer programs, scripts, which may be accessed from a hard disk, floppy disk, optical disk (these various disk-based systems may be considered secondary storage 524), flash drive, ROM 526, RAM 528, or network connection device 532. Although only one processor 522 is shown, multiple processors may be present. Therefore, when instructions may be discussed as being executed by one processor, the instructions may still be executed by one or more processors simultaneously, successively, or in other ways.

[0103] Although the technical architecture is described with reference to computers, it should be understood that the technical architecture can be formed by two or more computers that communicate with each other and collaborate to perform tasks. For example, but not by way of limitation, the application can be partitioned in a manner that allows concurrent and / or parallel processing of application instructions. Alternatively, the data processed by the application can be partitioned in a manner that allows two or more computers to process different parts of the data set concurrently and / or in parallel. In one embodiment, the technical architecture 500 can use virtualization software to provide the functionality of multiple servers that are not directly related to the number of computers in the technical architecture 500. In one embodiment, the above-disclosed functionality can be provided by executing an application and / or multiple applications in a cloud computing environment. Cloud computing can include using dynamically scalable computing resources to provide computing services over a network connection. A cloud computing environment can be established by an enterprise and / or rented from a third-party provider as needed.

[0104] By programming and / or loading executable instructions onto the technical architecture, at least one of the CPU 522, RAM 528, and ROM 526 is changed, partially transforming the technical architecture into a special-purpose machine or device having the new functions taught by the present disclosure. It is of vital importance to the fields of electrical engineering and software engineering that the functions that can be realized by loading executable software into a computer can be converted into hardware implementations through well-known design rules.

[0105] Although the foregoing description has described exemplary embodiments, those skilled in the art will appreciate that many variations are possible within the scope and spirit of the invention.

Claims

1. A method for generating an image of a person using a neural network, the method comprising: Receive camera parameters describing the viewing angle; receiving pose parameters describing a shape and a pose of a parameterized human body model; Processing the camera parameters and the pose parameters to generate geometry information and appearance information; as well as The geometry information and the appearance information are processed to generate the image of the person, the image depicting the person as viewed from the perspective and wherein the body of the person has the shape and the pose described by the pose parameters.

2. The method of claim 1 , wherein processing the camera parameters and the pose parameters to generate geometry information and appearance information comprises: processing the camera parameters to generate a representation of a first 3D space including the person in a predetermined pose; Based on the camera parameters and the pose parameters, obtaining one or more index positions; as well as The geometry information and the appearance information are generated from the representation by sampling the representation at the one or more index locations.

3. The method of claim 2, wherein processing the camera parameters to generate a representation of a first 3D space including a person in a predetermined pose comprises: receiving a first latent vector comprising a random number; converting the camera parameters and the first latent vector into a first conditional feature vector; as well as The first conditional feature vector is processed to generate the representation of the first 3D space including the person in the predetermined pose.

4. The method according to claim 2 or 3, wherein the representation of the first 3D space including the person in the predetermined posture includes 3 feature planes, and the feature planes are configured to provide feature data associated with points in the first 3D space.

5. The method according to any one of claims 2 to 4, wherein obtaining the one or more index positions comprises, for each index position: processing the pose parameters and the camera parameters to generate a first plurality of coordinates indicating the positions of corresponding first spatial points in a first image of the parameterized human body model arranged in the pose described by the pose parameters; applying a mapping transformation to the first plurality of coordinates to generate a second plurality of coordinates indicating the positions of corresponding second spatial points in a second image of the parameterized human body model in the predetermined pose, wherein the mapping transformation is based on the pose described by the pose parameters and the predetermined pose; and The one or more index positions are obtained based on the corresponding second plurality of coordinates.

6. The method according to any one of claims 5, wherein obtaining the one or more index positions comprises: The second coordinates and the pose parameters are processed by a deformation network module.

7. The method according to any one of claims 2 to 6, wherein processing the camera parameters and the pose parameters to generate geometry information and appearance information further comprises: Feature data obtained by sampling the representation at the one or more index locations is processed by a multilayer perceptron to generate the appearance information.

8. The method according to any one of claims 2 to 7, wherein processing the camera parameters and the pose parameters to generate geometry information and appearance information further comprises: generating 3D mesh data of the parameterized human model arranged in the predefined pose and having the shape described by the pose parameters; At the one or more index positions, sampling the 3D mesh data of the parameterized human model to obtain a first distance value; obtaining a second distance value using the first distance value and the samples of the representation at the one or more index locations; as well as The signed distance value obtained by modifying the first distance value using a signed distance value is provided as the geometric shape information.

9. The method of any preceding claim, wherein processing the geometry information and the appearance information to generate the image of a person comprises: Processing the geometric shape information and the appearance information by a volume rendering module of the neural network to generate a feature image and an RGB image; as well as The feature image and the RGB image are processed by a decoder module of the neural network to generate the image of the person.

10. The method of claim 9, wherein the resolution of the image of the person is higher than the resolution of the RGB image.

11. A method according to any preceding claim, wherein the pose parameters comprise skinned multi-person linear model parameters.

12. A method according to any preceding claim, wherein the geometry information characterises the 3D geometry of the person and the appearance information characterises the RGB appearance of the person.

13. A method of training a neural network as claimed in any preceding claim, the method comprising: (a) Generate training images of people by doing the following: Provide camera parameters that describe the viewing angle; providing pose parameters describing the shape and pose of the parameterized human body model; The camera parameters and the pose parameters are processed by a generator neural network, the generator neural network being configured to: (i) Generate geometry information and appearance information, and (ii) processing the geometric shape information and the appearance information to generate the training image of the person, the training image depicting the person viewed from the viewing angle and having the shape and the posture described by the posture parameters; (b) processing, by a discriminator neural network module, the training image of the person to generate a prediction of whether the training image of the person is an image of a real person or an image of a fake person, and (c) based on the prediction, modifying one or more network parameters of the generator neural network and the discriminator neural network.

14. The method of claim 13, wherein (i) generating geometric shape information and appearance information comprises: processing the camera parameters to generate a representation of a first 3D space including the person in a predetermined pose; Based on the camera parameters and the pose parameters, obtaining one or more index positions; as well as The geometry information and the appearance information are generated from the representation by sampling the representation at the one or more index locations.

15. The method of claim 14, wherein processing the camera parameters to generate a representation of a first 3D space including a person in a predetermined pose comprises: receiving a first latent vector comprising a random number; converting the camera parameters and the first latent vector into a first conditional feature vector; as well as The first conditional feature vector is processed to generate the representation of the first 3D space including the person in the predetermined pose.

16. The method of claim 14 or 15, wherein the representation of the first 3D space including the person in the predetermined pose comprises three feature planes configured to provide feature data associated with points in the first 3D space.

17. The method according to any one of claims 14 to 16, wherein obtaining the one or more index positions comprises: For each index position: processing the pose parameters and the camera parameters to generate a first plurality of coordinates indicating the positions of corresponding first spatial points in a first image of the parameterized human body model arranged in the pose described by the pose parameters; applying a mapping transformation to the first plurality of coordinates to generate a second plurality of coordinates indicating the positions of corresponding second spatial points in a second image of the parameterized human body model in the predetermined pose, wherein the mapping transformation is based on the pose described by the pose parameters and the predetermined pose; and Based on the second plurality of coordinates, the one or more index locations are obtained.

18. The method of any one of claims 17, wherein obtaining the one or more index positions comprises: The second coordinates and the pose parameters are processed by a deformation network module.

19. The method according to any one of claims 14 to 18, wherein processing the camera parameters and the pose parameters to generate geometry information and appearance information further comprises: Feature data obtained by sampling the representation at the one or more index locations is processed by a multilayer perceptron to generate the appearance information.

20. The method of any one of claims 14 to 19, wherein processing the camera parameters and the pose parameters to generate geometry information and appearance information further comprises: generating 3D mesh data of the parameterized human model arranged in the predefined pose and having the shape described by the pose parameters; sampling the 3D mesh data of the parametric human model at the one or more index positions to obtain a first distance value; obtaining a second distance value using the first distance value and the samples of the representation at the one or more index locations; as well as A signed distance value obtained by modifying the first distance value by using the second distance value is provided as the geometric shape information.

21. The method of any preceding claim, wherein processing the geometry information and the appearance information to generate the image of a person comprises: Processing the geometric shape information and the appearance information by a volume rendering module of the neural network to generate a feature image and an RGB image; as well as The feature image and the RGB image are processed by a decoder module of the neural network to generate the training image of the person.

22. The method of claim 21, wherein the resolution of the training image of the person is higher than the resolution of the RGB image.

23. The method of claims 13-22, wherein the pose parameters comprise skinned multi-person linear model parameters.

24. The method of claims 13-23, wherein the geometry information characterizes the 3D geometry of the person and the appearance information characterizes the RGB appearance of the person.

25. A system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1-24.

26. One or more non-transitory computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of any one of claims 1-25.

Images