Real-time body animation based on single images

Abstract

Systems and methods for single image based body animation are provided. An example method includes receiving an input image, the input image including a body of a person; segmenting the input image into a body portion and a background portion, wherein the body portion includes pixels corresponding to the body of the person; fitting a model to the body portion, wherein the model is configured to receive pose parameters representing a pose of the body and generate an output image, the output image including an image of the body assuming the pose; receiving a series of additional pose parameters, each of the series of additional pose parameters representing one of a series of additional poses of the body; providing each of the series of additional pose parameters to the model to generate a series of output images of the body assuming the additional poses; and generating an output video based on the series of output images.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to digital image processing. More specifically, the present disclosure relates to methods and systems for real-time body animation based on a single image. BACKGROUND

[0002] Body animation can be used in many applications, such as advertising, entertainment programs, social media networks, computer games, videos, video conversations, virtual reality, augmented reality, etc. Animation of a person’s body based on a single photograph can be particularly useful in various applications. For example, a person on a photograph can be “brought to life” by performing motions similar to real videos, such as dancing, performing acrobatics, fighting, etc. Animation of a person’s body based on a single photograph requires creating a realistic model of a particular person’s body and making the model perform actions or interact in a scene. SUMMARY

[0003] This section provides a simplified summary of a series of concepts that will be further described in the detailed description section below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in helping to determine the scope of the claimed subject matter.

[0004] According to one embodiment of the present disclosure, a method for real-time body animation based on a single image is provided. The method can include receiving, by a computing device, an input image. The input image can include a person’s body. The method can also include segmenting, by the computing device, the input image into a body portion and a background portion. The body portion can include pixels of the input image corresponding to the person’s body. The method can also include fitting, by the computing device, a model to the body portion. The model can be configured to receive a set of pose parameters representing a pose of the body and generate an output image based on the set of pose parameters. The output image can include an image of the body assuming the pose. The method can also include receiving, by the computing device, a series of additional sets of pose parameters. Each of the additional sets of pose parameters can represent at least one of additional poses of the body. The method can include providing, by the computing device, each of the additional sets of pose parameters in the series of additional sets of pose parameters to the model to generate a series of output images of the body assuming the additional poses. The method can also include generating, by the computing device, an output video based on the series of output images, wherein each frame of the output video includes at least one of the output images.

[0005] Segmenting the input image can be performed by a neural network. The series of additional sets of pose parameters can be generated based on a motion video. The motion video can feature another person assuming the additional poses.

[0006] The model can include a set of joints located in a three-dimensional (3D) space. The joints can indicate locations of joints in the body. The model can include a mesh including mesh points located in the 3D space. Each of the mesh points can be assigned a set of skinning weights. Each of the skinning weights can be associated with at least one of the joints. The model can include a texture map for generating a texture on the mesh.

[0007] The set of pose parameters can include rotation angles of the joints relative to a reference point. Generating the output image can include transforming the mesh by transforming the mesh points. Each of the mesh points can be rotated by an angle. The angle can be determined based on the rotation angles of the joints and the skinning weights. Generating the output image can further include applying the texture map to the transformed mesh to generate a texture of the transformed mesh.

[0008] The adaptation of the model can include determining a generic model based on the body part. The generic model can include a set of keypoints indicating joints in the body and a set of shape parameters indicating a shape of the body. The adaptation can further include determining a first contour of the body image based on the body part and determining a second contour of the body image based on the generic model. The adaptation can further include determining a set of paired points. Each pair of the paired points can include a first point located on the first contour and a second point located on the second contour. The adaptation can further include warping the generic model based on the set of paired points to obtain a warped model. The adaptation can further include determining the mesh and the set of joints based on the warped model.

[0009] The set of joints can be generated based on the mesh. The set of joints can include a set of keypoints. The texture map can be generated by unfolding the mesh to generate a two-dimensional (2D) representation of the mesh. Generating the texture map can further include determining, for each face of the 2D representation of the mesh, whether the face corresponds to a portion of the body visible in the input image. If the face corresponds to a portion of the body visible in the input image, a segmentation of the body part can be assigned to the face of the 2D representation of the mesh. If the face does not correspond to a portion of the body visible in the input image, a predicted face can be generated based on the body part and the predicted face can be assigned to the face of the 2D representation of the mesh.

[0010] The set of keypoints can be determined by a first neural network and the generic model can be determined by a second neural network.

[0011] According to another embodiment, a system for real-time body animation based on a single image is provided. The system can include at least one processor and memory storing processor-executable code, wherein the at least one processor can be configured to implement the operations of the above-described method for image-based real-time body animation when executing the processor-executable code.

[0012] According to yet another aspect of the disclosure, a non-transitory processor-readable medium is provided, the non-transitory processor-readable medium storing processor-readable instructions. When executed by a processor, the processor-readable instructions cause the processor to implement the above-described method for real-time body animation based on a single image.

[0013] Additional objects, advantages, and novel features of the examples will be set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings. The objects and advantages of the concepts can be implemented and achieved by means of the structures, substances, and combinations particularly pointed out in the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0014] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that

[0015] Figure 1 is a block diagram illustrating an example environment in which a method for real-time body animation based on a single image can be practiced.

[0016] Figure 2 is a block diagram illustrating a system for body animation based on a single image according to an example embodiment.

[0017] Figure 3 An example input image and a segmentation mask showing the contours of a person are shown.

[0018] Figure 4 An example graph of key points is shown.

[0019] Figure 5 An example first contour determined based on a segmentation mask and an example second contour determined as a projection of a generic model on the input image are shown.

[0020] Figure 6 A front view showing an example gravity center map, an example depth map, an example normal map, a corresponding warped gravity center map, a warped depth map, and a warped normal map are shown.

[0021] Figure 7 A visualization of gravity center coordinates is shown.

[0022] Figure 8 An example point cloud and a mesh generated based on the point cloud are shown.

[0023] Figure 9 An example warped depth map, an example warped normal map, an example point cloud generated based on the warped depth map and the warped normal map, and an example mesh generated based on the point cloud are shown.

[0024] Figure 10 An exemplary input image, a mesh of a generic model adapted to the input image, and a reconstructed mesh are shown.

[0025] Figure 11 An exemplary 3D surface to 2D surface UV unwrapping process is shown.

[0026] Figure 12 An exemplary 3D surface to 2D surface UV unwrapping process is shown for a cube.

[0027] Figure 13 An animation of an exemplary 3D model is shown.

[0028] Figure 14 is a flowchart showing exemplary transformations of individual images during an animation process.

[0029] Figure 15 is a flowchart showing a method for real-time body animation based on individual images according to one exemplary embodiment.

[0030] Figure 16 An exemplary computer system that can be used to implement a method for real-time body animation based on individual images is shown. DETAILED DESCRIPTION

[0031] The following detailed description of embodiments includes a description of the figures, which form a part of the detailed description. The description of the methods described in this section are not prior art to the claims and are included in this section for purposes of comprehension and explanation. The figures illustrate examples in accordance with exemplary embodiments. These exemplary embodiments, also referred to herein as “examples,” are described in sufficient detail to enable those skilled in the art to practice the subject matter. Other embodiments can be utilized and structural, logical, and operational changes can be made without departing from the scope of the claims. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope is defined by the appended claims and equivalents thereof.

[0032] The present disclosure can be implemented using a variety of technologies. For example, the methods described herein can be implemented by software running on a computer system and / or by hardware utilizing either a combination of a microprocessor and other specially designed application specific integrated circuits (ASICs), programmable logic devices, or any combination thereof, specifically designed for the function(s) described herein. In particular, the methods described herein can be implemented by a series of computer-executable instructions residing on a non-transitory storage medium such as a disk drive or computer-readable medium. It should be noted that the methods disclosed herein can be implemented by a computing device such as a mobile device, a personal computer, a server, a network node, or the like.

[0033] For the purposes of this patent document, the terms "or" and "and" should be construed as "and / or" unless otherwise indicated or unless the context clearly dictates otherwise. The term "a" should be construed as "one or more" unless otherwise indicated or unless the context clearly dictates otherwise. The terms "comprise," "comprising," "include," and "including" are interchangeable with respect to the terms "comprise," "comprising," "include," and "including" are interchangeable with respect to the terms "consist of" and "consist of." For example, the term "including" should be interpreted as "including but not limited to."

[0034] The present disclosure relates to methods and systems for real-time body animation based on a single image. The methods and systems of the present disclosure can be designed to work in real-time on a mobile device, such as a smartphone, tablet, or mobile phone, without the need to connect to the internet or use server-side computing resources, although embodiments can extend to methods involving web services or cloud-based resources.

[0035] Some embodiments of the present disclosure can enable real-time animation of a human body based on a single input image. The input image can be segmented to obtain a segmentation mask of the body. The input image can be analyzed to obtain a graph of keypoints representing the joints of the body and recover a pose of the body. A generic model can be adapted to the input image of the body and the graph of keypoints. The generic model can be trained on a dataset of images of different people with different body shapes and poses. A 3D model of the body to be used for animation can be further generated using the generic model and the segmentation mask. The 3D model can be substantially adapted to the contours of the body. The 3D model can include a set of joint nodes indicating the positions of the joints in the body, a reconstructed mesh of 3D points, skinning weights of the 3D points in the reconstructed mesh, and a texture map for texturing a pre-shrunk mesh. The 3D model can receive a set of pose parameters representing a pose. An image of the body assuming the pose can be rendered based on the 3D model and the set of pose parameters.

[0036] The 3D model can be further used to animate the body in the input image. For example, the 3D model can be provided with a series of additional sets of pose parameters representing additional poses to generate a series of frames. Each frame of the generated series of frames can include an image of the body in one of the additional poses. The generated series of frames can be further used to generate a video featuring the body performing a motion, where the body assumes the additional poses while performing the motion. The series of additional sets of pose parameters can be selected from a motion database. Each of the motions in the motion database can be represented in the form of a set of pose parameters. The motions in the motion database can be pre-generated using motion capture movements of real actors performing the motions. The motions in the motion database can also be pre-generated using a generic model and an editor for visualization of the generic model.

[0037] Example embodiments now are described with reference to the drawings. The drawings are diagrammatic illustrations of idealized example embodiments. Thus, the example embodiments discussed herein should not be understood or construed as being limited to the particular examples presented herein, as these example embodiments can include deviations and variations from the examples presented herein that will be apparent to those of skill in the art.

[0038] Figure 1 is a block diagram illustrating an example environment 100 in which methods for real-time body animation based on a single image can be practiced. The environment 100 can include a computing device 105. The computing device 105 can refer to a mobile device, such as a mobile phone, a smartphone, or a tablet. However, in other embodiments, the computing device 105 can refer to a personal computer, a laptop, a netbook, a set-top box, a television device, a multimedia device, a personal digital assistant, a game console, an entertainment system, an infotainment system, an in-vehicle computer, or any other computing device.

[0039] In certain embodiments, the computing device 105 can include a system 200 for body animation based on a single image. The system 200 can be implemented as instructions stored in a memory of the computing device 105 and executable by one or more processors of the computing device 105. The system 200 can receive an input image 110 and a set of pose parameters 130. The input image can include at least a body 115 of a person and a background 120. In some other embodiments, the input image can be stored in the computing device 105 or in a cloud-based computing resource communicatively connected to the computing device 105.

[0040] A set of pose parameters 130 can represent one or more poses that the body 115 can assume. In some embodiments, a pose parameter can represent a rotation angle of a key point associated with the body 115 with respect to a reference point in a three-dimensional (3D) space and / or an axis in the 3D space. For example, a key point can represent a joint (also referred to as a joint point) in a skeleton associated with the body 115. When the key point is rotated according to the rotation angle, the body 115 can assume a pose associated with the rotation angle.

[0041] In some embodiments, the system 200 can analyze the input image and generate a frame 150. The frame 150 can include an image of the body 115 assuming a pose associated with a pose parameter. Optionally, the frame 150 can also include an image of other objects, such as an image of the background 120 of the input image 110. A set of pose parameters 130 can represent a set of consecutive poses that the body 115 can assume during a particular motion, such as a dance move, an acrobatic jump, a fight move, and the like. The system 200 can generate a set of consecutive frames, where each frame in the set of consecutive frames corresponds to one of the consecutive pose parameters. The system 200 can further generate an output video 140 based on the set of consecutive frames. The output video can include images of the body 115 performing the particular motion defined by the set of consecutive pose parameters.

[0042] Figure 2 is a block diagram illustrating a system 200 for single-image based body animation according to an example embodiment. The system 200 can include a segmentation and pose estimation module 205, a generic model adaptation module 210, a reconstruction module 215, an assembly and skinning module 220, a texture module 225, a motion database 230, and an animation module 235.

[0043] The segmentation and pose estimation module 205 can be configured to receive an input image 110. The input image 110 can include pixels representing an image of a body 115 of a person. The module 205 can be configured to generate a segmentation mask. The segmentation mask can be an image showing an outline of the person on the input image 110.

[0044] Figure 3 An example of an image 110 and a segmentation mask 310 showing an outline of a person are illustrated. The segmentation mask can include "white" pixels 320 corresponding to pixels (body parts) of the body 115 of the person and "black" pixels 330 corresponding to static pixels (background parts) in the input image 110.

[0045] Segmentation of the input image in the body part 320 and the background part 330 can be performed by a neural network configured to determine for each pixel on the input image 110 whether the pixel corresponds to a human body. The architecture of the neural network performing the segmentation can include sequential convolutions followed by transposed convolutions and up-sampling. The architecture can also include symmetrical layers and "bridges" between the symmetrical layers as data is passed from earlier layers to the last layer. In some embodiments, to achieve faster inference, the shape of the input image can be reduced. In certain embodiments, padding can be applied to the input image to enable the neural network to run on images of any shape. The segmentation mask 310 can be further provided to the mesh reconstruction module 215 and the texture module 225.

[0046] Returning to Figure 2 The module 205 can also determine a pose of the body in the input image based on the input image. The pose can be determined in the form of a graph. The graph can include a set of key points and edges connecting some of the key points.

[0047] Figure 4 An example graph 410 and an example graph 420 are shown. Both the graph 410 and the graph 420 include key points related to joints of a person or key parts of a person (e.g., eyes, nose, neck, shoulders, legs, elbows, etc.). The graph 420 includes more key points in the face region of the person than the graph 410.

[0048] An additional neural network can be configured to determine a predefined graph of key points (e.g., the graph 410 or the graph 420) based on the input image. Each key point can be represented in both XY coordinates in the plane of the input image and XYZ coordinates in a 3D space. The neural network for determining the key points can have a lightweight convolution with a special architecture. For example, separate convolutions can be used to determine the XY coordinates and to determine the XYZ coordinates. The neural network can be trained in a supervised manner based on a large amount of prepared provable information ("ground truth" data). The graph of key points can be further provided to the generic model adaptation module 210.

[0049] Returning to Figure 2 The generic model adaptation module 210 can be configured to generate a generic model based on the input image and the graph of key points. The generic model can represent a general appearance of a person and a pose of the person. The generic model can include shape parameters. The shape parameters can include a vector of 3D points representing a shape of the body of the person. The generic model can also include a vector of pose parameters, where each of the pose parameters determines an axis-angle rotation of at least a joint in the body. The joints of the body can correspond to the key points in the graph of key points. In some embodiments, a mesh representing the body of the person can be generated using the generic model.

[0050] A generic model can be designed to be complex enough to cover a variety of shapes and poses of a person. On the other hand, the generic model is not computationally complex. The generic model can be a parametric function of a fixed number of model parameters, shape parameters, and pose parameters. The generic model can represent a human body in a variety of different shapes and poses that a real person can perform. Representing the generic model as a parametric function can enable saving memory of a computing device and can enable using optimized matrix computations to compute motions to improve computation speed.

[0051] The generic model can be trained by a neural network on two datasets. The first dataset can include 3D scans of people in different poses. The second dataset can include scans of bodies of people in different shapes. The goal of the training is to optimize the trainable parameters of the generic model to minimize the difference between the scans and images reconstructed using the generic model. Since the two datasets can be different, the parameters of the generic model related to poses can be trained based on the first dataset and the parameters related to shapes can be trained based on the second dataset

[0052] The generic model can also include pose parameters related to a head of a person. The pose parameters related to the head can be used to represent eyebrows, chin, and so on. The pose parameters related to the head of a person can be learned using a third dataset that includes face shapes and facial expressions. The parameters trained on the first dataset, the second dataset, and the third dataset can be aligned so that the parameters affect the generic model in the same way even though the parameters are trained on different datasets.

[0053] In some embodiments, the parameters learned by the generic model can include skinning weights, shape coefficients, pose parameters, and joint regressors. The skinning weights can represent values used to determine how each joint affects each vertex of a mesh associated with the generic model. The mesh can represent a shape of a body of a person. The mesh can be animated using the skinning weights. The skinning weights can be represented by an N joints x N vertices matrix, where N joints is a number of joints, and N vertices is a number of vertices in the mesh. The shape coefficients can be used to alter an initial generic model using shape parameters so that the generic model is appropriately shaped in terms of height, weight, waist, low hip, and so on. The joint regressors can include values used to determine an initial position of a joint of a person relative to a shape of the person. The joint parameters can be represented by a matrix similar to the matrix of the skinning weights. After training, the generic model can generate a shape and a pose of a human body based on a set of shape parameters and a set of pose parameters.

[0054] A neural network can be used to adapt a generic model to the input image 110. The neural network can be configured to pass the input image through a convolutional encoder. The output of the convolutional encoder can be further passed to an iterative regression factor that outputs shape parameters and pose parameters of the generic model. The iterative regression factor can minimize a reprojection error. The reprojection error can be computed as a difference between real joints and predicted joints of a person on the input image.

[0055]

[0056] where K is the number of joints, x real (i) are coordinates of real joints, and x pred (i) are coordinates of predicted joints. Only currently visible joints can be considered in the reprojection error.

[0057] The iterative regression factor can include a generative adversarial network (GAN). The GAN can be used to ensure that the generated generic model looks like a real person mesh. A traditional GAN objective can be used, which is given by the following equation:

[0058] min D max G E x～Pdata log(D(x)) + E Z～Pz(Z) log(l - D(G(z)))

[0059] To implement the neural network on a mobile device, the convolution can be performed by a similar method as used in MobileNet. The GAN can be implemented using the capabilities of a framework such as TensorFlow. The generic model can be provided to the reconstruction module 215.

[0060] The reconstruction module 215 can be configured to generate a 3D model to be used in an animation based on the generic model and the segmentation mask. The generic model can describe a limited space of human shapes. The generic model can not represent clothes, hair, finger positions on hands, and other specific details of a person. The generic model can be used to create a 3D model. The 3D model can draw as many details of a specific person shape as possible. In particular, the 3D model can be constructed to fit substantially exactly to the contours of the person in the input image. In other words, the 3D model can be constructed to cover the contours in the segmentation mask. In further embodiments, the 3D model can be constructed to cover the hair, clothes, and fingers of the person on the input image, so that the animation of the 3D model looks realistic.

[0061] The 3D model can include a reconstructed mesh and a set of joints located in a three-dimensional (3D) space. The joints can indicate locations of joints in the body. The reconstructed mesh can include 3D points that are different from 3D points of the mesh of the generic model. Each of the points of the reconstructed mesh can be assigned a set of skinning weights. Each of the skinning weights can be associated with at least one of the joints. The 3D model can further include a texture map to generate a texture on the mesh.

[0062] The module 215 can generate a depth map, a normal map, and a barycenter map of the generic model generated by the module 210. In some embodiments, the depth map, the normal map, and the barycenter map can be presented via a portable network graphics (png) image of both the front side and the back side of the generic model. The module 215 can determine a first contour and a second contour of the body of the person in the input image. The first contour can be determined based on the segmentation mask. The second contour can be determined as a projection of the generic model on the input image.

[0063] Figure 5 An exemplary first contour determined based on the segmentation mask and an exemplary second contour determined as a projection of the generic model on the input image are shown. The module 215 can match boundary points 540 located on the contour of the first contour with boundary points 550 located on the contour of the second contour. The boundary points 540 can be determined using coordinates of key points 530 of the body in the input image. The key points 530 can be determined by the neural network in the module 205. The boundary points 550 can be determined based on joint locations determined based on the generic model. Each of the boundary points 540 can be found as a point on the contour of the first contour that is closest to one of the key points 530. Each of the boundary points 550 can be found as a point on the contour of the second contour that is closest to one of the joint locations determined based on the generic model.

[0064] After the boundary points 540 are matched with the boundary points 550, the module 215 can perform a linear interpolation 560 between the boundary points 540 and the boundary points 550 to obtain points located between the boundary points. Matching the boundary points using the key points 530 and the joint locations can be faster and more accurate than matching the boundary points based on minimizing distances with dynamic programming, as it is performed in the current existing methods.

[0065] The normal map, the depth map, and the barycenter map can be further warped by using an average value coordinate algorithm on information about the points between the boundary points. Thus, the warped normal map, the warped barycenter map, and the warped depth map fit the original contour of the person in the segmentation mask and can be further used to determine the 3D model of the animation.

[0066] Figure 6A front view showing the exemplary barycentric map 610, exemplary depth map 620, exemplary normal map 630, and corresponding warped barycentric map 640, warped depth map 650, and warped normal map 660 is shown. The module 215 can store the depth map by storing the coordinates (x, y) and z value for each point, e.g., pixel in the input image. The normal map can be stored by storing, for each (x, y) coordinate, the normal vector at this point, which is a 3D vector (Nx, Ny, Nz) in the axis coordinates x, y, z. The barycentric map can be stored by storing, for each (x, y) coordinate, 1) the index of the face in the mesh associated with the generic model that includes the projection point (x, y), and 2) the first two barycentric coordinates (a and β). The third barycentric coordinate can be computed from a and β.

[0067] Figure 7 A visualization of the barycentric coordinates is shown.

[0068] Returning to Figure 2 , the module 215 can further construct a reconstructed mesh. First, a point cloud can be generated based on the warped depth map and the warped normal map. In the warped depth map and the warped normal map, each point is represented by 6 values: the coordinates (x, y, z) and the normal vector (Nx, Ny, Nz). Generating the point cloud can include generating a dense point cloud of (x, y, z) points. A first mesh for the front surface of the body and a second mesh for the back surface of the body can be further separately generated by passing through the point cloud. The first mesh and the second mesh can be further merged into one reconstructed mesh representing the 3D surface of the body. The reconstructed mesh can be fitted to the contours in the segmentation mask. During the generation of the reconstructed mesh, the module 215 can store the (x, y) coordinates of the depth map and the normal map of the generic model for each vertex of the mesh prior to warping.

[0069] Figure 8 An exemplary point cloud 810 and a mesh 820 generated based on the point cloud 810 are shown.

[0070] Figure 9 An exemplary warped depth map 910, an exemplary warped normal map 920, an exemplary point cloud 930 generated based on the warped depth map 910 and the warped normal map 920, and an exemplary mesh 940 generated based on the point cloud 930 are shown.

[0071] Figure 10 An exemplary input image 1010, a mesh 1020 of a generic model fitted to the input image 1010, and a reconstructed mesh 1030 of a 3D model are shown. The reconstructed mesh 1030 can match the contours of the person in the input image 1010.

[0072] Returning to Figure 2The assembly and skinning module 220 can determine skinning weights for vertices in the reconstructed mesh generated by module 215. The skinning weights and positions of the joints can be further used to render the mesh for animation.

[0073] In some embodiments, the positions of the joints can be considered the same as the positions in the generic model determined in module 210. In other embodiments, the positions of the joints can be derived by applying a warp to the skinning weights of the generic model and the joint regression factors. In certain embodiments, the positions of the joints can be determined by warping the positions of the joints to the starting pose. The warp can be determined based on the 2D projection of the positions of the joints in the starting pose.

[0074] The skinning weights for the vertices in the reconstructed mesh can be determined using the following procedure. For each vertex in the reconstructed mesh, module 220 can determine the (x, y) coordinates on the warp depth map, the warp normal map, and the warp barycentric map. Module 220 can further determine the point in the generic model corresponding to the (x, y) coordinates based on the warp barycentric map. Module 220 can further determine the index of the face of the mesh of the generic model and the barycentric coordinates of the face based on the point. The skinning weight of the point can be determined as the weighted average of the skinning weights (from the generic model) of the three vertices of the face, where the weights are determined by the barycentric coordinates of the face. This way of determining the skinning weights can be faster than warping the skinning weight matrix, since it enables the skinning weights of the vertices to be determined on the fly without having to process and store all the skinning weights of all the vertices in the reconstructed mesh. However, in certain embodiments, the skinning weight matrix can be determined for all the vertices of the reconstructed mesh and stored in the memory of the computing device.

[0075] The texture module 225 can generate a texture map for the reconstructed mesh. The texture map can be generated using the input image and the segmentation mask. Generating the texture map can include UV unwrapping of the reconstructed mesh to obtain a UV texture map. The texture map can be designed in such a way that all other reconstructed meshes have the same UV texture map, even though the other reconstructed meshes can differ in terms of the outline, the number of vertices, and the number of faces. This can enable transferring the texture from one reconstructed mesh generated for an input image and a segmentation mask to another reconstructed mesh generated for the same input image and the same segmentation mask. This can also enable modifying the texture map to change, for example, the colors and the clothes in the input image.

[0076] UV unwrapping can be defined as the process of flattening the surface of a 3D object into a 2D surface. UVs are the coordinates (also known as texture coordinates) of the vertices of the points of the surface.

[0077] Figure 11 An example UV unwrapping process of an example 3D surface 1110 to a 2D surface 1120 is shown.

[0078] Figure 12 An exemplary UV unwrapping process of a 3D surface of a cube 1210 to a 2D surface 1220 is shown. The 2D surface 1220 can be used as a texture map for the 3D surface of the cube 1210. Note that two adjacent points on two different faces near an edge 1230 of the cube 1210 can correspond to two points in the 2D surface that are located at two different parts of the texture map in a spaced apart manner.

[0079] During the generation of the reconstructed mesh, the module 215 can track the correspondence between the three vertices of a face in the reconstructed mesh and the three points A, B, and C (not necessarily vertices) in the mesh of the generic model. The generic model can also include a texture map. The module 225 can determine the coordinates on the texture map of the generic model that correspond to the vertices of a face in the reconstructed mesh. Using the triangle on the texture map of the generic model determined by the three points A, B, and C, the face on the texture map of the reconstructed mesh can be further generated. If the three points A, B, and C are located on different parts of the texture map of the generic model, the triangle A, B, C can be split into multiple triangles, where each of the triangles is located on the same part of the texture map of the generic model. The described method enables creating UV unwrapping for reconstructed meshes that have the same boundary and semantics. For example, the texture of a person’s face can be located at the same location in the texture map generated for all reconstructed meshes of the 3D model.

[0080] The projection of the reconstructed mesh matches the contour of the person. Therefore, the front side of the reconstructed mesh (the visible part of the texture) can be textured by considering the corresponding segmentation of the input image that includes the person’s body as the visible part of the texture. The unseen part of the texture of the reconstructed mesh can be repaired in different ways. For example, the opposite side of the reconstructed mesh can be textured by mirroring the texture of the front side or synthesizing the texture based on the front side. All unseen parts of the texture can also be filled using an algorithm or neural network based repair method. The repair can be performed using a coordinate based method, where the texture is not represented as a color picture, but as coordinates of points on the original input image, where the color is taken from the point. In this way, the texture can have clear and realistic colors without blurring.

[0081] The neural network used to repair the unseen part can be trained based on images of people from different perspectives. The unseen texture can be predicted by the neural network. During the training of the neural network, the 3D model can generate images with textures that are not visible from a certain angle. The rendered images can then be compared to the “ground truth”. The weights of the neural network can then be determined using backpropagation.

[0082] In some embodiments, to acquire a coherent and reliable looking good texture mesh, one can first draw the unseen parts in the images generated from the 3D model and then transfer them to the texture map. The 3D model can generate images from different viewpoints with the unseen parts highlighted. Then a generative neural network or a simpler algorithm can be used to fill in the "holes" on the images. The texture mesh can be back-projected to the texture map of the 3D model, replacing the previously missing parts with the recovered ones.

[0083] Return Figure 2 The animation module 235 can generate a frame of an output video having a characteristic of a motion of a person in an input image based on a series of multiple sets of pose parameters. The series of multiple sets of pose parameters can be stored in the motion database 230. The series of multiple sets of pose parameters can represent at least one motion. Each set of the series of multiple sets of pose parameters can be used to generate one frame of the output video. The series of multiple sets of pose parameters can be acquired by using motion capture of movements performed by real actors and digitizing the movements. In some embodiments, the series of multiple sets of pose parameters can be artificially generated by a 3D motion designer operating a generic model in an editor. The motion database can include different series of multiple sets of pose parameters representing different motions, such as dances, acrobatics, and so on.

[0084] Each frame can be generated by re-computing the 3D model using a set of pose parameters corresponding to the pose. Thus, new positions of the vertices of the reconstructed mesh in 3D space can be generated and textured using the texture map to generate an image of the body of the person assuming the pose. The image can be inserted into a frame. Standard animation rendering procedures for rigged models can be used, including computing a rotation matrix for each joint by the following equation:

[0085]

[0086] where rot(...) is the rotation part of the 4x4 rigid matrix, and j is the translation part of the 4x4 rigid matrix.

[0087] The rotation matrix can be converted to a dual quaternion, mixed with the skinning weights, normalized, and then converted back to a rigid transformation matrix. The rigid transformation matrix is needed to ensure the rigidity of the mixed transformation and to prevent skinning artifacts. The rotation-translation motion (rigid transformation) with a dual quaternion can be defined as:

[0088]

[0089] The dual quaternion can be mixed as follows:

[0090]

[0091] where w- is a skinning weight coefficient, and q- is a vector of dual quaternions of the transformation. In other embodiments, linear blend skinning can be used to blend the rotation matrices. In these embodiments, the rotation matrices can be blended by a summation of linear combinations of the rotation matrices, where the coefficients of the linear combinations are normalized.

[0092] Figure 13 An animation of an example 3D model 1300 is shown. Image 1310 includes an image of a body generated by the 3D model based on a first set of pose parameters from a series of multiple sets of pose parameters. Image 1320 includes an image of a body generated by the 3D model based on a second set of pose parameters from the series of multiple sets of pose parameters.

[0093] Figure 14 is a flowchart illustrating an example transformation of a single image during animation rendering. Process 1400 can be performed by Figure 2 the system 200 shown in FIG. 1. The input image 110 can include a body of a person. The input image 110 can be provided to the segmentation and pose estimation module 205 to generate a segmentation mask 310 of an outline of the person. The module 205 can also generate a set of keypoints 410 indicating locations of joints of the body of the person. The generic model adaptation module 210 can further use the set of keypoints 410 and the input image to generate a generic model 1020. The generic model 1020 and the segmentation mask 310 can be provided to the reconstruction module 215 to generate a 3D model 1030 to be used in an animation. The 3D model 1030 can be further processed by the rigging and skinning module 220 to estimate skinning weights. Texture mapping can be further generated using a reconstructed mesh of the 3D model 1030 and the segmentation mask to texture the reconstructed mesh. Image 1410 shows an image generated by the 3D model, where the reconstructed mesh is textured using the texture mapping. Image 1420 shows a frame including an image generated by the animation module 235 based on a set of pose parameters.

[0094] Figure 15 is a flowchart illustrating a method 1500 for real-time body animation based on a single image, according to one example embodiment. The method 1500 can be performed by the computing device 105 and the system 200.

[0095] The method 1500 can begin in block 1505 by receiving an input image. The input image includes a body of a person.

[0096] In block 1510, the method 1500 can include segmenting the input image into a body portion and a background portion. The body portion can include pixels of the input image corresponding to the body of the person.

[0097] In block 1515, the method 1500 can include fitting the model to the body part. The model can be configured to receive a set of pose parameters representing a pose of the body and generate an output image based on the set of pose parameters. The output image can include an image of the body assuming the pose.

[0098] In block 1520, the method 1500 can include receiving a series of additional sets of additional pose parameters. Each of the additional sets can represent at least one of an additional pose of the body.

[0099] In block 1525, the method 1500 can include providing each of the additional sets of the series of additional sets of pose parameters to the model to generate a series of output images of the body assuming the additional poses.

[0100] In block 1530, the method 1500 can include generating an output video based on the series of output images. Each frame of the output video can include at least one of the series of output images.

[0101] Figure 16 An example computing system 1600 that can be used to implement the methods described herein is shown. The computing system 1600 can be implemented in the context of the similar computing device 105 and system 200.

[0102] As shown in Figure 16 The hardware components of the computing system 1600 can include one or more processors 1610 and a memory 1620, as shown in FIG. 16. The memory 1620 stores, in part, instructions which are executed by the processor 1610, along with possibly data, which are also stored by the processor 1610 in the memory 1620, when the system 1600 is in operation. The memory 1620 can store a usable code when the system 1600 is running. The system 1600 can also include an optional mass storage device 1630, an optional portable storage medium reader 1640, one or more optional output devices 1650, one or more optional input devices 1660, an optional network interface 1670, and one or more optional peripheral devices 1680. The computing system 1600 can also include one or more software components 1695, such as software components that can implement the portrait animation methods described herein.

[0103] Figure 16 The components shown in FIG. 16 are those typically found in computing devices of this type. However, components having different and / or different numbers of components can be used. For example, a memory device 1620 can be incorporated on a chipset rather than stored apart from the processor 1610 as shown. The same can be true for a portable storage medium reader 1640, network interface 1670, and peripheral device(s) 1680. In some embodiments, software components 1695 include, for example, operating systems which can be used to perform the methods described herein.

[0104] Mass storage devices 1630, which can be implemented using magnetic disk drives, solid state disk drives, or optical disk drives, are non-volatile storage devices used to store data and instructions for use by processor 1610. Mass storage devices 1630 can store system software (e.g., software components 1695) for implementing embodiments described herein.

[0105] Portable storage media drive 1640 operates in conjunction with a portable non-volatile storage medium, such as a compact disk (CD) or a digital video disk (DVD), to input and output data and code to and from computing system 1600. System software (e.g., software components 1695) for implementing embodiments described herein can be stored on such a portable medium and input to computing system 600 via portable storage media drive 1640.

[0106] Optional input devices 1660 provide a portion of a user interface. Input devices 1660 can include an alphanumeric keypad, such as a keyboard, for inputting alphanumeric and other information, or a pointing device, such as a mouse, a trackball, a stylus, or cursor direction keys. Input devices 1660 can also include a camera or a scanner. In addition, user interface components can be provided by a combination of input devices 1660 and output devices 1650, as is shown in Figure 16 System 1600 includes optional output devices 1650. Suitable output devices include speakers, printers, network interfaces, and monitors.

[0107] Network interface 1670 can be used to communicate with external devices, external computing devices, servers, and networking systems via one or more communications networks, such as one or more wired networks, wireless networks, or optical networks, including, for example, the Internet, intranets, Local Area Networks (LANs), Wide Area Networks (WANs), cellular telephone networks, Bluetooth radios, and IEEE 802.11 based radio frequency networks, etc. Network interface 1670 can be a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. Optional peripheral devices 1680 can include any type of computer support device, to add additional functionality to the computer system.

[0108] Components included in computing system 1600 are intended to represent a broad category of computer components. Thus, computing system 1600 can be a server, a personal computer, a handheld computing device, a telephone, a mobile computing device, a workstation, a minicomputer, a mainframe computer, a network node, or any other computing device. Computing system 1600 can also include different bus configurations, networking platforms, multiprocessor platforms, etc. Various operating systems (OS) can be used, including UNIX, Linux, Windows, Macintosh OS, Palm OS, and other suitable operating systems.

[0109] Some of the functionality described above can consist of instructions stored on storage media such as computer-readable media or processor-readable media. The instructions can be retrieved and executed by a processor. Some examples of storage media are memory devices, tapes, disks, and the like. Memory devices include devices such as random access memory (RAM), read-only memory (ROM), and the like. The instructions can be implemented in various forms. For example, in some examples, instructions are implemented using hardware logic circuitry. In some examples, instructions are implemented using a combination of hardware logic circuitry and software. In some examples, instructions are implemented using software. A software language can be Java, C, C++, Visual C++, Visual Basic, Java Script, etc. The instructions can be part of an operating system or a specific application, component, program, module, object, or the like. A processor can execute the instructions. The instructions can be stored on a computer-readable medium, which can be a storage device or memory. The instructions presented here can be in some examples updated or modified.

[0110] Notably, any hardware platform suitable for performing the processing described herein is suitable for use with the present application. As used herein, the terms "computer-readable storage medium" and "computer-readable storage media" refer to any media that participate in providing instructions to a processor for execution. Such media can take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as a fixed disk. Volatile media includes dynamic memory, such as a system RAM. Transmission media includes coaxial cables, copper wire, and fiber optics, among others, including a wire that comprises one embodiment of a bus. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), any other memory chip or cartridge, a carrier wave, or any other medium from which a computer can read.

[0111] Various forms of computer-readable media can be involved in carrying one or more sequences of one or more instructions to a processor for execution. A bus carries the data to system RAM, from which a processor can retrieve and execute the instructions. The instructions received by system processor can optionally be stored on a fixed disk, either before or after execution by processor.

[0112] Thus, methods and systems for real-time body animation based on a single image have been described. While embodiments have been described with reference to the specific example embodiments, it will be evident that various modifications and changes can be made to these example embodiments without departing from the broader spirit and scope of the application. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for single-image-based body animation, the method comprising: receiving, by a computing device, an input image, the input image comprising a body of a person; segmenting, by the computing device, the input image into a body portion and a background portion, wherein the body portion comprises pixels of the input image that correspond to the body of the person; fitting, by the computing device, a model to the body portion, wherein the model is configured to: receive a set of pose parameters representing a pose of the body; and generate, based on the set of pose parameters, an output image comprising an image of a body assuming the pose; receiving, by the computing device, a series of additional sets of pose parameters, each of the additional sets of pose parameters representing at least one of an additional pose of a body, wherein the series of additional sets of pose parameters is generated based on a motion video, wherein frames of the motion video feature another person assuming the additional poses; providing, by the computing device, each of the series of additional sets of pose parameters to the model to generate a series of output images of the body assuming the additional poses; and generating, by the computing device and based on the series of output images, an output video, wherein each frame of the output video comprises at least one of the output images, wherein the model comprises: a set of joints in a three-dimensional space, the joints indicating locations of joints in the body; a mesh comprising mesh points in the three-dimensional space, each of the mesh points being assigned a set of skinning weights, each of the skinning weights being associated with at least one of the joints; and a texture map to generate a texture on the mesh, and wherein fitting the model comprises: determining, based on the body portion, a generic model comprising a set of key points indicating joints in the body and a set of shape parameters indicating a shape of the body; determining a first silhouette of a body image based on the body portion; determining a second silhouette of the body image based on the generic model; determining a set of pairs of points, wherein each of the pairs of points comprises a first point located on the first silhouette and a second point located on the second silhouette; warping the generic model based on the set of pairs of points to obtain a warped model; and determining the mesh and the set of joints based on the warped model.

2. The method of claim 1, wherein, Segmenting the input image is performed by a neural network.

3. The method of claim 1, wherein: the set of pose parameters comprises rotation angles of the joints relative to a reference point; and generating the output image comprises: transforming the mesh by transforming the mesh points, wherein each of the mesh points is rotated by an angle determined based on the rotation angles of the joints and the skinning weights; and applying the texture map to the transformed mesh to generate a texture of the transformed mesh.

4. The method of claim 1, wherein, The set of joints is generated based on the mesh.

5. The method of claim 1, wherein, The set of joints is the set of key points.

6. The method of claim 1, wherein, The texture map is generated by: unfolding the mesh to generate a two-dimensional representation of the mesh; and for each face of the two-dimensional representation of the mesh: determining whether the face corresponds to a portion of the body that is visible in the input image; based on determining that the face corresponds to a portion of the body that is visible in the input image, assigning a segmentation of the body portion to the face of the two-dimensional representation of the mesh; and based on determining that the face does not correspond to a portion of the body that is visible in the input image: generating a predicted face based on the body portion; and assigning the predicted face to the face of the two-dimensional representation of the mesh.

7. The method of claim 1, wherein, A set of the keypoints are determined by a first neural network, and the generic model is determined by a second neural network.

8. A system for single image based body animation, the system comprising at least one processor, memory storing processor-executable code, wherein, At least one of the processors is configured to implement the following when executing the processor-executable code: receiving an input image, the input image comprising a body of a person; segmenting the input image into a body portion and a background portion, wherein the body portion comprises pixels of the input image that correspond to the body of the person; fitting a model to the body portion, wherein the model is configured to: receive a set of pose parameters representing a pose of the body; and generate an output image based on the set of pose parameters, the output image comprising an image of the body assuming the pose; receiving a series of additional sets of pose parameters, each set of the series of additional sets of pose parameters representing at least one of an additional pose of the body, wherein the series of additional sets of pose parameters are generated based on a motion video, wherein frames of the motion video feature another person assuming the additional pose; providing each set of the additional sets of pose parameters to the model to generate a series of output images of the body assuming the additional pose; and generating an output video based on the series of output images, wherein each frame of the output video comprises at least one of the output images, wherein the model comprises: a set of joints in three-dimensional space, the joints indicating locations of joints in the body; a mesh comprising mesh points in three-dimensional space, each of the mesh points being assigned a set of skinning weights, each of the skinning weights being associated with at least one of the joints; and a texture map to generate a texture on the mesh, and wherein fitting the model comprises: determining a generic model based on the body portion, the generic model comprising a set of keypoints indicating joints in the body and a set of shape parameters indicating a shape of the body; determining a first silhouette of the body image based on the body portion; determining a second silhouette of the body image based on the generic model; determining a set of pairs of points, wherein each pair of the pairs of points comprises a first point located on the first silhouette and a second point located on the second silhouette; warping the generic model based on the set of pairs of points to obtain a reconstructed model; and determining the mesh and the set of joints based on the reconstructed model.

9. The system of claim 8, wherein, segmenting the input image is performed by a neural network.

10. The system of claim 8, wherein: the pose parameters include rotation angles of the joint nodes relative to a reference point; and generating the output image includes: transforming the mesh by transforming the mesh points, wherein each of the mesh points is rotated by an angle that is determined based on the rotation angles of the joint nodes and the skinning weights; and applying the texture map to the transformed mesh to generate a texture of the transformed mesh.

11. The system of claim 8, wherein, a set of the joint nodes is generated based on the mesh.

12. The system of claim 8, wherein, a set of the joint nodes is a set of the key points.

13. The system of claim 8, wherein, the texture map is generated by: unfolding the mesh to generate a two-dimensional representation of the mesh; and for each face of the two-dimensional representation of the mesh: determining whether the face corresponds to a portion of the body that is visible in the input image; based on determining that the face corresponds to a portion of the body that is visible in the input image, assigning a segmentation of the body portion to the face of the two-dimensional representation of the mesh; and based on determining that the face does not correspond to a portion of the body that is visible in the input image: generating a predicted face based on the body portion; and assigning the predicted face to the face of the two-dimensional representation of the mesh.

14. A non-transitory processor-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to implement a method for body animation based on a single image, the method comprising: receiving an input image, the input image including a body of a person; segmenting the input image into a body portion and a background portion, wherein the body portion includes pixels of the input image that correspond to the body of the person; fitting a model to the body portion, wherein the model is configured to: receive pose parameters representing a pose of the body; and generate an output image based on the pose parameters, the output image including an image of the body in the pose; receive a series of additional pose parameters, each of the series of additional pose parameters representing at least one of an additional pose of the body, wherein the series of additional pose parameters is generated based on a motion video, wherein frames of the motion video feature another person in an additional pose; provide each additional set of pose parameters of the series of additional pose parameters to the model to generate a series of output images of the body in the additional pose; and generate an output video based on the series of output images, wherein each frame of the output video includes at least one of the series of output images, wherein the model includes: a set of joint nodes in three-dimensional space, the joint nodes indicating locations of joints in the body; a mesh including mesh points in three-dimensional space, each of the mesh points assigned a set of skinning weights, each of the skinning weights associated with at least one of the joint nodes; and texture mapping to generate a texture on the mesh, and wherein adapting the model comprises: determining a generic model based on the body part, the generic model comprising a set of key points indicative of joints in the body and a set of shape parameters indicative of a shape of the body; determining a first contour of a body image based on the body part; determining a second contour of the body image based on the generic model; determining a set of pairs of points, wherein each pair of the pairs of points comprises a first point located on the first contour and a second point located on the second contour; warping the generic model based on the set of pairs of points to obtain a warped model; and determining the mesh and the set of joint points based on the warped model.

Images