Method, system and computer program for generating a 3D head deformation model

Abstract

The electronic device performs a method for generating a three-dimensional (3D) head deformation model, the method including the steps of receiving a two-dimensional (2D) facial image; identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; mapping the first set of key points to a second set of key points that are placed on a plurality of vertices of a mesh of the 3D head template model based on a set of user-provided key point annotations to be placed on a 3D head template model; performing a deformation on the mesh of the 3D head template model to reduce a difference between the first set of key points and the second set of key points, thereby obtaining a deformed 3D head mesh model; and applying a blendshape method to the deformed 3D head mesh model to obtain a head model according to the 2D facial image.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. patent application Ser. No. 17 / 202,112, entitled “METHODS AND SYSTEMS FOR PERSONALIZED 3D HEAD MODEL DEFORMATION,” filed on March 15, 2021, and which is incorporated by reference in its entirety into this application.

[0002] Technical Field The present disclosure relates generally to imaging technology, and more particularly to image processing and head / face modeling methods and systems. [Background technology]

[0003] Commercially available face capture systems with multiple sensors (e.g., multi-view cameras, depth sensors, etc.) are used to obtain accurate three-dimensional (3D) face models for people with or without explicit markers. These tools capture human face geometry and texture information from multiple sensors and fuse the multimodal information into a general 3D face model. Benefiting from the multimodal information from various sensors, the obtained 3D face model is accurate. However, these commercial systems are expensive and require the purchase of additional software to process the raw data. In addition, these systems are usually deployed in face capture studios and actors or volunteers are required to obtain the data, which makes the data collection process time-consuming and even more costly. In short, face capture systems are expensive and time-consuming to obtain 3D face data. On the contrary, with the widespread availability of smartphones or cameras today, there is potentially a large amount of RGB (red, green, blue) images available. Taking RGB images as input to create a 3D face model can benefit from the large amount of image data.

[0004] A two-dimensional (2D) RGB image is merely a projection of the 3D world onto a 2D plane. Recovering 3D geometry from a 2D image is an ill-posed problem that requires optimization or learning algorithms to regularize the reconstruction process. Parameterized face model 3D Morphable Model (3DMM)-based methods have been developed and used for 3D face reconstruction. In particular, face models such as the Basel Face Model (BFM) and the Surrey Face Model (SFM) are commonly used face models, which require commercial licenses. Face model-based methods take as their base a set of scanned 3D human face models (showing various facial features and expressions) and generate a parameterized representation of the facial features and expressions based on the 3D face models. Based on the parameterization, the new 3D face can be represented as a linear combination of the base 3D face models. Due to the nature of these methods, the 3D face models used to form the base and parameter space limit the expressive power of face model-based methods. In addition, the optimization process of fitting 3D MMM parameters from input face images or 2D landmarks further sacrifices detailed facial features in the face images. Therefore, face model-based methods cannot accurately restore 3D facial features, and commercial licenses are required to use face models such as BFM and SFM.

[0005] With the popularity of deep learning algorithms, semantic segmentation algorithms have attracted much attention. Such algorithms can separate each pixel in a face image into different categories, such as background, skin, hair, eyes, nose, and mouth.

[0006] Although the semantic segmentation method can achieve relatively accurate results, the semantic segmentation of every pixel is a very complicated problem, which often requires a complex network structure and results in high computational complexity. In addition, a large amount of training data needs to be labeled in order to train the semantic segmentation network, and the semantic segmentation needs to separate the pixels of the entire image, which is very tedious, time-consuming and costly. Therefore, it is not suitable for scenes that do not require high average color accuracy but require high efficiency.

[0007] Keypoint-driven deformation methods that optimize the Laplacian and other derived operators have been well studied in academia. The formula for biharmonic deformation is given by Δ 2 We can write x' = 0. We define the constrained key points, i.e., boundary conditions, as x b '=x bc In the above formula, Δ is the Laplacian operator, x' is the position of the unknown deformed mesh vertex, and x bc is the position of a given keypoint after the deformation. A solution of a bi-Laplace equation is required in each dimension. Biharmonic functions are solutions to the bi-Laplace equation, but they are also minimizers of the so-called "Laplacian energy".

[0008] The nature of energy minimization is the smoothing of the mesh. If we were to apply the aforementioned minimizer directly, all the detailed features would be smoothed out. Moreover, when the keypoint positions remain unchanged, the deformed mesh is expected to be exactly the same as the original mesh. Due to these considerations, the preferred use of biharmonic deformations is to solve for the displacement of vertices other than their positions. In this way, the deformed positions can be written as x'=x+d, where d is the unknown vertex displacement in each dimension. Naturally, the formula for biharmonic deformation is d b =x bc -x b Assuming that, Δ 2 d=0, where d b is the displacement of the keypoint after the deformation.

[0009] With the rapid development of the game industry, customized facial avatar generation has become more and more common. For ordinary players who do not have artistic skills, it is very difficult to adjust the control parameters to generate a face that can show subtle changes.

[0010] In some existing face generation systems and methods, such as the Justice Face Generation System, the prediction of the face model is to predict 2D information in the image, such as segmentation of eyebrows, mouth, nose and other pixels in the photo. These 2D segmentations are susceptible to out-of-plane rotation and partial occlusion, and essentially require a frontal face. In addition, the similarity of the final game face avatar to the input is determined by a face recognition system, which limits the method to only realistic style games. If the game style is a cartoon style that is very different from real faces, the method cannot be used.

[0011] In some other existing face generation systems and methods, such as Moonlight Blade (Moonlight Blade Face Generation System), a real face is reconstructed from an input image. This method is limited to realistic style games and cannot be applied to cartoon style games. The output parameter of this method is then a reconstructed game style face mesh, after which template matching is performed for each part of the mesh. This approach limits the combination of different face parts. The overall diversity of the game face is closely related to the number of pre-generated templates. If the number of templates for a particular part, such as the shape of the mouth, is small, it may not generate many different variations, and the generated face may lack diversity. Summary of the Invention

[0012] Learning-based face reconstruction and keypoint detection methods rely on 3D ground-truth data as the gold standard to train a model that approximates the ground truth as closely as possible. Thus, the 3D ground truth determines the upper limit of the learning-based approach. To ensure the accuracy of face reconstruction and desirable keypoint detection, some embodiments generate a 3D face model ground truth using 2D facial keypoints annotation without using an expensive face capture system. The approach disclosed herein generates a 3D ground-truth face model that preserves detailed facial features of the input image, overcoming the shortcomings of existing face models such as 3D MMM-based methods that lose facial features, and also avoids the use of parameterized face models such as BFM and SFM (both of which require commercial licenses) required by some existing face model-based methods.

[0013] Apart from facial keypoint detection, in some embodiments, a multi-task learning and transfer learning solution is implemented for the facial feature classification task, so that more information can be extracted from the input facial image, and such information complements the keypoint information. Together with the predicted facial features, the detected facial keypoints are valuable for computer or mobile games to create a facial avatar for the player.

[0014] In some embodiments, a lightweight method is disclosed herein for extracting average colors of each part of a human face, including the average colors of skin, eyebrows, pupils, lips, hair and eyeshadow, from a single photo. At the same time, an algorithm is also used to automatically transform a texture map based on the average colors, so that the transformed texture still has the original brightness and color difference, but the main color is the target color.

[0015] With the rapid development of computer vision and artificial intelligence (AI) technology, the capture and reconstruction of 3D human facial key points has achieved high accuracy levels. More and more games are using AI detection to make game characters more vivid. The method and system disclosed herein customizes a 3D head avatar based on the reconstructed 3D key points. The general key point-driven deformation is applicable to any mesh. The head avatar customization process and deformation method proposed herein can find its application in scenarios such as automatic avatar creation and facial expression regeneration.

[0016] A method and system for automatically generating in-game facial avatars based on a single photo is disclosed herein. Through predicting facial key points, automatically processing the key points, and using deep learning methods to predict model parameters, the system disclosed herein can automatically generate in-game facial avatars that 1) have the characteristics of real faces in the photo, and 2) conform to the target game style. The system can be simultaneously applied to face generation for realistic and cartoon style games, and can be easily automatically adjusted according to different game models and bone definitions.

[0017] According to a first aspect of the present application, a method for constructing a face location map from a two-dimensional (2D) face image of a real person includes the steps of generating a coarse face location map from the 2D face image, predicting a first set of key points for the 2D face image based on the coarse face location map, identifying a second set of key points for the 2D face image based on user-provided keypoint annotations, and updating the coarse face location map to reduce a difference between the first set of key points and the second set of key points for the 2D face image.

[0018] In some embodiments, the method for constructing a face location map from a 2D face image of a real person further comprises extracting a third set of key points as a final set of key points based on the updated face location map, wherein the third set of key points have the same positions as the first set of key points in the face location map.

[0019] In some embodiments, the method for constructing a face position map from a 2D face image of a real person further comprises reconstructing a three-dimensional (3D) face model of the real person based on the updated face position map.

[0020] According to a second aspect of the present application, a method for extracting colors from a two-dimensional (2D) facial image of a real person includes the steps of identifying a plurality of keypoints in the 2D facial image based on a keypoint prediction model; rotating the 2D facial image until a selected target keypoint from the plurality of keypoints is aligned; locating a plurality of parts in the rotated 2D facial image, each part being defined by a respective subset of the plurality of keypoints; extracting an average color of each of the plurality of parts defined by a corresponding subset of the keypoints from pixel values ​​of the 2D facial image; and using the extracted colors of the plurality of parts in the 2D facial image to generate a personalized three-dimensional (3D) model of the real person matching each facial feature color of the 2D facial image.

[0021] According to a third aspect of the present application, a computer-implemented method for generating a three-dimensional (3D) head deformed model includes the steps of receiving a two-dimensional (2D) face image; identifying a first set of key points in the 2D face image based on an artificial intelligence (AI) model; mapping the first set of key points to a second set of key points located on a plurality of vertices of a mesh of the 3D head template model based on a set of user-provided key point annotations located on the 3D head template model; performing a deformation on the mesh of the 3D head template model to reduce a difference between the first set of key points and the second set of key points, thereby obtaining a deformed 3D head mesh model; and applying a blendshape method to the deformed 3D head mesh model to obtain a head model according to the 2D face image.

[0022] According to a fourth aspect of the present application, a method for customizing an in-game avatar's standard face using a two-dimensional (2D) facial image of a real person includes the steps of identifying a set of real keypoints in the 2D facial image, transforming the set of real keypoints into a set of game-style keypoints to be associated with the in-game avatar, generating a set of control parameters for the in-game avatar's standard face by applying the set of game-style keypoints to a keypoint-to-parameter (K2P) neural network model, and transforming the in-game avatar's standard face based on the set of control parameters, wherein the transformed face of the avatar has facial features of the 2D facial image.

[0023] According to a fifth aspect of the present application, an electronic device includes one or more processing units, a memory, and a number of programs stored in the memory, the programs, when executed by the one or more processing units, causing the electronic device to perform one or more of the methods set forth above.

[0024] According to a sixth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units, the programs, when executed by the one or more processing units, causing the electronic device to perform one or more of the methods set forth above.

[0025] The various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected primarily for ease of reading and explanation purposes, and not to describe or limit the subject matter of the present invention. [Brief description of the drawings]

[0026] So that the present disclosure may be more fully understood, a more particular description may be made by reference to the features of various embodiments, some of which are illustrated in the accompanying drawings. The accompanying drawings, however, merely illustrate the relevant features of the present disclosure and therefore should not be considered limiting, since the description may admit of other advantageous features.

[0027] [Figure 1] FIG. 1 illustrates an example keypoint definition, according to some implementations of the present disclosure.

[0028] [Diagram 2] FIG. 2 is a block diagram illustrating an example keypoint generation process, according to some implementations of the present disclosure.

[0029] [Diagram 3] 1A-1C illustrate an example process for transforming an initial coarse position map according to some implementations of the present disclosure.

[0030] [Figure 4] A diagram showing an example transformed position map that does not cover the entire face area, in accordance with some implementations of the present disclosure.

[0031] [Diagram 5] A diagram showing an example process of refining a transformed position map to cover the entire face area, according to some implementations of the present disclosure.

[0032] [Figure 6] FIG. 1 illustrates some example results of a position map refinement algorithm, according to some implementations of the present disclosure.

[0033] [Figure 7A] 1A-1C are diagrams illustrating several example comparisons of a final position map against an initial coarse position map, according to some implementations of the present disclosure. [Figure 7B] 1A-1C are diagrams illustrating several example comparisons of a final position map against an initial coarse position map, according to some implementations of the present disclosure.

[0034] [Figure 8A] FIG. 1 illustrates an example eyeglass classification network structure, according to some implementations of the present disclosure.

[0035] [Figure 8B] FIG. 2 illustrates an exemplary female hair prediction network structure, according to some implementations of the present disclosure.

[0036] [Figure 8C] FIG. 1 illustrates an exemplary male hair prediction network structure, according to some implementations of the present disclosure.

[0037] [Figure 9A] 1A-1C illustrate some example eyeglass classification prediction results, according to some implementations of the present disclosure.

[0038] [Figure 9B] FIG. 1 illustrates some exemplary female hair prediction results, according to some implementations of the present disclosure.

[0039] [Figure 9C] FIG. 1 illustrates some example male hair prediction results, according to some implementations of the present disclosure.

[0040] [Figure 10] 1 is a flowchart illustrating an example process for constructing a face position map from a 2D face image of a real person, according to some implementations of the present disclosure.

[0041] [Figure 11] 1 is a flowchart illustrating an example color extraction and adjustment process, according to some implementations of the present disclosure.

[0042] [Figure 12]FIG. 2 illustrates an exemplary skin color extraction method, according to some implementations of the present disclosure.

[0043] [Figure 13] FIG. 2 illustrates an example eyebrow color extraction method, according to some implementations of the present disclosure.

[0044] [Figure 14] FIG. 2 illustrates an example pupil color extraction method, according to some implementations of the present disclosure.

[0045] [Figure 15] FIG. 2 illustrates an example hair color extraction region used in a hair color extraction method, according to some implementations of the present disclosure.

[0046] [Figure 16] A diagram showing an example separation between hair pixels and skin pixels in a hair color extraction region, according to some implementations of the present disclosure.

[0047] [Figure 17] FIG. 2 illustrates an exemplary eyeshadow color extraction method, according to some implementations of the present disclosure.

[0048] [Figure 18] 1A-1C are diagrams illustrating some example hue adjustment results according to some implementations of the present disclosure.

[0049] [Figure 19] 1 is a flowchart illustrating an example process for extracting color from a 2D face image of a real person, according to some implementations of the present disclosure.

[0050] [Figure 20] 1 is a flowchart illustrating an example head avatar deformation and generation process according to some implementations of the present disclosure.

[0051] [Figure 21]FIG. 2 illustrates an example head template model configuration, according to some implementations of the present disclosure.

[0052] [Figure 22] 1A-1C are diagrams illustrating some example keypoint markings on a realistic-style 3D model and a cartoon-style 3D model, in accordance with some implementations of the present disclosure.

[0053] [Diagram 23] FIG. 13 illustrates an example comparison between a template model rendering, manually marked keypoints, and AI-detected keypoints, in accordance with some implementations of the present disclosure.

[0054] [Figure 24] FIG. 2 illustrates an example affine transformation of a triangle, in accordance with some implementations of the present disclosure.

[0055] [Diagram 25] 13A-13C are diagrams illustrating example comparisons of several head model deformation results with and without a blendshape process, in accordance with some implementations of the present disclosure.

[0056] [Figure 26] FIG. 13 illustrates an exemplary comparison of affine and biharmonic deformations with different weights, in accordance with some implementations of the present disclosure.

[0057] [Figure 27] FIG. 13 illustrates some example results automatically generated from some randomly chosen pictures of women using a realistic template model, in accordance with some implementations of the present disclosure.

[0058] [Figure 28] 1 is a flowchart illustrating an example process for generating a 3D head deformation model from a 2D face image of a real person, according to some implementations of the present disclosure.

[0059] [Figure 29] FIG. 2 illustrates exemplary keypoint processing flow steps according to some implementations of the present disclosure.

[0060] [Diagram 30] FIG. 2 illustrates an example keypoint smoothing process, according to some implementations of the present disclosure.

[0061] [Diagram 31] FIG. 1 is a block diagram illustrating an example key-point-to-control-parameter (K2P) conversion process according to some implementations of the present disclosure.

[0062] [Diagram 32] 1A-1C illustrate some example results of automatic face generation for a mobile game, according to some implementations of the present disclosure.

[0063] [Diagram 33] 1 is a flowchart illustrating an example process for customizing a standard face for an in-game avatar using a 2D facial image of a real person, according to some implementations of the present disclosure.

[0064] [Diagram 34] 1 is a schematic diagram of an example hardware structure of an image processing device according to some implementations of the present disclosure.

[0065] According to common practice, the various features illustrated in the drawings may not be drawn to scale. Thus, dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0066] Specific implementations will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are described to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used and that the subject matter may be practiced without these specific details without departing from the scope of the claims. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be implemented in many types of electronic devices.

[0067] Before describing the embodiments of the present application in further detail, the names and terms associated with the embodiments of the present application will be explained, and the names and terms associated with the embodiments of the present application will have the following explanation.

[0068] Facial Keypoints: predefined landmarks that determine the shape of certain facial features, such as the corners of the eyes, the chin, the tip of the nose, and the corners of the mouth.

[0069] Facial features: face border, eyes, eyebrows, nose, mouth and other features.

[0070] Face Reconstruction: Reconstruct the 3D geometric structure of the human face and commonly used representations including mesh models, point clouds or depth maps.

[0071] RGB Image: Red, Green, Blue three channel image format.

[0072] Position Map: Using red, green and blue channels in a regular image format, it stores the x, y and z coordinates of the face area, which is a representation of a 3D human face.

[0073] Facial feature classification: Includes hairstyle classification with or without glasses classification.

[0074] Convolutional Neural Network (CNN): A class of deep neural networks that are most commonly applied to the analysis of visual imagery.

[0075] Base Network: A CNN-like network that is used by one or more downstream tasks to act as a feature extractor.

[0076] Laplacian operator: A differential operator given by the divergence of the gradient of a function on Euclidean space.

[0077] Differentiable manifold: A type of topological space that is locally similar to a linear space so that it is possible to perform calculus.

[0078] Biharmonic functions: Fourth-order differentiable functions with a square Laplacian operator equal to 0, defined on a differentiable manifold.

[0079] Keypoint-driven deformation: A class of methods for deforming meshes by changing the positions of specific vertices.

[0080] Biharmonic deformation: A deformation method that uses the optimization of biharmonic functions with some boundary conditions.

[0081] Affine deformation: A keypoint-driven deformation method proposed in this disclosure that optimizes the affine transformation of triangles to achieve the goal of mesh deformation.

[0082] Face Model: A standard face mesh in a predefined target game.

[0083] Bones / Sliders: Control parameters that deform the face model.

[0084] As mentioned above, even if both input 2D images and 2D keypoints are fed into the optimization process to adapt the 3D MMM parameters, the optimization must balance between the fit of the 3D face model based on the base (i.e., 3D face model set) and the fidelity of the 2D keypoints. The optimization leads to the obtained 3D face model ignoring the 2D input keypoints, thereby sacrificing the detailed facial information brought by the 2D input keypoints. Among the existing 3D face reconstruction methods, face capture solutions can generate accurate reconstructions, but are expensive and time-consuming, and the obtained data shows limited variations of facial features (limited number of actors). On the other hand, face model-based methods can take 2D images or 2D landmark annotations as input, but the obtained 3D models are not accurate. To meet the requirements of rapid development of computer / mobile games, it is necessary to both produce the desired 3D model accuracy and reduce the required cost and time. To meet these requirements, a novel 3D ground-truth face model generation algorithm disclosed herein takes 2D images, 2D keypoint annotations, and a coarse 3D face model (in position map format) as input, transforms the coarse 3D model based on the 2D keypoints, and finally generates a 3D face model with good preservation of detailed facial features.

[0085] Besides solving the important problems in face reconstruction and keypoint prediction, a multi-task learning and transfer learning based approach for facial feature classification is also disclosed herein, which is partially built on the face reconstruction and keypoint prediction framework. In particular, glasses classification (with or without glasses) is achieved via multi-task learning, reusing the base network of face reconstruction and keypoint prediction. A linear classifier is trained on top of the existing face reconstruction and keypoint prediction framework, which largely reuses the existing model and avoids the introduction of another larger network for image feature extraction. In addition, another shared base network is used for male and female hairstyle classification. Hairstyle is a kind of important facial feature that complements facial keypoints or 3D face model. In the process of creating a 3D avatar for a user, adding hairstyle and glasses prediction can better reflect the user's facial features and provide a better personalized experience.

[0086] Facial keypoint prediction has been a research topic in computer vision for decades. With the recent development of artificial intelligence and deep learning, convolutional neural networks (CNNs) have promoted the progress of facial keypoint prediction. 3D face reconstruction and facial keypoint detection are two intertwined problems, and solving one can simplify the other. Traditional methods first solve 2D facial keypoint detection, and then further infer a 3D facial model based on the estimated 2D facial keypoints. However, when a face in an image is tilted (nodding or shaking its head), certain facial keypoints are occluded, leading to erroneous 2D facial keypoint estimation, and therefore the 3D facial model constructed on the erroneous 2D facial keypoints is inaccurate.

[0087] Ground truth data determines the upper limit of deep learning-based methods, so existing 3D face model datasets are not only limited in number but also available only for academic research. Meanwhile, face model-based methods need to use the Basel Face Model (BFM) or the Sally Face Model (SFM), both of which require commercial licenses. High accuracy and large amount of 3D ground truth is the most important issue in training any face reconstruction or keypoint estimation model.

[0088] Besides facial keypoint prediction, facial feature classification is an important aspect of creating a user's 3D avatar. With the predicted facial keypoints, we can only perform style transfer of the user's facial parts (i.e., eyes, eyebrows, nose, mouth, and facial contours). However, to better reflect the user's facial features, matching the user's hairstyle and adding a pair of glasses if the user is wearing glasses in the input image can be very helpful. Based on these requirements, a multi-task learning and transfer learning based facial feature classification approach is developed to achieve male / female hairstyle prediction and glasses prediction (with / without), which can make the created facial avatar more personalized to improve the user's experience.

[0089] In some embodiments, a keypoint representation as shown in FIG. 1 is used to represent the three-dimensional shape of the main parts of the face. FIG. 1 is a diagram illustrating an example keypoint definition according to some implementations of the present disclosure. The keypoints are numbered in a sequence that defines a particular feature of the face. In other words, there is a mapping relationship between the sequence number of the keypoints and a particular location on the face. For example, sequence number 9 corresponds to the bottom of the chin, sequence number 21 corresponds to the tip of the nose, etc. The keypoints focus on the boundaries of the main parts of the face, such as the facial contour, eye contour, and eyebrow contour. More keypoints mean a more accurate shape representation, but less predictable. In some embodiments, a definition of 96 keypoints is adopted in FIG. 1. In some embodiments, the user can change the specific definition and number of keypoints according to the user's own needs.

[0090] Many algorithms can predict the 3D coordinates of keypoints on a human face. Better performing methods use deep learning algorithms based on large amounts of offline 3D training data. However, in some embodiments, any 3D keypoint prediction algorithm can be used. In some embodiments, the definition of the keypoints is not fixed and the user can customize the definition as needed.

[0091] To solve the problem of 3D ground truth face model generation, the following automated algorithm is developed, which takes as input a 2D RGB image, 2D keypoint annotations, and a coarse location map. Figure 2 is a block diagram illustrating an exemplary keypoint generation process according to some implementations of the present disclosure. For example, a 2D RGB image of a face is used as an input image 202, and the 2D RGB image has a corresponding initial coarse location map 204, where each pixel in the initial coarse location map represents the spatial coordinates of the corresponding face point in the 2D RGB image. The 2D keypoint annotations 208 represent a set of user-provided keypoints, which are used to refine the set of detected keypoints 206 from the initial coarse location map 204.

[0092] FIG. 3 illustrates an example process for transforming an initial coarse position map according to some implementations of the present disclosure.

[0093] In some embodiments, a 3D reconstruction method is used to convert an input face image into a position map that contains 3D depth information for face features. For example, the position map may be a 2D three color (RGB) channel map with a 256×256 matrix array, with each array element having coordinates (x, y, z) that represent a 3D location on the face model. The 3D position coordinates (x, y, z) are represented by the RGB pixel values ​​on the position map for each array element. A particular face feature is located at a fixed 2D location in the 2D position map. For example, the tip of the nose can be identified by a 2D array element location of X=128 and Y=128 in the position map. Similarly, a particular key point identified for a particular face feature on the face may be located at the same array element location on the 2D position map. However, a particular key point may have different 3D position coordinates (x, y, z) depending on different input face images in the position map.

[0094] In some embodiments, as shown in Figures 2 and 3, a 3D reconstruction method is utilized to obtain an initial coarse position map (204, 304) from an input image (202, 302). Then, the input 2D keypoint annotations (208, 308) are used to adjust the (x, y) coordinates of the corresponding keypoints (206, 306) in the initial position map, ensuring that the adjusted (x, y) coordinates of the keypoints in the adjusted position map are the same as the annotated 2D keypoints. Specifically, first, a set of 96 keypoints is obtained from the initial position map P. Based on the keypoint index, the set of 96 keypoints is referred to as K=k_i, where each k_i is the 2D coordinate (x, y) of a keypoint, i=0,...,95. From the 2D keypoint annotations (208, 308), a second set of 96 keypoints A=a_i is obtained, which are 2D (x,y) coordinates, i=0,...,95. Then, a spatial transformation mapping (210, 310) from K to A is estimated, defined as T:Ω->Ω, where Ω⊂R^2. The obtained transformation T is then applied to the initial position map P to obtain a transformed position map P' (212, 312). In this way, the transformed position map P' (212, 312) preserves detailed facial features of the person in the input image (202, 302), and at the same time, the transformed position map P' (212, 312) is a reasonable 3D depth information. Therefore, the solution disclosed herein provides an accurate and practical alternative solution for generating 3D ground truth information, avoiding the use of expensive and time-consuming face capture systems.

[0095] In some embodiments, the 96 face key points cover only a portion of the entire face area (i.e., below the eyebrows, inside the face contour), so for example in FIG. 3, the ear-to-chin key point is along the jaw but not on the visible face contour. When the face in the input image is tilted, the entire face area is not covered by the contours of the key points that are connected together. In addition, when doing manual key point annotation, key points can only be labeled along the visible face contour (i.e., there is no way to accurately annotate occluded key points), regardless of whether the face in the image is tilted or not. As a result, in the transformed position map P'(212, 312), some of the face area does not have valid values ​​because the transformation mapping T(210, 310) has no estimation in the region. In addition, because the forehead area is above the eyebrows, T also has no estimation in that area. All these issues cause the transformed position map P'(212, 312) to not have valid values ​​in certain areas. FIG. 4 is a diagram illustrating an example transformed position map that does not cover the entire face area, according to some implementations of the present disclosure.

[0096] In FIG. 4, the top circle (402, 406) highlights the forehead area, while the right circle (404, 408) shows the area where the keypoint contour is smaller than the visible facial contour.

[0097] In some embodiments, to solve the above problems and make the algorithm robust to tilted faces that commonly exist in face images, a refinement process 214 is used as shown in Fig. 2. The key points from the transformed position map are shifted along the face contour to match the visible face contour based on the head pose and the coarse 3D face model. Then, the missing values ​​in the face contour area can be filled in the obtained position map. However, the values ​​in the forehead region are still missing. To cover the forehead region, the control points are extended by adding eight landmarks at the four corners of the image to both key point sets K and A.

[0098] 5 illustrates an example process for refining a transformed position map to cover the entire face area, according to some implementations of the present disclosure. The position map refinement process is illustrated in FIG.

[0099] In some embodiments, first, the head pose is determined based on the coarse position map P, and it is determined that the head is tilted left or right, with left or right being defined in the 3D face model space (e.g., the face is tilted left, as shown in FIG. 5). Based on the determination that the face is tilted left or right, the key points on the corresponding side of the face contour are adjusted. The key points on the right side of the face contour have indices from 1 to 8, and the key points on the left side of the face contour have indices from 10 to 17. Using a face tilted left as an example, a 2D projection of the initial position map P is calculated to obtain a depth map such as image 502 shown in FIG. 5. The key points k_i, i=10,...,17 of the left face contour are individually shifted rightward until they reach the boundary of the depth map. The new coordinates are then used to replace the original key point positions. Similarly, when the face is tilted right, the processed key points are indexed by k_i, i=1,...,8, and the search direction is left. After adjusting the face contour keypoints, the updated keypoints are visualized as in image 504 of Figure 5, and the updated coverage of the position map is displayed as in image 506 of Figure 5. The updated position map has better face coverage in the face contour area, but there are still missing values ​​in the forehead area.

[0100] In some embodiments, to cover the forehead area, two anchor points are added as additional keypoints k_i, i=96,...,103 at each corner of the image region Ω to obtain an updated keypoint set K' (as shown in image 508 of FIG. 5). The same is done for the manually annotated keypoint set a_i, i=96,...,103 to obtain an updated A'. Using the updated keypoint sets K' and A', the transformation mapping T' is re-estimated and then applied to the initial position map P to obtain a final position map P'' (216 in FIG. 2) that covers the entire face area (as shown in image 510 of FIG. 5). Final keypoints 218 are derived from the final position map 216.

[0101] 6 illustrates some example results of a position map refinement algorithm according to some implementations of the present disclosure. 602 illustrates the initial transformed position map. 604 illustrates the updated position map after fixing the face contour. 606 illustrates the final position map.

[0102] 7A and 7B show some example comparisons of a final location map against an initial coarse location map according to some implementations of the present disclosure. In one example of FIG. 7A, the nose in the initial location map and its associated 3D model and key points 702 is inaccurate in not fully reflecting the person's facial features (highlighted by the arrow), but after applying the methods described herein, the nose is well aligned with the image in the final location map and its associated 3D model and key points 704 (highlighted by the arrow). In the second example of FIG. 7B, there are multiple inaccuracies in the initial location map and its associated 3D model and key points 706, such as inconsistencies in the facial contour, open mouth and nose shape (indicated by the arrow). In the final location map and its associated 3D model and key points 708, all these errors are corrected (indicated by the arrow).

[0103] Hairstyle and eyeglasses classification is important for mobile gaming applications of face avatar creation process. In some embodiments, a multi-task learning and transfer learning based solution is implemented herein to solve these problems.

[0104] In some embodiments, four different classification tasks (head) are implemented for female hair prediction. The classification categories and parameters are listed below:

[0105] Classification head 1: Curve

[0106] Straight(0);Curve(1)

[0107] Classification Head 2: Length

[0108] Short(0);Long(1)

[0109] Category Head 3: Bangs

[0110] No bangs or split(0); Left part(1); Right part(2); M-shape(3); Straight bangs(4); Natural bangs(5); Air bangs(6)

[0111] Classification Head 4: Braid

[0112] single braid (0); two or more braids (1); single bun (2); two or more buns (3); other (4).

[0113] In some embodiments, three different classification tasks (head) are implemented for male hair prediction. The classification categories and parameters are listed below:

[0114] Category Head 1: Very short (0), Curly (1), Other (2)

[0115] Category Head 2: No bangs (0), split bang (1), natural bang (2)

[0116] Category Head 3: Left Split Bang (0) and Right Split Bang

[0117] In some embodiments, eyeglasses classification is a binary classification task. The classification parameters are:

[0118] Without glasses (0); with glasses (1).

[0119] Among different deep learning image classification models, those that achieve state-of-the-art accuracy on ImageNet usually have large model size and complex structure, such as EfficientNet, noisy student, and FixRes. When deciding which architecture should be used as the base network of the feature extractor, a balance must be made between prediction accuracy and model size. In practice, a 1% improvement in classification accuracy may not bring any obvious change to the end user, but the model size may increase exponentially. Given that the trained model needs to be deployed on the client side, the smaller base network can be flexible to be deployed on both the server side and the client side. Thus, for example, MobileNetV2 is adopted as a base network to perform transfer learning of different classification heads. The MobileNetV2 architecture is based on an inverted residual structure, where the input and output of the residual block are thin bottleneck layers, as opposed to traditional residual models that use augmented representations at the input. MobileNetV2 uses lightweight depthwise convolution to filter features in the intermediate augmentation layers.

[0120] A multi-task learning approach is used for eyeglasses classification. A binary classifier is trained using feature vectors with cross-entropy loss at the bottleneck layer of a U-shape based network by reusing the network of keypoint prediction as the base network and freezing the parameters. Figure 8A is a diagram illustrating an example eyeglasses classification network structure according to some implementations of the present disclosure. Figure 8B is a diagram illustrating an example female hair prediction network structure according to some implementations of the present disclosure. Figure 8C is a diagram illustrating an example male hair prediction network structure according to some implementations of the present disclosure.

[0121] 9A shows some example eyeglass classification prediction results according to some implementations of the present disclosure, FIG. 9B shows some example female hair prediction results according to some implementations of the present disclosure, and FIG. 9C shows some example male hair prediction results according to some implementations of the present disclosure.

[0122] FIG. 10 is a flowchart 1000 illustrating an example process of constructing a face location map from a 2D face image of a real person according to some implementations of the present disclosure. In the real world, different people have different facial characteristics, which may cause the same key points (e.g., the position of the eyebrows on the person's face) corresponding to the same facial characteristic to have very different spatial coordinates. The problem of face detection is made more challenging because the 2D face images used to generate the 3D face model are captured at different angles and under different lighting conditions, and research in this area has been a very active topic in the technical field of computer vision. In this application, several methods are proposed to improve the efficiency and accuracy of face key point detection from any 2D face image of objects ranging from real people to cartoon characters. In some embodiments, a user-provided set of face key points of the same face image is provided as a reference to correct or improve the set of face key points initially detected by the computer-implemented method. For example, since there is a one-to-one mapping relationship between user-provided and computer-generated facial keypoints based on their respective sequence numbers, the refinement of the computer-generated facial keypoints is defined as an optimization problem of reducing the difference between two sets of facial keypoints, e.g., as measured by their corresponding spatial coordinates in a position map.

[0123] The process of constructing a face position map may include a step 1010 of generating a coarse face position map from a 2D face image.

[0124] The process also includes a step 1020 of predicting a first set of keypoints in the 2D face image based on the coarse face location map.

[0125] The process further includes identifying 1030 a second set of keypoints in the 2D facial image based on user-provided keypoint annotations.

[0126] The process further includes a step 1040 of updating the coarse face position map to reduce the difference between the first set of key points in the 2D face image and the second set of key points. For example, by reducing the difference between the first set of key points in the 2D face image and the second set of key points with respect to the corresponding spatial coordinates, the first set of key points in the 2D face image based on the coarse face position map is modified to be more similar to the second set of key points in the 2D face image based on the user-provided key point annotations, which are often considered to be more accurate, and the modification of the first set of face key points automatically triggers an update of the initial coarse face position map from which the first set of key points is generated. The updated coarse face position map can then be used to predict a more accurate set of key points from the 2D face image. It is noted that the second set of key points in the 2D face image based on the user-provided key point annotations is not meant to be done manually. Instead, the user may adopt another computer-implemented method to perform the annotation. In some embodiments, the second set of keypoints is only a fraction of the number of keypoints (e.g., 10-20) as the first set of keypoints (e.g., 96 or more), but the fact that the second set of keypoints is much more accurate contributes to an overall improvement over the first set of keypoints.

[0127] In one implementation, the process further includes step 1050 of extracting a third set of keypoints as a final set of keypoints based on the updated face position map / final position map, where the third set of keypoints has the same position as the first set of keypoints in the face position map. In some embodiments, the positions of the keypoints in the face position map are represented by the 2D coordinates of the array elements in the position map. As mentioned above, the updated face position map benefits from the second set of keypoints in the 2D face image based on the user-provided keypoint annotations, and thus the third set of keypoints is more accurate and can be used in areas such as computer vision for more accurate face detection or computer graphics for more accurate 3D face modeling.

[0128] In one implementation, as an alternative or in addition to step 1050, the process further includes step 1060 of reconstructing a 3D face model of the real person based on the updated face position map. In one example, the 3D face model is a 3D depth model.

[0129] Additional implementations may include one or more of the following features.

[0130] In some embodiments, the updating step 1040 may include converting the coarse face location map into a transformed face location map and refining the transformed face location map. As described above, the transformed face location map may retain more detailed facial features of the person in the input image than the initial coarse face location map, and thus the 3D face model based on the transformed face location map is more accurate.

[0131] In some embodiments, the transforming includes estimating a transformation mapping from the coarse face position map to the transformed face position map from learning differences between the first set of keypoints and the second set of keypoints, and applying the transformation mapping to the coarse face position map.

[0132] In some embodiments, the refining includes adjusting key points corresponding to the transformed face position map on the occluded side of the face contour to cover the entire face area according to the determination that the 2D face image is tilted. As described above, different 2D face images may be captured at different angles, and this refining step can correct biases or errors introduced by different image capture conditions and retain a more accurate 3D face model of the 2D face image. Furthermore, the transformed face position map can retain more detailed facial features of the person in the input image than the initial coarse face position map, and thus the 3D face model based on the transformed face position map is more accurate.

[0133] In some embodiments, the first set of keypoints may include 96 keypoints.

[0134] In some embodiments, the process of constructing the face location map may include facial feature classification.

[0135] In some embodiments, facial feature classification is performed via deep learning methods.

[0136] In some embodiments, facial feature classification is performed via multi-task learning or transfer learning methods.

[0137] In some embodiments the facial feature classification includes a hair prediction classification.

[0138] In some embodiments, the hair prediction classification includes female hair prediction with multiple classification tasks that may include curve, length, bangs and braids.

[0139] In some embodiments, the hair prediction classification includes male hair prediction with multiple classification tasks that may include curve / length, bangs and hair split.

[0140] In some embodiments, the facial feature classification includes eyeglasses prediction classification, which includes classification tasks that may include with eyeglasses and without eyeglasses.

[0141] The method and system disclosed herein can generate accurate 3D face models (i.e., location maps) based on 2D keypoint annotations for 3D ground truth generation. This approach not only avoids the use of BFM and SFM face models, but also better preserves detailed facial features and prevents the loss of these important features caused by face model-based methods.

[0142] Besides providing keypoints, a deep learning based solution is used that provides complementary facial features such as hairstyle and glasses, which are essential to personalize a face avatar based on a user-input face image.

[0143] Although hairstyle and glasses prediction for facial feature classification is disclosed herein as an example, the framework is not limited to these exemplary tasks. The framework and solution are based on multi-task learning and transfer learning, which means that the framework is easy to extend to include other facial features such as female makeup type classification, male beard type classification, and mask presence / absence classification. The design of the framework is more suitable for extending to more tasks based on the requirements of various computer or mobile games.

[0144] In some embodiments, a keypoint-based light weighted color extraction method is introduced herein. The light weighted image processing algorithm estimates local pixels quickly without segmentation of every pixel, resulting in higher efficiency.

[0145] During the training process, the user only labels a few key points such as eye corners, mouth borders and eyebrows without the need to have pixel-level labels.

[0146] The light-weight color extraction method disclosed herein can be used in personalized face generation systems for various games. To provide more free personalized character generation, many games have begun to adopt free adjustment methods. In addition to adjusting the shape of the face, users can also select various color combinations. For aesthetic purposes, in-game faces often use predefined textures instead of real face textures. The method and system disclosed herein allows users to automatically extract the average color of each part of the face by only uploading a photo. At the same time, the system can automatically modify the texture according to the extracted color, so that each part of the personalized face is generated in a color closer to the actual color of the user photo, improving the user experience. For example, if the user's skin tone is darker than the average skin tone of most people, the skin tone of the in-game character will be darker accordingly. FIG. 11 is a flow diagram illustrating an example color extraction and adjustment process according to some implementations of the present disclosure.

[0147] In order to locate the different parts of the face, keypoints are defined for the main feature parts of the face, as shown in Fig. 1 above. The above algorithm is used for keypoint prediction. Unlike semantic segmentation methods, keypoints are only predicted in the image without the need to classify each pixel, which greatly reduces the cost of prediction and labeling training data. These keypoints allow the different parts of the face to be roughly located.

[0148] 12 illustrates an exemplary skin color extraction method according to some implementations of the present disclosure. To extract features in the image, the face area of ​​the original image 1202 needs to be rotated so that key points 1 and 17 on the left and right sides of the face are aligned with the corresponding key points on the left and right sides of a standard face.

[0149] Next, the area of ​​skin tone pixel inspection is determined: the lower coordinates of the eye keypoints are selected as the upper boundary of the detection area, the lower coordinates of the nose keypoint are selected as the lower boundary of the detection area, and the left and right boundaries are determined by the face boundary keypoints. Thus, the skin tone detection area is obtained, as shown in area 1208 of image 1206.

[0150] Not all pixels in this area 1208 are skin pixels, the pixels may also contain some eyelashes, nostrils, smile lines, hair, etc. Therefore, the median R,G,B value of all pixels in this area is selected as the final predicted average skin color.

[0151] FIG. 13 illustrates an exemplary eyebrow color extraction method according to some implementations of the present disclosure. For the average color of the eyebrows, first, the primary eyebrow, i.e., the eyebrow closer to the lens, is selected as the target. In some embodiments, if both eyebrows are the primary eyebrows, the eyebrow pixels on both sides are extracted. Assuming that the left eyebrow is the primary eyebrow, a quadrilateral area consisting of key points 77, 78, 81, and 82 is selected as the eyebrow pixel search area, as shown in FIG. 13. This is because the eyebrows closer to the outside are too thin, so the effect of small keypoint errors is large. Since the eyebrows closer to the inside are often sparse and mixed with the skin color, the central eyebrow area 1302 is selected to collect pixels. Then, each pixel needs to be compared with the average skin color first, and only pixels with differences larger than a certain threshold will be collected. Finally, similar to the skin color, the median R, G, B values ​​of the collected pixels are selected as the final average eyebrow color.

[0152] FIG. 14 shows an example pupil color extraction method according to some implementations of the present disclosure. Similar to eyebrow color extraction, when extracting pupil color, the side of the eye closest to the lens is selected first. In some embodiments, if both eyes are the primary eyes, the pixels from both sides are collected together. In addition to the pupil itself, the enclosed area contained inside the eye key points may also include eyelashes, whites of the eye, and reflections. These should be removed as much as possible in the process of pixel collection to ensure that most of the final pixels are derived from the pupil itself.

[0153] To remove eyelash pixels, the eye keypoints are shrunk inwards by a certain distance along the y-axis (vertical in FIG. 14) to form the area 1402 shown in FIG. 14. To remove eye whites and reflections (shown as circle 1404 in FIG. 14), this area 1402 further filters out such pixels. For example, if a pixel's R, G, and B values ​​are all greater than a predefined threshold, the pixel is filtered out. In this way, it can be guaranteed that the pixels collected are mostly from the pupil itself. Similarly, the median color is used as the average pupil color.

[0154] In some embodiments, for lip color extraction, only pixels in the lower lip area are detected. The upper lip is often thin and relatively sensitive to keypoint errors, and the upper lip is also light in color, which does not represent lip color well. Therefore, after rotating and rectifying the photo, all pixels in the area surrounded by the lower lip keypoints are collected, and the median color is used to represent the average lip color.

[0155] FIG. 15 shows an exemplary hair color extraction region used in a hair color extraction method according to some implementations of the present disclosure. Hair color extraction is more difficult than previous parts. The main reason is that each person's hairstyle is unique and the background of the photo is complex and diverse. Therefore, it is difficult to identify the hair pixels. One method to accurately find the hair pixels is to use a neural network to segment the hair pixels of the image. Since the annotation cost of image segmentation is very high and very high-precision color extraction is not required in game applications, a method based on approximate prediction of key points is used.

[0156] To obtain hair pixels, first, a detection area is determined. As shown in Fig. 15, the detection area 1502 is a rectangle. The lower boundary is the eyebrow corners on both sides, and the height (vertical line 1504) is the distance 1506 from the top of the eyebrow to the bottom of the eye. The left and right are key points 1, 17 that extend a certain distance to the left and right respectively. The hair pixel detection area 1502 thus obtained is shown in Fig. 15.

[0157] FIG. 16 shows an example separation between hair pixels and skin pixels in a hair color extraction region according to some implementations of the present disclosure. In general, the detection area includes three types of pixels: skin, hair, and background. In some more complicated cases, it also includes headwear. Since the left and right extent of the detection area is relatively conservative, it is assumed that the included hair pixels will be much more than the background pixels in most cases. Therefore, the main process is to separate the pixels of the detection area into hair or skin.

[0158] For each line of pixels in the detection area, the skin color change is often continuous, for example from light to dark, and there is often an obvious change at the junction of skin color and hair. Therefore, the center pixel of each row is selected as the starting point 1608, and skin pixels are detected for the left and right sides. First, a relatively conservative threshold is used to find a more reliable skin color pixel, and then it is extended to the left and right. If the color of the adjacent pixel is relatively close, it is also marked as skin color. Such a method takes into account the gradation of skin color, and can obtain a relatively accurate result. As shown in FIG. 16, in the hair color extraction region 1602, the darker regions such as 1604 represent skin color pixels, and the lighter regions such as 1606 represent hair color pixels. The median R, G, B value of the collected hair color pixels in the hair color region is selected as the final average hair color.

[0159] FIG. 17 shows an exemplary eyeshadow color extraction method according to some implementations of the present disclosure. Eyeshadow color extraction is a bit different from the previous parts because eyeshadow is a makeup that may or may not be present. Therefore, when extracting eyeshadow color, we first determine whether eyeshadow is present, and if so, extract its average color. Similar to eyebrow and eye pupil color extraction, eyeshadow color extraction is performed only on the main eye part close to the lens.

[0160] First, we need to determine which pixels belong to the eyeshadow. For the detection area of ​​the eyeshadow pixels, we use the area 1702 inside the lines 1704 and 1706 as shown in Figure 17. The left and right sides of the area 1702 are defined as the inner and outer corners of the eye, and the upper and lower sides of the area are the lower edge of the eyebrow and the upper edge of the eye. In addition to the possible eyeshadow pixels in this area 1702, there may also be eyelashes, eyebrows and skin that need to be excluded when extracting the eyeshadow.

[0161] In some embodiments, the top edge of the detection area is moved further down to eliminate the effect of eyebrows. To reduce the effect of eyelashes, pixels with brightness below a certain threshold are excluded. To distinguish eyeshadow from skin tone, the difference between the hue of each pixel and the average skin tone is checked. Only when the difference is greater than a certain threshold, the pixel is collected as a possible eyeshadow pixel. The reason for using hue instead of RGB value is that the average skin tone is mainly collected under the eye, while the skin tone above the eye may have a large change in brightness. Color is relatively stable because it is not sensitive to brightness. Therefore, hue is more suitable to determine whether a pixel is skin or not.

[0162] Through the above process, it can be determined whether the pixels in each detection area belong to eyeshadow. In some embodiments, if there is no eyeshadow, an error may occur and some pixels may still be recognized as eyeshadow.

[0163] To reduce the above errors, each column of the detection area is checked. If the number of eyeshadow pixels in the current column is greater than a certain threshold, the current column is marked as an eyeshadow column. If the ratio of eyeshadow columns to the width of the detection area is greater than a certain threshold, eyeshadow is deemed to be present in the current image, and the median color of the collected eyeshadow pixels is used as the final color. In this way, a few pixels incorrectly classified as eyeshadow will not cause a wrong decision for the entire eyeshadow.

[0164] Considering the art style, most games do not allow free adjustment of the colors of all the above parts. For parts with open color adjustment, they are often only allowed to match a set of defined colors. Take hair as an example, if a hairstyle allows five hair colors to be selected, the hairstyle in the resource pack will contain texture images corresponding to each hair color. As long as you select the texture image with the closest color according to the hair color prediction result during detection, you can get the desired hair rendering effect.

[0165] In some embodiments, when only one color texture image is provided, the color of the texture image can be reasonably modified according to any color detected. To facilitate color conversion, the commonly used RGB color space representation is converted to the HSV color model. The HSV color model is composed of three dimensions: hue H, saturation S, and lightness V. Hue H is represented in the model as a 360-degree color range, with red at 0 degrees, green at 120 degrees, and blue at 240 degrees. Saturation S represents the mixture of the spectrum color and white. The higher the saturation, the brighter the color. As the saturation approaches 0, the color approaches white. Lightness V represents the brightness of the color, with values ​​ranging from black to white. After color adjustment, the HSV median value of the texture image is expected to match the predicted color. Thus, the hue value calculation for each pixel can be expressed as: H i '=(H i +H'-H)%1, where H i ' and H i represents the hue of pixel i before and after the adjustment, and H and H′ represent the median hues of the texture images before and after the adjustment.

[0166] Unlike hue, which is a continuous space connected from end to end, saturation and lightness have boundary singularities such as 0 and 1. If a linear processing method similar to hue adjustment is used, when the median values ​​of the initial picture and the adjusted picture are close to 0 or 1, many pixel values ​​will have too high or too low saturation or lightness. This phenomenon will cause unnatural colors. To solve this problem, the following nonlinear curve is used to adapt the saturation and lightness before and after pixel adjustment:

[0167] y=1 / (1+(1-α)(1- x ) / (α x )), α∈(0,1).

[0168] In the above equation, x and y are the saturation or lightness values ​​before and after the adjustment, respectively. The only parameter left undetermined is α, which can be derived as follows:

[0169] α=1 / (1+x / (1-x)×(1-y) / y)

[0170] This formula can ensure that α is in the range from 0 to 1. Take saturation as an example, the initial median saturation S can be easily calculated based on the input picture. Then, the target saturation value S t can be obtained by hair color extraction and color space conversion. Therefore, α=1 / (1+ S / (1- S ) × (1-S t ) / S t ) for each pixel S in the default texture image. i For S, the adjusted value can then be calculated by the following formula: i '=1 / (1+(1-α)(1-S i ) / (α S i )). The same calculation applies to lightness.

[0171] In order to make the display effect of the adjusted texture picture closer to the real picture, special processing is applied to different parts. For example, to keep the low saturation of hair, S' = S' × V'^0.3 is set. Figure 18 shows some example color adjustment results according to some implementations of the disclosure. Column 1802 shows some default texture pictures provided by a particular game, column 1804 shows some texture pictures adjusted from the corresponding default texture picture in the same row according to the actual picture shown at the top of column 1804, and column 1806 shows some texture pictures adjusted from the corresponding default texture picture in the same row according to the actual picture shown at the top of column 1806.

[0172] FIG. 19 is a flowchart 1900 illustrating an example process for extracting color from a 2D facial image of a real person, according to some implementations of this disclosure.

[0173] The process of extracting color from a 2D facial image of a real person includes step 1910 of identifying a number of keypoints in the 2D facial image based on a keypoint prediction model.

[0174] The process also includes a step 1920 of rotating the 2D face image until the selected keypoints from the plurality of keypoints are aligned.

[0175] The process further includes a step 1930 of locating a plurality of features within the rotated 2D face image, each feature being defined by a respective subset of the plurality of key-points.

[0176] The process further includes extracting 1940 an average color from the pixel values ​​of the 2D face image for each of a number of features defined by a corresponding subset of the keypoints.

[0177] The process further includes a step 1950 of using the extracted colors of multiple features in the 2D facial image to generate a personalized 3D model of a real person that matches the respective facial feature colors of the 2D facial image.

[0178] Additional implementations may include one or more of the following features.

[0179] In some embodiments, the keypoint prediction model in identifying step 1910 is formed based on machine learning from keypoints that have been manually annotated by a user.

[0180] In some embodiments, the selected key points used for alignment in the rotating step 1920 are located on symmetrical left and right sides of the 2D face image.

[0181] In some embodiments, extracting the average color of each of the multiple parts in step 1940 may include selecting the median R, G, B values ​​of all pixels within each defined area in the corresponding part as the predicted average color.

[0182] In some embodiments, extracting an average color of each of the plurality of features in step 1940 may include determining an area for skin color extraction within the skin feature and selecting a median value of R, G, B values ​​of all pixels within the area for skin color extraction as a predicted average color of the skin feature. In some embodiments, the area for skin color extraction within the skin feature is determined as an area of ​​the face below the eyes and above the bottom end of the nose.

[0183] In some embodiments, extracting the average color of each of the multiple parts in step 1940 may include eyebrow color extraction in the eyebrow part, which may include selecting the eyebrow as a target eyebrow according to a determination that the eyebrow is on a side closer to a viewer of the 2D facial image, selecting both eyebrows as target eyebrows according to a determination that both eyebrows are equally closer to the viewer of the 2D facial image, extracting a middle eyebrow area of ​​the target eyebrows, comparing each pixel value in the middle eyebrow area with an average skin color, collecting pixels in the middle eyebrow area having a pixel value difference from the average skin color exceeding a threshold, and selecting a median R, G, B value of the collected pixels for eyebrow color extraction as a predicted average color of the eyebrow part.

[0184] In some embodiments, extracting the average color of each of the multiple parts in step 1940 may include pupil color extraction within the eye part, which may include selecting an eye as the target eye according to a determination that the eye is on a side closer to the viewer of the 2D facial image, selecting both eyes as the target eyes according to a determination that both eyes are equally closer to the viewer of the 2D facial image, extracting an area within the target eye that does not include eyelashes, comparing each pixel value within the extracted area with a predetermined threshold, collecting pixels within the extracted area having a pixel value difference that exceeds the predetermined threshold, and selecting a median R, G, B value of the collected pixels for pupil color extraction as a predicted average color of the pupil.

[0185] In some embodiments, extracting the average color of each of the multiple parts in step 1940 may include lip color extraction within the lip part, which may include collecting all pixels within an area surrounded by the lower lip keypoints and selecting a median of the R, G, B values ​​of the collected pixels for lip color extraction as the predicted average color of the lip part.

[0186] In some embodiments, extracting the average color of each of the multiple parts in step 1940 may include hair color extraction within the hair part, which may include identifying an area including a portion of the forehead extending into both hair parts, determining a pixel color change exceeding a predetermined threshold from the middle of the area to the left and right boundaries, dividing the area into a hair area and a skin area based on the pixel color change exceeding the predetermined threshold, and selecting a median R, G, B value of the pixels of the hair area within the area as the predicted average color of the hair part.

[0187] In some embodiments, an area including the forehead portion extending into the hair parts on either side is identified as a rectangular area having a lower boundary at the corners of both eyebrows, left and right boundaries at a fixed distance outward from key points located on the symmetrical left and right sides of the 2D face image, and a height the distance from the top of the eyebrows to the bottom of the eyes.

[0188] In some embodiments, extracting the average color of each of the plurality of parts in step 1940 may include extracting an eyeshadow color within the eyeshadow part, which may include selecting an eye as a target eye according to a determination that the eye is on a side closer to the viewer of the 2D face image, selecting both eyes as target eyes according to a determination that both eyes are equally closer to the viewer of the 2D face image, extracting a middle area within the eyeshadow part close to the target eye, collecting pixels within the extracted middle area having a brightness exceeding a predetermined brightness threshold and having a pixel hue value difference from an average skin hue value exceeding a predetermined threshold to exclude eyelashes, labeling a pixel column within the extracted middle area as an eyeshadow column according to a determination that a number of collected pixels within the pixel column exceeds a threshold, and selecting a median R, G, B value of the collected pixels for eyeshadow color extraction as a predicted eyeshadow color of the eyeshadow part according to a determination that a ratio of the eyeshadow column to a width of the extracted middle area exceeds a certain threshold.

[0189] In some embodiments, the process of extracting color from a 2D facial image of a real person may further include transforming the texture map based on the average color while preserving the original brightness and color differences of the texture map, which may include converting the average color from an RGB color space representation to an HSV (hue, saturation, lightness) color space representation, and adjusting the color of the texture map to reduce the difference between the median HSV value of the average color and the median HSV value pixel of the texture map.

[0190] The methods and systems disclosed herein can be used for applications in different scenarios, such as character modeling and game character generation. The lightweight methods can be flexibly applied to different devices, including mobile devices.

[0191] In some embodiments, the definition of face key points in the current system and method is not limited to the current definition, and other definitions are possible as long as the contours of each part can be fully expressed. In addition, in some embodiments, the colors directly returned in the scheme may not be used directly, but may be matched with a predefined color list to achieve further color screening and control.

[0192] Deformation methods that optimize the Laplacian operator require the mesh to be a differentiable manifold. However, in practice, meshes created by game artists often contain artifacts such as duplicated vertices and unprotected edges that can corrupt the manifold properties. Therefore, methods such as biharmonic deformations can only be used after careful cleaning up of the mesh. The affine deformation method proposed here does not use the Laplacian operator and therefore does not have such a strong constraint.

[0193] The family of deformation methods represented by the biharmonic deformation suffers from a lack of deformation capability in some cases. Harmonic functions that solve the Laplacian operator once often fail to achieve smooth results due to low smoothness requirements. Poly-harmonic functions that solve high-order (>=3) Laplacian operators fail on many meshes due to their high requirement of being at least 6th order differentiable. It is observed that in most cases, only biharmonic deformations that solve the Laplacian operator twice yield acceptable results. Nevertheless, due to the lack of tuning freedom, the deformation may still be unsatisfactory. The affine deformation proposed herein can achieve fine deformation tuning by changing the smoothness parameter, and its deformation result range covers the range when using biharmonic deformation.

[0194] FIG. 20 is a flow diagram showing an example head avatar deformation and generation process according to some implementations of the present disclosure. Using the technique proposed in the present disclosure, the head mesh can be deformed appropriately without binding with a skeleton. Therefore, the workload required from the artist is greatly reduced. The present technique accommodates different styles of meshes to obtain better generality. In the production of game assets, artists could save head models in various formats using tools such as 3DMax or Maya, but the internal representation of these formats is all polygon meshes. The polygon mesh can be easily converted to a pure triangle mesh, which is called a template model. For each template model, 3D key points are manually marked once on the template model. It can then be used to deform into a characteristic head avatar according to the 3D key points detected and reconstructed from any human face picture.

[0195] FIG. 21 is a diagram illustrating an example head template model configuration according to some implementations of the present disclosure. The head template model 2102 is typically composed of parts such as face 2110, eyes 2104, eyelashes 2106, teeth 2108, and hair, as shown in FIG. 21. Without binding a skeleton, mesh deformation relies on the connected structure of the template mesh. Therefore, the template model needs to be divided into these semantic parts, and the face mesh needs to be deformed first. By setting and following certain key points of the face mesh, all other parts can be adjusted automatically. In some embodiments, an interactive tool is provided to detect all topologically connected parts, which the user can use to conveniently export those semantic parts for further deformation.

[0196] In some embodiments, image keypoints of human faces can be obtained through some detection algorithms or AI models. These keypoints need to be mapped to vertices on a template model in order to drive mesh deformation. Due to the randomness of mesh connections and the lack of 3D human keypoint marking data, there are no tools that can automatically and accurately mark 3D keypoints on any head model. Therefore, interactive tools have been developed that can manually and quickly mark keypoints on 3D models. FIG. 22 illustrates some example keypoint markings on realistic style 3D models such as 2202, 2204 and cartoon style 3D models such as 2206, 2208 according to some implementations of the present disclosure.

[0197] In the marking procedure, the positions of the marked 3D keypoints on the 3D model should be maximally consistent with the picture keypoints. Since the keypoints are marked on the discrete vertices of the 3D model mesh, the import of deviations is inevitable. To offset such deviations, one way is to define appropriate rules in the pose processing. FIG. 23 illustrates an exemplary comparison between the template model rendering, the manually marked keypoints, and the AI-detected keypoints according to some implementations of the present disclosure. In some embodiments, the relatively realistically created These For the model, we apply keypoint detection and reconstruction algorithms to the template model. of rendering (2302) For example, the artificial intelligence 3D keypoint results (2306) can be further compared with the manually marked ones (2304), and thus the deviation between the two groups of keypoints is calculated. When detecting human pictures, the calculated deviation will be reduced from the detected keypoints in the real image, eliminating the negative effects of artificial markings.

[0198] The method of affine deformation disclosed herein is a keypoint-driven mathematical modeling that ultimately solves a system of linear equations. The method disclosed herein takes one step of deforming the template mesh using the detected keypoints as boundary conditions, and employs different constraints in the process of optimization. Figure 24 illustrates an example affine transformation of a triangle according to some implementations of the present disclosure.

[0199] In some embodiments, the transformation from the template mesh to the predicted mesh is considered as an assembly of affine transformations for each triangle. The affine transformation of a triangle can be defined as a 3×3 matrix T and a translation vector d. As shown in FIG. 24, the position of the transformed vertices after the affine transformation is v i '=Tv i+d, i∈1...4, where v1, v2, v3 respectively represent each vertex of the triangle, and v4 is an extra point introduced in the direction of the normal of the triangle, satisfying the equation v4=v1+(v2-v1)×(v3-v1) / sqrt(|(v2-v1)×(v3-v1)|). In the above equation, the cross product result is normalized to be proportional to the length of the triangle side. The reason for introducing v4 is that the coordinates of three vertices are not enough to determine a unique affine transformation. After introducing v4, the derivation equation is obtained: T=[v2'-v'1v'3-v'1v'4-v'1]×[v2-v1v3-v1v4-v1] -1 , the non-translation part of the matrix T is determined. Matrix V = [v2-v1v3-v1v4-v1] -1 Since depends only on the template mesh and is invariant to other deformation coefficients, it can be precomputed as a sparse coefficient matrix for later construction of the linear system.

[0200] So far we have shown the non-transformative part of the formula for the affine transformation T. To construct the linear system of optimization, assuming that the number of vertices of the mesh is N and the number of triangles is F, the following four constraints are taken into account:

[0201] Keypoint location constraints: E k =Σ i=1 ||v' i -c' i || 2 , c' i represents the position of the detected keypoints after mesh deformation.

[0202] Adjacency smoothness constraint: E s =Σ i=1 Σ j∈agj(i) ||T i -T j || 2This means that the affine transformations between adjacent triangles should be as similar as possible. The adjacency relationships can be pre-queried and stored to avoid duplicated calculations and improve the performance of the constructing system.

[0203] Attribute constraint: E i =Σ i=1 ||T i -I|| 2 , where I represents the identity matrix. This constraint means that the affine transformation should be as unchanged as possible, which helps to preserve the properties of the template mesh.

[0204] Original position constraint: E1 = Σ i=1 N||v'Ic i || 2 , where c i represents the position of each vertex on the template mesh before deformation.

[0205] The final constraint is the weighted sum of the above constraints: minE=w k E k +w s E s +w i E i +w1E1, where the weight w k , w s , w i , w1E1 are ranked from strongest to weakest. Using the above constraints, we can finally construct a linear system whose size is (F+N)×(F+N), and the weights are multiplied with the corresponding coefficients in the system. The unknowns are the coordinates of each vertex after deformation, besides the extra point v'4 of each triangle. The former terms are useful, so the result of v'4 is discarded. In the process of continuous deformation, all constraint matrices can be reused, except for the constraints on the keypoint positions. The affine transformation can achieve real-time performance of 30 fps on ordinary personal computers and intelligent phones for meshes with thousands of vertices.

[0206] FIG. 25 illustrates an example comparison of several head model deformation results with and without the blendshape process, according to some implementations of the present disclosure.

[0207] In some embodiments, when deforming a head model of a game avatar, the region of interest is usually only the face. The top, back and neck should be left unchanged, otherwise the mesh may penetrate between the head and the hair or torso. To avoid this problem, the result of the affine deformation and the template mesh are linearly interpolated in the blendshape manner. Weights for blending can be painted in the 3D modeling software or calculated with slight modifications in biharmonic or affine deformation. For example, the weights of key points are set to 1, while more markers (dark points in 2504 in FIG. 25) are added to the head model and their weights are set to 0. In some embodiments, inequality constraints are added in the solving process to force all weights to be in the range from 0 to 1, but doing so would significantly increase the complexity of the solution. Through experimentation, good results can be obtained by clipping weights that are smaller than 0 or larger than 1. As shown in 2504 of FIG. 25, the parts of the model with the darkest color have a weight of 1s, and parts of the model with no color have a weight of 0s. In blend weighted rendering 2504, there is a natural transition between the light keypoints and the dark markers. With blendshapes, the back side of the deformed model (shown in 2506 of FIG. 25) remains the same as the original model (shown in 2502 of FIG. 25). Without blendshapes, the back side of the deformed model (shown in 2508 of FIG. 25) does not remain the same as the original model (shown in 2502 of FIG. 25).

[0208] In some embodiments, the affine deformation can achieve different deformation effects by manipulating the weights of the constraints, including simulating the results of biharmonic deformation. Figure 26 is a diagram illustrating an example comparison of affine and biharmonic deformations with different weights, in accordance with some implementations of the present disclosure. As shown in Figure 26, the smoothness is calculated by the neighbor smoothness weight w s and characteristic weight w i The ratio of the smoothness of the deformed sphere to the smoothness of the original sphere is given by the smoothness of the neighboring smoothness weights. The dark dots are the keypoints, and the color intensity represents the displacement between the deformed position of the vertex and its original position. In every deformation result, one keypoint remains unchanged, while the other keypoint moves to the same position. This shows that by gradually increasing the neighbor smoothness weight relative to the feature weight, the smoothness of the deformed sphere also increases accordingly. In addition, the results of the biharmonic deformation can be consistent with those of the affine deformation, where the smoothness is somewhere between 10 and 100. This indicates that the affine deformation has more degrees of freedom in deformation compared to the biharmonic deformation.

[0209] Using the workflow described herein, games can easily integrate the functionality of intelligent generation of head avatars. For example, FIG. 27 shows some example results automatically generated from some randomly selected female pictures (not shown in FIG. 27) using a realistic template model according to some implementations of the present disclosure. Every personalized head avatar reflects some characteristics of the corresponding picture.

[0210] FIG. 28 is a flowchart 2800 illustrating an example process for generating a 3D head deformation model from a 2D face image of a real person, according to some implementations of this disclosure.

[0211] The process of generating a 3D head deformation model from a 2D face image includes step 2810 of receiving a two-dimensional (2D) face image.

[0212] The process also includes a step 2820 of identifying a first set of key points for the 2D facial image based on an artificial intelligence (AI) model, such as a convolutional neural network.

[0213] The process additionally includes a step 2830 of mapping the first set of keypoints to a second set of keypoints located on a plurality of vertices of a mesh of the 3D head template model based on a set of user-provided keypoint annotations placed on the 3D head template model.

[0214] The process additionally includes a step 2840 of performing a deformation on the mesh of the 3D head template model to obtain a deformed 3D head mesh model by reducing the difference between the first set of key points and the second set of key points. In some embodiments, there is a correspondence between the key points in the first set and the key points in the second set. After projecting the second set of key points into the same space as the first set of key points, a function is generated that measures the position difference between each of the first set of key points and the second set of key points. By performing a deformation on the mesh of the 3D head template model, the second set of key points in the space is optimized when a function that measures the position difference (e.g., position, adjacent smoothness, characteristics, etc.) between each of the first set of key points and the second set of key points is minimized.

[0215] The process additionally includes a step 2850 of applying a blendshape method to the deformed 3D head mesh model to obtain a personalized head model according to the 2D face image.

[0216] Additional implementations may include one or more of the following features.

[0217] In some embodiments, the mapping step 2830 may further include associating a first set of key points on the 2D face image with a plurality of vertices on a mesh of the 3D head template model, identifying a second set of key points based on a set of user-provided key point annotations on the plurality of vertices on the mesh of the 3D head template model, and mapping the first set of key points and the second set of key points based on corresponding identified features by each facial key point.

[0218] In some embodiments, performing 2840 the deformation may include deforming a mesh of the 3D head template model into a deformed 3D head mesh model by using a mapping from the first set of key points to the second set of key points, using deformation boundary conditions associated with the first set of key points.

[0219] In some embodiments, performing 2840 the deformation may further include applying different constraints in the process of deformation optimization including one or more of keypoint positions, neighborhood smoothness, properties and original positions.

[0220] In some embodiments, performing 2840 the deformation may further include applying a constraint to the deformation process that is a weighted sum of one or more of the keypoint positions, neighborhood smoothness, properties, and original positions.

[0221] In some embodiments, identifying 2820 the first set of keypoints includes using a convolutional neural network (CNN).

[0222] In some embodiments, the transformation comprises an affine transformation without a Laplacian operator, hi some embodiments, the affine transformation achieves transformation tuning by modifying a smoothness parameter.

[0223] In some embodiments, the mesh of the 3D head template model can be deformed without binding it to a skeleton. In some embodiments, the facial deformation model includes a realistic style model or a cartoon style model.

[0224] In some embodiments, applying the blend shape method to the deformed 3D head mesh model in step 2850 includes specifying respective blend weights for key points of the deformed 3D head mesh model depending on the position of the key point, and applying different levels of deformation to key points having different blend weights.

[0225] In some embodiments, applying the blendshape method to the deformed 3D head mesh model in step 2850 means keeping the back of the deformed 3D head mesh model the same shape as the original back shape of the 3D head template model before deformation.

[0226] In some embodiments, semantic parts on the template model are not limited to eyes, eyelashes, or teeth. Ornaments such as glasses can potentially be adaptively adjusted by adding and tracking new keypoints on the face mesh.

[0227] In some embodiments, the keypoints on the template model are added manually. In some other embodiments, deep learning techniques can be used to automatically add keypoints for different template models.

[0228] In some embodiments, the procedure for solving the affine transformations can utilize some numerical tricks to further improve its computational performance.

[0229] In some embodiments, the systems and methods disclosed herein form a lightweight keypoint-based facial avatar generation system that has many advantages, such as the following:

[0230] Low requirements on the input image: The system and method do not require the face to be directly facing the camera, and some degree of in-plane rotation, out-of-plane rotation and occlusion do not appreciably affect performance.

[0231] Applicability to both real games and cartoon games. This system is not limited to a real game style, but can be applied to a cartoon game style as well.

[0232] Lightweight and customizable. Each module of the system is relatively lightweight and suitable for mobile devices. The modules of the system are separated, and users can adopt different combinations according to different game styles to configure the final face generation system.

[0233] In some embodiments, for a given single photo, the main face is first detected and keypoint detection is performed. In real pictures, the face may not be facing the camera, and real faces are not always perfectly symmetrical. Therefore, the keypoints in the original image are pre-processed to achieve a uniform set of symmetric and smooth keypoints. The keypoints are then adjusted according to the game's specific style, such as enlarged eyes and thin face. After obtaining the stylized keypoints, the stylized keypoints are converted into control parameters of the face model in the game, typically bone parameters or slider parameters.

[0234] In some embodiments, the viewing angle of a real face may not be directly facing the camera, and problems such as asymmetry and keypoint detection errors may exist. Figure 29 illustrates an example keypoint processing flow step according to some implementations of the present disclosure. The detected keypoints from the original image 2904 cannot be used directly and certain processing is required. Here, the process is divided into three steps: normalization, symmetrization, and smoothing, as shown in Figure 29.

[0235] In some embodiments, it is necessary to adjust the in-game standard face model based on the prediction of the actual face key points. The process needs to ensure that the in-game standard face model and the actual face key points are aligned in terms of scale, position and orientation. Therefore, the normalization 2906 of the predicted key points and the key points on the game face model includes the following parts: scale normalization, translation normalization and angle normalization.

[0236] In some embodiments, all the 3D face keypoints of the original detection are defined as p, where the i-th keypoint is p i ={x i ,y i ,z i For example, the normalized origin is the midpoint between keypoints no. 1 and no. 17 (see keypoint definition in Figure 1), i.e., c = (p1 + p 17 ) / 2. For scale, the distance between the 1st and 17th keypoints from the origin is adjusted to 1, so that the scale and translation normalized 3D keypoints are p'=(pc) / ||p1-c||.

[0237] In some embodiments, after normalizing the scale and translation, the face orientation is further normalized. As shown in image 2902 of FIG. 29, a face in a real photo may not be facing directly into the lens, and there will always be a certain distortion, which may exist on the three coordinate axes. The predicted 3D keypoints of the face along the x, y and z coordinate axes are sequentially rotated so that the face orientation faces the camera. When rotating along x, the z coordinates of keypoints 18 and 24 (see keypoint definition in FIG. 1) are aligned, i.e., the depth of the top of the nose bridge is made the same as the depth of the nose base, resulting in a rotation matrix R X When rotating along the y-axis, we align the z-coordinates of keypoints 1 and 17 to obtain the rotation matrix R YWhen rotating along the z-axis, we align the y-coordinates of keypoints 1 and 17 to obtain the rotation matrix R Z Therefore, the keypoints whose orientations are aligned and normalized are given as follows: P norm =R Z ×R Y ×R X ×P'

[0238] In some embodiments, the scale, position and angle of the normalized keypoints are adjusted to be uniform, but the resulting keypoints are often not a complete face. For example, the bridge of the nose is not a straight line in the center, and facial features may not be symmetrical. This is because real faces in photos may not be perfectly symmetrical due to facial expressions or their own characteristics, which will introduce additional errors when predicting keypoints. Although real faces may not be symmetrical, if the in-game face model is not symmetrical, it will look bad and greatly reduce the user experience. Therefore, symmetry of keypoints as shown in 2908 is a necessary process.

[0239] Because the keypoints are normalized, in some embodiments, a simple symmetric method is to average the y and z coordinates of all symmetric keypoints and replace the original y and z coordinates. This method works well in most cases, but performance suffers when the face is rotated by a large angle on the y axis.

[0240] In some embodiments, using the human face in FIG. 29 as an example, when the face is deflected to the left at a large angle, part of the eyebrows will be invisible. At the same time, the left eye will be smaller than the right eye due to perspective. Although 3D key points can partially compensate for the effects caused by perspective, the 2D projection of the 3D key points corresponding to the key points still needs to be kept on the picture. Therefore, an excessively large angle deflection will cause obvious differences in the size of the eyes and eyebrows in the detection result of the 3D key points. To address the effects caused by the angle, when the face deflection angle along the y axis is large, the eyes and eyebrows close to the lens are used as the main eyes and main eyebrows, and are copied to the opposite side to reduce the error caused by the angle deflection.

[0241] In some embodiments, prediction errors of key points are inevitable, so that the symmetrized key points may not yet match the real face depending on individual cases. Because the shapes of real faces and facial features are quite different, it is difficult to achieve a relatively accurate description using predefined parameterized curves. Therefore, when smoothing as shown in 2910, only some areas such as the face contour, eyes, eyebrows, and lower lip are smoothed. These areas basically remain monotonous and smooth, i.e., there is no jagged state. In this case, the target curve should always be a convex curve or a concave curve.

[0242] In some embodiments, whether the keypoints meet the definition of a convex curve (or concave curve) is checked one by one for the relevant boundary. FIG. 30 illustrates an example keypoint smoothing process 2910 according to some implementations of the present disclosure. As shown in FIG. 30, without loss of generality, the target curve should be convex. For each keypoint 3002, 3004, 3006, 3008, and 3010, it is checked whether its position is on the line of the adjacent left and right keypoints. If the condition is met, it means that the current keypoint meets the requirement of a convex curve. If not, move the current keypoint to the line connecting the left and right keypoints. For example, in FIG. 30, keypoint 3006 does not meet the limit of a convex curve and is moved to position 3012. When moving multiple keypoints, the curve may not be guaranteed to be convex or concave after the movement. Therefore, in some embodiments, multiple smoothing is used to obtain a relatively smooth keypoint curve.

[0243] Different games have different face styles. In some embodiments, the key points of the real face need to be converted into the style required by the game. The faces of real-style games are similar, but the faces of cartoons are very different. Therefore, it is difficult to have a unified standard for the stylization of the key points. The definition of the stylization used in practice comes from the designer of the game, who adjusts the facial characteristics according to the specific game style.

[0244] In some embodiments, a more general face adjustment scheme is implemented that most games may require, such as face length adjustment, width adjustment, facial features, etc. Custom correction can be done according to different game art styles, adjustment levels, zoom ratios, etc. At the same time, users can also customize any special style adjustment method, such as changing the eye shape to a rectangle. The system can support any adjustment method.

[0245] In some embodiments, a standard game face is deformed with stylized facial key points so that the deformed facial key points reach the target key point positions. Since most games use control parameters such as bones and sliders to adjust the face, a set of control parameters is needed to move the key points to the target positions.

[0246] Since the definition of bones or sliders in different games may differ and modifications are possible at any time, it is not practical to directly define a simple parameterized function from keypoints to bone parameters. In some embodiments, machine learning methods are used to convert keypoints to parameters through a neural network called K2P (Keypoint-to-Parameter) network. Since the number of typical parameters and keypoints is not large (typically less than 100), in some embodiments, a K-layer fully connected network is used.

[0247] FIG. 31 is a block diagram illustrating an exemplary keypoint to control parameter (K2P) conversion process according to some implementations of the present disclosure. To use a machine learning method, in some embodiments, bone or slider parameters are first randomly sampled and fed to the game client 3110, and keypoints are extracted on the generated game face. In this way, a lot of training data can be obtained (parameter 3112 and keypoint 3114 pairs). Next, a self-supervised machine learning method is implemented, which is divided into two steps: the first step is to train a P2K (parameter-to-keypoint) network 3116 to simulate the process of generating keypoints from game parameters. In the second step, a large number of unlabeled real face images 3102 are used to generate real face keypoints 3104, and then a large number of stylized keypoints 3106 according to the method described herein. These unlabeled stylized keypoints 3106 are the self-supervised learning training data. In some embodiments, a set of keypoints K is input to the K2P network 3108 for learning to obtain the output parameters P. Since the ground truth of the ideal parameters corresponding to these keypoints is not available, P is further input to the P2K network 3116 trained in the first step to obtain the keypoint K'. In some embodiments, the K2P network 3108 can be trained by calculating the Mean Square Error (MSE) loss between K and K'. In some embodiments, during the second step, the P2K network 3116 is fixed and does not keep adjusting. With the help of the P2K network 3116, the process of controlling the parameters of the game client 3110 to the keypoints is simulated using a neural network, which lays the basis for the training of the K2P network 3108 in the second step.In this way, the final face generated by the parameters remains close to the key points of the generated target stylized face.

[0248] In some embodiments, at the same time, weights to certain keypoints, such as eye keypoints, are added by adjusting the corresponding weights when calculating the MSE loss between K and K'. The definition of the keypoints is predefined and is not affected by bones or sliders in the game client, making it easier to adjust the weights.

[0249] In some embodiments, in practical applications, neural networks can be trained separately for parts that can be separated to improve the accuracy of the model. For example, if some bone parameters only affect keypoints in the eye area, and other parameters do not affect this area, these parameters and this part of keypoints form an independent set of areas. A separate K2P model 3108 can be trained for each group of such areas, and each model can adopt a lighter network design. This not only further improves the accuracy of the model, but also reduces the amount of calculation.

[0250] FIG. 32 shows some example results of automatic face generation for mobile games according to some implementations of the present disclosure. As shown in FIG. 32, the results from original face images (3202 and 3206) to game face avatar image generation (3204 and 3208) are shown. In some embodiments, when stylizing, the open mouth is closed and different levels of constraints and cartoonization are applied to the nose, mouth, face shape, eyes and eyebrows. The final generated result still retains certain human face characteristics and meets the aesthetic requirements of the game style.

[0251] FIG. 33 is a flowchart 3300 illustrating an example process for customizing a standard face for an in-game avatar using a 2D facial image of a real human, according to some implementations of the disclosure.

[0252] The process of customizing a standard face for an in-game avatar using a 2D facial image of a real human includes step 3310 of identifying a set of real key points in the 2D facial image.

[0253] The process also includes a step 3320 of converting the set of real-world key-points into a set of game-style key-points that are associated with an avatar in the game.

[0254] The process additionally includes a step 3330 of generating a set of standard facial control parameters for an in-game avatar by applying the set of game style keypoints to a keypoint-to-parameter (K2P) neural network model. As described above in connection with FIG. 31, the K2P network 3108 is a deep learning neural network model that predicts a set of facial control parameters based on a set of input avatar keypoints, because different sets of avatar keypoints may correspond to different sets of facial control parameters, such that when the set of facial control parameters is applied to the standard face of the avatar, the adjusted standard facial keypoints may have a set of keypoints similar to the set of input avatar keypoints. As described above in connection with FIG. 31, in contrast to the K2P network 3108, the P2K network 3116 is a deep learning neural network model that predicts a set of avatar keypoints based on a set of input facial control parameters, because when the two neural network models are considered to perform the inverse process of each other, different sets of facial control parameters may result in different sets of avatar keypoints, such that the set of output avatar keypoints associated with the P2K network 3116 should match the set of input avatar keypoints associated with the K2P network 3108.

[0255] The process additionally includes a step 3340 of adjusting a plurality of facial features of the standard face by applying the set of facial control parameters to the standard face.

[0256] Additional implementations may include one or more of the following features.

[0257] In some embodiments, in step 3330, the K2P neural network model is trained by: obtaining a plurality of training 2D facial images of a real person; generating training game style keypoints for each of the plurality of training 2D facial images; feeding each set of training game style keypoints to the K2P neural network model to obtain a set of control parameters; feeding the set of control parameters to a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a set of predicted game style keypoints corresponding to the set of training game style keypoints; and updating the K2P neural network model by reducing the difference between the set of training game style keypoints and the corresponding set of predicted game style keypoints.

[0258] In some embodiments, the pre-trained P2K neural network model is configured to receive a set of control parameters, including bone or slider parameters, associated with an avatar in the game, and predict a set of game style key points for the avatar in the game according to the set of control parameters.

[0259] In some embodiments, the difference between the set of keypoints for the training game style and the corresponding set of keypoints for the predicted game style is the sum of the mean squared error between the set of keypoints for the training game style and the corresponding set of keypoints for the predicted game style.

[0260] In some embodiments, the trained K2P neural network model and the pre-trained P2K neural network model are game specific.

[0261] In some embodiments, the set of real key-points in the 2D face image corresponds to facial features of a real person in the 2D face image.

[0262] In some embodiments, the standard face of an in-game avatar can be customized according to facial images of different real-life people for different characters in the game.

[0263] In some embodiments, the deformed face of the avatar is a cartoon-style face of a real person. In some embodiments, the deformed face of the avatar is a realistic-style face of a real person.

[0264] In some embodiments, converting the set of real-world keypoints to a set of game-style keypoints in step 3320 includes normalizing the set of real-world keypoints to a canonical space, symmetrizing the normalized set of real-world keypoints, and adjusting the symmetrized set of real-world keypoints according to a predefined style associated with an avatar in the game.

[0265] In some embodiments, normalizing the set of real keypoints to the canonical space includes: scaling the set of real keypoints to the canonical space; and rotating the scaled set of real keypoints according to the orientation of the set of real keypoints in the 2D face image.

[0266] In some embodiments, converting the set of real-world keypoints into a set of game-style keypoints further comprises smoothing the set of symmetrized keypoints to meet a predetermined convex or concave curve requirement.

[0267] In some embodiments, adjusting the symmetrized set of real-world key points according to a predefined style associated with the in-game avatar includes one or more of a face length adjustment, a face width adjustment, a facial feature adjustment, a zoom adjustment, and an eye shape adjustment.

[0268] The system and method disclosed herein can be applied to various game automatic face generation systems for both realistic and cartoon style games. The system has an easy-to-integrate interface and improves user experience.

[0269] In some embodiments, the systems and methods disclosed herein can be used in 3D face avatar generation systems for various games to automate complex manual tuning processes to improve user experience. A user can take a selfie or upload an existing photo. The system can extract features from the face in the photo and automatically generate control parameters (such as bones or sliders) for the game's face through an AI face generation system. The game uses these parameters to generate a face avatar, so the created face has the user's facial features.

[0270] In some embodiments, the system can be easily customized for different games, including keypoint definition, stylization methods, skeleton / slider definition, etc. Users can choose to adjust only certain parameters, automatically retrain the model, or add custom control algorithms. In this way, the present invention can be easily deployed to a variety of games.

[0271] Further embodiments also include various subsets of the above embodiments combined or otherwise rearranged in various other embodiments.

[0272] Here, the image processing device of the embodiment of the present application is implemented in conjunction with the description of the accompanying drawings. The image processing device may be implemented in various forms, for example, in different types of computer devices, such as a server or a terminal (for example, a desktop computer, a notebook computer, or a smartphone). The hardware configuration of the image processing device of each embodiment of the present application is further described below. It can be understood that Fig. 34 shows only an exemplary structure, not all structures of the image processing device, and the partial or whole structure shown in Fig. 34 may be implemented according to requirements.

[0273] Referring to FIG. 34, FIG. 34 is a schematic diagram of an arbitrary hardware structure of an image processing device according to an embodiment of the present application, which may be applied to a server or various terminals running application programs in practical application. The image processing device 3400 shown in FIG. 34 includes at least one processor 3401, a memory 3402, a user interface 3403, and at least one network interface 3404. The components in the image processing device 3400 are coupled to each other by a bus system 3405. It can be understood that the bus 3405 is configured to implement connection and communication between the components. In addition to including a data bus, the bus system 3405 may further include a power bus, a control bus, and a status signal bus. However, for the sake of clear description, all buses are marked as a bus system 3405 in FIG. 34.

[0274] The user interface 3403 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad, a touch screen, or the like.

[0275] It may be appreciated that the memory 3402 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0276] The memory 3402, in an embodiment of the present application, is configured to store different types of data to support the operation of the image processing device 3400. Examples of data include any computer programs, such as an executable program 34021 and an operating system 34022 used to execute operations on the image processing device 3400, and a program used to execute the image processing method of an embodiment of the present application may be included in the executable program 34021.

[0277] The image processing method disclosed in the embodiment of the present application may be applied to or executed by the processor 3401. The processor 3401 may be an integrated circuit chip and has signal processing capabilities. In the implementation process, each step of the image processing method may be completed by using an integrated logic circuit of hardware in the processor 3401 or an instruction in software form. The aforementioned processor 3401 may be a general-purpose processor, a digital signal processor (DSP), another programmable logic device, a discrete gate, a transistor logic device, a discrete hardware component, etc. The processor 3401 may implement or execute the method, steps and logic block diagrams provided in the embodiment of the present application. The general-purpose processor may be a microprocessor, any conventional processor, etc. The steps of the method provided in the embodiment of the present application may be directly executed by a hardware decoding processor, or may be executed by combining hardware and software modules in the decoding processor. The software modules may be disposed in a storage medium. The storage medium is disposed in the memory 3402. The processor 3401 reads the information in the memory 3402 and combines the information with its hardware to perform the steps of the image processing method provided in the embodiments of the present application.

[0278] In some embodiments, image processing and 3D face and head generation can be realized on a cloud, on a group or network of servers.

[0279] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transferred via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. As such, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for certain of the implementations described in this application. A computer program product may include a computer-readable medium.

[0280] The terms used in the description of implementations herein are merely for the purpose of describing particular implementations and are not intended to limit the scope of the claims. When used in the description of implementations and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of the described features, elements and / or components, but do not preclude the presence or addition of one or more other features, elements, components and / or groups thereof.

[0281] It is also understood that although terms such as first, second, etc. may be used in this specification to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first electrode can be referred to as a second electrode, and similarly, a second electrode can be referred to as a first electrode, without departing from the scope of implementation. The first electrode and the second electrode are both electrodes, but are not the same electrode.

[0282] The description in this application has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the above descriptions and the associated drawings. The embodiments have been chosen and described in order to best explain the principles of the invention, practical applications and to enable those skilled in the art to understand the invention for various implementations and to best utilize the underlying principles and various implementations with various modifications as may be appropriate for the particular use contemplated. It is therefore to be understood that the claims are not limited to the specific implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. 1. A computer-implemented method for generating a three-dimensional (3D) head deformation model, comprising the steps of: receiving a two-dimensional (2D) face image; identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; modifying the first set of key points in the 2D rendering of the 3D head template model by applying previously calculated deviations between a set of previously AI-detected key points placed on the 3D head template model and a set of previously manually marked key points on the 3D head template model placed on a number of vertices of a mesh of the 3D head template model; based on a set of manually marked keypoint annotations placed on the 3D head template model, mapping the modified first set of keypoints to a second set of keypoints placed on the vertices of the mesh of the 3D head template model; performing a deformation on the mesh of the 3D head template model to reduce a difference between the modified first set of key points and the second set of key points to obtain a deformed 3D head mesh model; applying a blendshape method to the deformed 3D head mesh model to obtain a head model according to the 2D face image; A method comprising:

2. The mapping step comprises: associating the modified first set of key points on the 2D face image with the vertices of the mesh of the 3D head template model; mapping the modified first set of keypoints to the second set of keypoints based on corresponding features identified by each facial keypoint; The method of claim 1 further comprising:

3. performing the deformation includes deforming the mesh of the 3D head template model into the deformed 3D head mesh model by using a mapping of the modified first set of key points to the second set of key points and by using boundary conditions for deformation associated with the modified first set of key points. The method of claim 1.

4. performing the deformation further includes applying different constraints in the process of deformation optimization, including one or more of keypoint positions, neighborhood smoothness, characteristics, and original positions; The method according to claim 3.

5. performing the deformation further includes applying a constraint to the deformation process, the constraint being a weighted sum of one or more of keypoint positions, neighborhood smoothness, characteristics, and original positions; The method according to claim 3.

6. and identifying the first set of keypoints includes using a convolutional neural network (CNN). The method of claim 1.

7. The transformation includes an affine transformation without a Laplacian operator. The method of claim 1.

8. The affine transformation achieves transformation tuning by varying a smoothness parameter. The method of claim 7.

9. the mesh of the 3D head template model is deformed without binding it to a skeleton; The method of claim 1.

10. The 3D head deformation model includes a realistic style model or a cartoon style model. The method of claim 1.

11. applying a blendshape method to the deformed 3D head mesh model, assigning respective blending weights for each of the second set of key points of the deformed 3D head mesh model according to a position of the respective key point; applying different levels of transformation to the second set of keypoints with different blending weights; The method of claim 1 , comprising:

12. applying a blendshape method to the deformed 3D head mesh model, maintaining a shape of the rear of the deformed 3D head mesh model the same as an original shape of the rear of the 3D head template model before deformation. The method of claim 1.

13. 13. An electronic device comprising one or more processing units, a memory coupled to the one or more processing units, and a plurality of programs stored in the memory that, when executed by the one or more processing units, cause the electronic device to perform a method according to any one of claims 1 to 12.

14. 13. A computer program for execution by an electronic device having one or more processing units, the computer program, when executed by said one or more processing units, causing said electronic device to carry out a method according to any one of claims 1 to 12.

Images