Method, electronic device, and program for forming personalized 3D head model and face model

Abstract

The electronic device performs a method for customizing a standard face of an avatar using a two-dimensional (2D) facial image of a subject (e.g., a real person), the method including the steps of identifying a set of target keypoints in the 2D facial image, converting the set of target keypoints into a set of avatar keypoints associated with the avatar, generating a set of facial control parameters for the standard face by applying a keypoint-to-parameter (K2P) neural network model to the set of avatar keypoints, where each of the set of facial control parameters is associated with one of a plurality of facial features of the standard face, and adjusting the plurality of facial features of the standard face by applying the set of facial control parameters to the standard face, where the adjusted standard face of the avatar has the facial features of the 2D facial image of the subject.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is a continuation of and claims priority to U.S. Patent Application No. 17 / 202,121, filed on March 15, 2021, entitled "METHODS AND SYSTEMS FOR FORMING PERSONALIZED 3D HEAD AND FACIAL MODELS", the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure generally relates to imaging technology, and more particularly, to methods and systems for image processing and head / face model formation.

Background Art

[0003] Regardless of the presence or absence of explicit markers, commercially available face capture systems equipped with multiple sensors (e.g., multi - view cameras, depth sensors, etc.) are used to obtain an accurate three - dimensional (3D) face model of a person. These tools capture the geometric shape and texture information of a human face from multiple sensors and fuse the multimodal information into a general 3D face model. Benefiting from the multimodal information from various sensors, the acquired 3D face model is accurate. However, these commercial systems are expensive and require additional software purchases to process the raw data. Furthermore, these systems are typically deployed in a face capture studio and require actors or volunteers to obtain data, making the data collection process time - consuming and costly. In short, face capture systems are expensive and time - consuming for acquiring 3D face data. In contrast, smartphones or cameras are widely available today, so there is potentially a large amount of RGB (red, green, blue) images available. Capturing RGB images as input to generate a 3D face model can take advantage of the large amount of image data.

[0004] A 2D RGB image is just a projection of the 3D world onto a 2D plane. Restoring 3D shapes from 2D images is an ill-posed problem that requires optimization or learning algorithms to regularize the reconstruction process. For 3D face reconstruction, parameterized face models based on the 3D Morphable Model (3DMM) have been developed and used. In particular, face models such as the Basel Face Model (BFM) and the Surrey Face Model (SFM) are commonly used, and these require a commercial license. Face model-based methods are obtained based on a set of scanned 3D human face models (showing various facial features and expressions), and then generate a parameterized representation of facial features and expressions based on the 3D face model. A new 3D face can be represented as a linear combination of the base 3D face model based on parameterization. Due to the nature of these methods, the 3D face models used to form the base and parameter spaces limit the expressiveness of face model-based methods. Furthermore, the optimization process of fitting 3DMM parameters from the input face image or 2D landmarks further sacrifices the detailed facial features of the face image. Therefore, face model-based methods cannot accurately recover 3D facial features, and a commercial license is required to use face models such as BFM and SFM.

[0005] With the popularity of deep learning algorithms, semantic segmentation algorithms have received much attention. Such algorithms can divide each pixel in a face image into different categories such as background, skin, hair, eyes, nose, and mouth.

[0006] The semantic segmentation method can achieve relatively accurate results. However, semantic segmentation of all pixels is a very complex problem, often requiring a complex network structure, resulting in high computational complexity. Furthermore, to train a semantic segmentation network, a large amount of training data needs to be labeled. Semantic segmentation needs to divide the pixels of the entire image, which is very cumbersome, time-consuming, and costly. Therefore, it is not suitable for scenes that do not require high average color accuracy but do require high efficiency.

[0007] Key-point-driven deformation methods that optimize the Laplacian and other derived operators have been well studied in the academic community. The mathematical expression of biharmonic deformation can be represented as Δ 2 x’ = 0. Constrained key points, i.e., boundary conditions, can be represented as x b ’ = x bc In the above equation, Δ is the Laplacian operator, x’ is the position of the unknown deformed mesh vertex, and x bc is given the position of the key point after deformation. Solutions to the bi-Laplace equation are required in each dimension. Biharmonic functions are solutions to the bi-Laplace equation and are also minimizers of the so-called "Laplacian energy".

[0008] The property of energy minimization is mesh smoothing. When directly applying the minimizer described above, all detailed features are smoothed. Furthermore, when the positions of the key points do not change, the deformed mesh is expected to be exactly the same as the original mesh. Among these considerations, a preferred usage of biharmonic deformation is to solve for the displacement of vertices other than their positions. In this way, the deformed position can be written as x’ = x + d, where d is the unknown vertex displacement in each dimension. Naturally, the biharmonic deformation equation results in Δ b d = x bc - x b subject to Δ 2 d = 0, where d bIt is the displacement of the key points after deformation.

[0009] With the rapid development of the game industry, customized face avatar generation is becoming increasingly popular. For ordinary players without artistic skills, it is very difficult to adjust control parameters to generate a face that can describe subtle variations.

[0010] In some existing face generation systems and methods such as the Justice Face Generation System, the prediction of the face model is to predict 2D information in the image such as the segmentation of eyebrows, mouth, nose, and other pixels in the photo. These 2D segmentations are susceptible to out-of-plane rotation and partial occlusion, and basically require a frontal face. Furthermore, since the similarity between the final game face avatar and the input is determined by the face recognition system, this method is limited to only realistic-style games. If the style of the game is a cartoon style that is completely different from a real face, this method cannot be used.

[0011] In some other existing face generation systems and methods such as the Moonlight Blade Face Generation System, the actual face is reconstructed from the input image. This method is limited to realistic-style games and cannot be applied to cartoon-style games. Second, the output parameters of this method are the reconstructed game-style face mesh, and then template matching is performed on each part of the mesh. This approach limits the combination of different face parts. The overall diversity of game faces is closely related to the number of pre-generated templates. If a specific part such as the shape of the mouth has a small number of templates, it can hardly generate various variations and there is no diversity in the generated faces.

Summary of the Invention

Means for Solving the Problems

[0012] Learning-based face reconstruction and keypoint detection methods rely on 3D ground truth data as the gold standard for training models as close as possible to ground truth. Therefore, 3D ground truth determines the upper limit of learning-based approaches. To ensure the accuracy of face reconstruction and desirable keypoint detection, in some embodiments, 2D face keypoint annotations are used to generate the ground truth of the 3D face model without using expensive face capture systems. The techniques disclosed herein preserve the detailed face features of the input image, overcome the drawbacks of existing face models such as 3DMM-based methods that lose face features, and avoid the use of parameterized face models such as BFM and SFM (both of which require commercial licenses) required by some existing face model-based methods to generate a 3D ground truth face model.

[0013] Separate from face keypoint detection, in some embodiments, multi-task learning and transfer learning solutions are implemented for the face feature classification task, and as a result, more information can be extracted from the input face image that complements the keypoint information. The detected face keypoints with predicted face features are valuable for computer or mobile games for creating an avatar of the player's face.

[0014] In some embodiments, a lightweight method for extracting the average color of each part of a human face from a single photograph including the average color of skin, eyebrows, pupils, lips, hair, and eyeshadow is disclosed herein. At the same time, an algorithm for automatically converting the texture map based on the average color is also used, and as a result, the converted texture still has the original brightness and color differences, but the main color becomes the target color.

[0015] With the rapid development of computer vision and artificial intelligence (AI) technologies, the capture and reconstruction of 3D human face key points have achieved a high-precision level. An increasing number of games are using AI detection to make game characters more vivid. The methods and systems disclosed in this specification customize 3D head avatars based on the reconstructed 3D key points. General key point-driven deformation is applicable to any mesh. The process and deformation methods for customizing head avatars proposed in this specification can find their applications in scenarios such as automatic avatar creation and expression reproduction.

[0016] Methods and systems for automatically generating face avatars in games based on a single photo are disclosed in this specification. Through the prediction of face key points, the automatic processing of key points, and the use of deep learning methods for predicting model parameters, the systems disclosed in this specification can automatically generate face avatars in games to 1) have the characteristics of the actual face in the photo and 2) conform to the target game style. This system can be applied simultaneously to the face generation of realistic-style games and cartoon-style games and can be easily adjusted automatically according to different game model or bone definitions.

[0017] According to a first aspect of the present application, a method for constructing a face position map from a 2D face image of an object includes generating a rough face position map from the 2D face image, predicting a first set of key points in the 2D face image based on the rough face position map, identifying a second set of key points in the 2D face image based on user-provided key point annotations, and updating the rough face position map so as to reduce the difference between the first set of key points and the second set of key points in the 2D face image.

[0018] In some embodiments, a method of constructing a face position map from a 2D face image of a real person further includes the step of extracting, as a final set of key points, a third set of key points based on the updated face position map, wherein the third set of key points has the same positions as the first set of key points in the face position map.

[0019] In some embodiments, a method of constructing a face position map from a 2D face image of a real person further includes the step of reconstructing a three-dimensional (3D) face model of the real person based on the updated face position map.

[0020] According to a second aspect of the present application, a method of extracting color from a two-dimensional (2D) face image of an object includes identifying a plurality of key points in the 2D face image based on a key point prediction model; rotating the 2D face image until a plurality of target key points from the identified plurality of key points are aligned with corresponding target key points of a standard face; identifying a plurality of portions in the rotated 2D face image, each portion being defined by a respective subset of the identified plurality of key points; extracting color for each of the plurality of portions defined by the corresponding subset of key points from the pixel values of the 2D face image; and generating a three-dimensional (3D) model of the object that matches each face feature color of the 2D face image using the colors extracted from the plurality of portions in the 2D face image.

[0021] According to a third aspect of the present application, a method for generating a three-dimensional (3D) head deformation model includes receiving a two-dimensional (2D) face image, identifying a first set of key points in the 2D face image based on an artificial intelligence (AI) model, mapping the first set of key points to a second set of key points based on a set of user-provided key point annotations located at a plurality of vertices of a mesh of a 3D head template model, performing a deformation on the mesh of the 3D head template model to obtain a deformed 3D head mesh model by reducing the difference between the first set of key points and the second set of key points, and applying a blend shape method to the deformed 3D head mesh model to obtain a personalized head model according to the 2D face image.

[0022] According to a fourth aspect of the present application, a method for customizing a standard face of an avatar using a two-dimensional (2D) face image of a subject is provided. The method includes identifying a set of target key points in the 2D face image, converting the set of target key points to a set of avatar key points associated with the avatar, generating a set of face control parameters of the standard face by applying a key point-to-parameter (K2P) neural network model to the set of avatar key points, wherein each of the set of face control parameters is associated with one of a plurality of face features of the standard face, and adjusting the plurality of face features of the standard face by applying the set of face control parameters to the standard face, wherein the adjusted standard face of the avatar has the face features of the 2D face image of the subject.

[0023] According to a fifth aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When the programs are executed by the one or more processing units, the electronic device is caused to execute the one or more methods described above.

[0024] According to a sixth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. When the programs are executed by the one or more processing units, the electronic device is caused to execute the one or more methods described above.

[0025] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein do not include all of them. In particular, considering the drawings, the specification, and the claims, many additional features and advantages will be apparent to those skilled in the art. Further, it should be noted that the language used herein is mainly selected for readability and for the purpose of explanation, and may not be selected to describe or limit the subject matter of the present invention.

[0026] For a more detailed understanding of the present disclosure, a more detailed description can be made by referring to the features of the various embodiments, some of which are shown in the accompanying drawings. However, the accompanying drawings only show the appropriate features of the present disclosure and should not be regarded as limiting, because its description may recognize other valid features.

Brief Description of the Drawings

[0027]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 8C

Figure 9A

Figure 9B

Figure 9C

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

[0028] According to general convention, various features shown in the drawings may not be drawn to scale. Accordingly, the dimensions of various features may be arbitrarily enlarged or reduced for clarity. Further, some of the drawings may not show all of the components of a given system, method, or apparatus. Finally, like reference numerals may be used throughout the specification and drawings to indicate like features.

Best Mode for Carrying Out the Invention

[0029] Here, reference is made in detail to specific embodiments in which examples are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to facilitate an understanding of the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives can be used without departing from the scope of the claims, and that the subject matter can be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices.

[0030] Before further elaborating on the embodiments of the present application, the names and terms related to the embodiments of the present application are explained, and the names and terms related to the embodiments of the present application have the following explanations.

[0031] Facial key points: Predetermined landmarks that define specific facial parts, such as the corners of the eyes, the jaw, the tip of the nose, and the shape of the mouth corners.

[0032] Facial parts: The boundaries of the face, eyes, eyebrows, nose, mouth, and other parts.

[0033] Facial reconstruction: Reconstructing the 3D geometric structure of a human face and commonly used representations including mesh models, point clouds, or depth maps.

[0034] RGB image: An image format with three channels of red, green, and blue.

[0035] Position map: Storing the x, y, and z coordinates of the face region, which is a 3D human face representation, using the red, green, and blue channels of a normal image format.

[0036] Facial feature classification: Including hair style classification regardless of the presence or absence of glasses classification.

[0037] Convolutional neural network (CNN): A type of deep neural network most commonly applied to the analysis of visual images.

[0038] Base network: A network such as a CNN that is used by one or more downstream tasks to function as a feature extractor.

[0039] Laplacian operator: A differential operator given by the divergence of the gradient of a function on Euclidean space.

[0040] Differentiable manifold: A type of topological space that is locally similar to a linear space to enable computations.

[0041] Biharmonic function: A fourth-differentiable function with a bi-Laplacian operator, defined on a differentiable manifold and equal to 0.

[0042] Keypoint-driven deformation: A method of deforming a mesh by changing the positions of specific vertices.

[0043] Biharmonic deformation: A deformation method that uses the optimization of a biharmonic function with some boundary conditions.

[0044] Affine deformation: A keypoint-driven deformation method proposed in the present disclosure that optimizes the affine transformation of triangles to achieve the purpose of mesh deformation.

[0045] Face model: A mesh of a standard face in a predetermined target game.

[0046] Bone / slider: A control parameter for deforming a face model.

[0047] As described above, even when both the input 2D image and 2D keypoints are supplied to the optimization process to fit the 3DMM parameters, the optimization must balance between the fitting of the 3D face model based on the basis (i.e., 3D face model set) and the fidelity of the 2D keypoints. This optimization leads to the obtained 3D face model not following the 2D input keypoints, and as a result, the detailed face information brought by the input 2D keypoints is sacrificed. Among existing 3D face reconstruction methods, face capture solutions can generate accurate reconstructions, but they are expensive and time-consuming, and the obtained data also shows limited variations of facial features (a limited number of actors). On the other hand, face model-based methods can obtain 2D images or 2D landmark annotations as inputs, but the obtained 3D models are not accurate. To meet the requirements of the rapid development of computer / mobile games, both generating the desired 3D model accuracy and reducing the required cost and time are necessary. To meet these requirements, the new 3D ground truth face model generation algorithm disclosed in this specification takes as inputs a 2D image, 2D keypoint annotation, and a rough 3D face model (in the form of a position map), converts the rough 3D model based on the 2D keypoints, and finally generates a 3D face model with detailed facial features sufficiently preserved.

[0048] In addition to solving important problems in face reconstruction and keypoint prediction, multi-task learning and transfer learning-based approaches for face feature classification are also disclosed herein and are partially built on top of the face reconstruction and keypoint prediction frameworks. In particular, by reusing the base network for face reconstruction and keypoint prediction, glasses classification (with or without glasses) is achieved by multi-task learning. A linear classifier on top of an existing face reconstruction and keypoint prediction framework is trained, significantly reusing the existing model and avoiding the introduction of another larger network for image feature extraction. Further, another shared base network is used for male and female hairstyle classification. Hairstyle is a type of important face feature that complements face keypoints or 3D face models. In the process of creating a user's 3D avatar, adding hairstyle and glasses prediction can better reflect the user's face features and provide a better personalized experience.

[0049] Face keypoint prediction has been a research topic in computer vision for decades. With the development of artificial intelligence and deep learning in recent years, convolutional neural networks (CNNs) have facilitated the progress of face keypoint prediction. 3D face reconstruction and face keypoint detection are two intertwined problems, and solving one can simplify the other. Traditional methods are to first solve 2D face keypoint detection and then further infer a 3D face model based on the estimated 2D face keypoints. However, if the face in the image is tilted (nodding or shaking the head), certain face keypoints are blocked, leading to incorrect 2D face keypoint estimation, and thus the 3D face model built on top of the incorrect 2D face keypoints will be inaccurate.

[0050] Since ground truth data determines the upper limit of deep learning-based methods, existing 3D face model datasets are not only limited in number but also only available for academic research. On the other hand, for face model-based methods, both need to use the Basel Face Model (BFM) or Surrey Face Model (SFM) which require commercial licenses. High-precision and large amounts of 3D ground truth are the most important issues when training face reconstruction or keypoint estimation models.

[0051] In addition to face keypoint prediction, face feature classification is an important aspect of user 3D avatar creation. With the predicted face keypoints, only style transfer of the user's face parts (i.e., eyes, eyebrows, nose, mouth, and face contour) can be performed. However, to better reflect the user's face features, it is very helpful to match the user's hairstyle and add glasses if the user is wearing glasses in the input image. Based on these requirements, multi-task learning and transfer learning-based face feature classification methods have been developed to achieve male / female hairstyle prediction and glasses prediction (present or absent), thereby making the created face avatar more personalized and improving the user experience.

[0052] In some embodiments, to represent the three-dimensional shape of the main part of the face, keypoint representation is used as shown in FIG. 1. FIG. 1 is a diagram showing exemplary keypoint definitions according to some embodiments of the present disclosure. In other words, there is a mapping relationship between the sequence number of the keypoint and a specific position on the face. For example, sequence number 9 corresponds to the lower part of the jaw, and sequence number 21 corresponds to the tip of the nose, and so on. Keypoints are numbered in a sequence that defines specific features of the face. Keypoints focus on the boundaries of the main parts of the face, for example, the contour of the face, the contour of the eyes, and the contour of the eyebrows. The more keypoints there are, the more difficult it is to make predictions, but the shape representation becomes more accurate. In some embodiments, the definition of 96 keypoints is adopted in FIG. 1. In some embodiments, the user can modify the specific definition and the number of keypoints according to their own needs.

[0053] Many algorithms can predict the three-dimensional coordinates of the keypoints of the human face. A method with better performance uses a deep learning algorithm based on a large amount of offline 3D training data. However, in some embodiments, any three-dimensional keypoint prediction algorithm can be used. In some embodiments, the definition of the keypoints is not fixed, and the user can customize the definition as needed.

[0054] To solve the problem of 3D ground truth face model generation, the following automatic algorithm is developed, which takes a 2D RGB image, 2D keypoint annotation, and a coarse position map as inputs. FIG. 2 is a block diagram showing an exemplary keypoint generation process according to some embodiments of the present disclosure. For example, a 2D RGB image of the face is used as the input image 202, and the 2D RGB image has a corresponding initial coarse position map 204, and each pixel in the initial coarse map represents the spatial coordinates of the corresponding face point in the 2D RGB image. The 2D keypoint annotation 208 represents a set of user-provided keypoints that are used to correct the set of keypoints 206 detected from the initial coarse map 204.

[0055] FIG. 3 is a diagram showing an exemplary process for converting an initial rough position map according to some embodiments of the present disclosure.

[0056] In some embodiments, a 3D reconstruction method is used to convert an input face image into a position map including 3D depth information of face features. For example, the position map may be a 2D three-color (RGB) channel map having a 256×256 matrix array, and each array element has coordinates (x, y, z) representing a 3D position on the face model. The 3D position coordinates (x, y, z) are represented by RGB pixel values on the position map of each array element. Specific face features are arranged at fixed 2D positions within the 2D position map. For example, the tip of the nose can be identified by the 2D array element position at X = 128 and Y = 128 within the position map. Similarly, specific keypoints identified for specific face features of a face can be arranged at the same array element positions on the 2D position map. However, the specific keypoints can have different 3D position coordinates (x, y, z) depending on different input face images of the position map.

[0057] In some embodiments, as shown in FIGS. 2 and 3, an initial rough position map (204, 304) is obtained from the input images (202, 302) using a 3D reconstruction method. Then, the (x, y) coordinates of the corresponding keypoints (206, 306) of the initial position map are adjusted using the input 2D keypoint annotations (208, 308) such that the adjusted (x, y) coordinates of the keypoints in the adjusted position map are the same as the annotated 2D keypoints. In particular, first, a set of 96 keypoints is obtained from the initial position map P. Based on the keypoint index, the set of 96 keypoints is called K = k_i, where each k_i is the 2D coordinates (x, y) of a keypoint and i = 0, ..., 95. From the 2D keypoint annotations (208, 308), a second set of 96 keypoints A = a_i that are 2D (x, y) coordinates is obtained, where i = 0, ..., 95. Second, a spatial transformation mapping (210, 310) is estimated from K to A, defined as T: Ω → Ω, where Ω ⊂ R^2. Then, the obtained transformation T is applied to the initial position map P to obtain a transformed position map P' (212, 312). In this way, the transformed position map P' (212, 312) preserves the detailed face features of the person in the input images (202, 302), and at the same time, the transformed position map P' (212, 312) has valid 3D depth information. Therefore, the solution disclosed herein provides an accurate and practical alternative solution for generating 3D ground truth information to avoid the use of expensive and time-consuming face capture systems.

[0058] In some embodiments, the 96 facial key points, for example, in FIG. 3, only cover the part of the entire face region (i.e., below the eyebrows and inside the facial contour), so the key points from the ear to the jaw are along the lower part of the jaw but not along the visible facial contour. When the face in the input image is tilted, the entire face region is not covered by the contour of the connected key points. In addition, when performing manual key point annotation, whether the face in the image is tilted or not, the key points can only be labeled along the visible face contour (i.e., there is no way to accurately annotate the occluded key points). As a result, in the transformed position map P’(212,312), a part of the face region does not have valid values by the transformation mapping T(210,310), and no estimation is performed in that region. Also, since the forehead region is above the eyebrows, T has no estimation in that region either. All of these problems contribute to the fact that the transformed position map P’(212,312) does not have valid values in specific regions. FIG. 4 is a diagram showing an exemplary transformed position map that does not cover the entire face region according to some embodiments of the present disclosure.

[0059] In FIG. 4, the upper circles (402, 406) highlight the forehead region, and the right circles (404, 408) show the region where the contour of the key points is smaller than the visible face contour.

[0060] In some embodiments, to solve the above problems and make the algorithm robust for tilted faces commonly present in facial images, a refinement process 214 as shown in FIG. 2 is used. The key points from the transformed position map are shifted along the facial contour and match the visible face contour based on the head pose and the rough 3D face model. Then, in the obtained position map, the missing values in the face contour region can be filled. However, the values in the forehead region are still missing. To cover the forehead region, the control points are extended by adding the 8 landmarks at the four corners of the image to both key point sets K and A.

[0061] FIG. 5 is a diagram showing an exemplary process for refining a position map converted to cover the entire face region according to some embodiments of the present disclosure. The position map refinement process is shown in FIG. 5.

[0062] In some embodiments, the head pose is first determined based on the coarse position map P to determine whether the head is tilted to the left or right, where left or right is defined within the 3D face model space (e.g., as shown in FIG. 5, the face is titled to the left). Based on the determination that the face is tilted to the left or right, the key points on the corresponding side of the face contour are adjusted. The key points on the right side of the face contour have indices from 1 to 8, and the key points on the left side of the face contour have indices from 10 to 17. As an example, using a face tilted to the left, a 2D projection of the initial position map P is calculated to obtain a depth map as image 502 shown in FIG. 5. The left face contour key points k_i, i = 10,..., 17 are individually shifted to the right until they reach the boundary of the depth map. Then, the original key point positions are replaced with the new coordinates. Similarly, when the face is tilted to the right, the processed key points are indexed by k_i, i = 1,..., 8, and the search direction is to the left. After adjusting the face contour key points, the updated key points are visualized as image 504 in FIG. 5, and the updated coverage of the position map is shown as image 506 in FIG. 5. The updated position map has better coverage of the face in the contour region of the face, but the forehead region still has missing values.

[0063] In some embodiments, to cover the frontal region, two anchor points are added at each corner of the image domain Ω as additional key points \(k_i\), \(i = 96,\ldots,103\) (as shown in image 508 of FIG. 5) to obtain an updated key point set \(K'\). To obtain the updated \(A'\), the same is done for the manual annotation key point set \(a_i\), \(i = 96,\ldots,103\). Using the updated key point sets \(K'\) and \(A'\), the transformation mapping \(T'\) is re-estimated and then applied to the initial position map \(P\) to obtain a final position map \(P''\) (216 in FIG. 2) that covers the entire face region (as shown in image 510 of FIG. 5). The final key points 218 are derived from the final position map 216.

[0064] FIG. 6 is a diagram showing some exemplary results of a position map refinement algorithm according to some embodiments of the present disclosure. 602 is a diagram of an initial transformed position map. 604 is a diagram of an updated position map after fixing the face contour. 606 is a diagram of the final position map.

[0065] FIGS. 7A and 7B show some exemplary comparisons of the final position map with the initial rough position map according to some embodiments of the present disclosure. In one example of FIG. 7A, the nose of the initial position map and its associated 3D model and key point 702 are incorrect and cannot fully reflect the human face features (highlighted by the arrow), but after applying the method described herein, the nose is well-aligned with the final position map image and its associated 3D model and key point 704 (highlighted by the arrow). In a second example of FIG. 7B, there are multiple inaccuracies in the initial position map and its associated 3D model, as well as in key points 706 such as mismatches in the face contour, opening mouth, and nose shape (indicated by the arrow). In the final position map and its associated 3D model and key point 708, all these errors are fixed (indicated by the arrow).

[0066] Hair style and glasses classification are important for a mobile game application for face avatar creation process. In some embodiments, to address these issues, multi-task learning and transfer learning based solutions are implemented herein.

[0067] In some embodiments, four different classification tasks (head) are implemented for female hair prediction. The classification categories and parameters are shown below. Classification head 1: Curve Straight (0), Curve (1) Classification head 2: Length Short (0), Long (1) Classification head 3: Bangs No bangs or splits (0), Left split (1), Right split (2), M-shaped (3), Straight bangs (4), Natural bangs (5), Air bangs (6) Classification head 4: Blade Single blade (0), Two or more blades (1), Single bun (2), Two or more buns (3), Others (4).

[0068] In some embodiments, three different classification tasks (head) are implemented for male hair prediction. The classification categories and parameters are shown below. Classification head 1: Extreme short (0), Curly (1), Others (2) Classification head 2: No bangs (0), Split bangs (1), Natural bangs (2) Classification head 3: Split bangs left (0), and Split bangs right (1)

[0069] In some embodiments, glasses classification is a binary classification task. The classification parameters are shown below. No glasses (0), Glasses (1) present.

[0070] Among various deep learning image classification models, those achieving state-of-the-art accuracy in ImageNet usually have a large model size and complex structures such as EfficientNet, Noisy Student, and FixRes. When determining which architecture to use as the base network of the feature extractor, it is necessary to balance both the prediction accuracy and the model size. In fact, a 1% improvement in classification accuracy may not bring obvious changes to end users, but the model size may increase exponentially. If the trained model needs to be deployed on the client side, a smaller base network can be made more flexible to be deployed on both the server side and the client side. Therefore, for example, MobileNetV2 is adopted as the base network for performing transfer learning for different classification heads. The MobileNetV2 architecture is based on an inverted residual structure, and the input and output of the residual block are thin bottleneck layers opposite to the conventional residual model that uses an expansion representation for the input. MobileNetV2 uses lightweight depthwise convolutions to filter the features of the intermediate expansion layer.

[0071] For glasses classification, a multi-task learning approach is used. By reusing the network for keypoint prediction as the base network and freezing the parameters, at the bottleneck layer of the U-shaped base network, a feature vector with cross-entropy loss is used to train the binary classifier. FIG. 8A is a diagram showing an exemplary glasses classification network structure according to some embodiments of the present disclosure. FIG. 8B is a diagram showing an exemplary female hair prediction network structure according to some embodiments of the present disclosure. FIG. 8C is a diagram showing an exemplary male hair prediction network structure according to some embodiments of the present disclosure.

[0072] FIG. 9A shows some exemplary glasses classification prediction results according to some embodiments of the present disclosure. FIG. 9B shows some exemplary female hair prediction results according to some embodiments of the present disclosure. FIG. 9C shows some exemplary male hair prediction results according to some embodiments of the present disclosure.

[0073] FIG. 10 is a flowchart 1000 showing an exemplary process for constructing a face position map from a 2D face image of a real person according to some embodiments of the present disclosure. In reality, since different people have different face characteristics, the same key point corresponding to the same face characteristic (for example, the position of the eyebrows on a person's face) may have very different spatial coordinates. The problem of face detection becomes more difficult because the 2D face images used to generate the 3D face model are captured under different angles and different light conditions, and research in this area is a very active topic in the technical field of computer vision. In the present application, a plurality of methods for improving the efficiency and accuracy of face key point detection from any 2D face image of an object ranging from a real person to a cartoon character are proposed. In some embodiments, a set of user-provided face key points of the same face image is provided as a reference for correcting or improving the set of face key points initially detected by a computer-implemented method. For example, since there is a one-to-one mapping relationship based on their respective sequence numbers between the user-provided face key points and the computer-generated face key points, the refinement of the computer-generated face key points is defined as an optimization problem that reduces the difference between the two sets of face key points, measured by their corresponding spatial coordinates in the position map, for example.

[0074] The process of constructing a face position map includes step 1010 of generating a rough face position map from a two-dimensional face image.

[0075] The process also includes step 1020 of predicting a first set of key points in the two-dimensional face image based on the rough face position map.

[0076] The process further includes step 1030 of identifying a second set of keypoints in the 2D face image based on user-provided keypoint annotations.

[0077] The process further includes step 1040 of updating the coarse face position map so as to reduce the difference between the first set of keypoints and the second set of keypoints in the two-dimensional face image. For example, by reducing the difference between the first set of keypoints and the second set of keypoints in the 2D face image with respect to corresponding spatial coordinates, the first set of keypoints in the 2D face image based on the coarse face position map is modified to be more similar to the second set of keypoints in the 2D face image based on user-provided keypoint annotations, which are often considered to be more accurate. The modification of the first set of face keypoints automatically triggers an update of the initial coarse face position map from which the first set of keypoints is generated. Then, the updated coarse face position map can be used to predict a more accurate set of keypoints from the 2D face image. Note that the second set of keypoints in the 2D face image based on user-provided keypoint annotations does not mean being done manually. Instead, the user can use a method implemented by another computer to perform the annotation. In some embodiments, the number of the second set of keypoints (e.g., 10 - 20) is only a fraction of the number of the first set of keypoints (e.g., 96 or more), but the fact that the second set of keypoints is much more accurate contributes to the overall improvement of the first set of keypoints.

[0078] In one embodiment, the process further includes step 1050 of extracting a third set of key points as the final set of key points based on the updated face position map / final position map, wherein the third set of key points has the same positions as the first set of key points in the face position map. In some embodiments, the positions of the key points in the face position map are represented by the 2D coordinates of the array elements in the position map. As described above, the updated face position map benefits from the second set of key points in the 2D face image based on the user-provided key point annotations, and thus the third set of key points is more accurate and can be used in areas such as computer vision for more accurate face detection or computer graphics for more accurate 3D face modeling.

[0079] In one embodiment, instead of or in addition to step 1050, the process further includes step 1060 of reconstructing a 3D face model of a real person based on the updated face position map. In one example, the 3D face model is a 3D depth model.

[0080] Additional embodiments can include one or more of the following features.

[0081] In some embodiments, step 1040 of updating can include a step of converting the rough face position map to a converted face position map and a step of refining the converted face position map. As described above, the converted face position map can preserve more detailed face features of the person in the input image than the initial rough face position map, and thus the 3D face model based on the converted face position map is more accurate.

[0082] In some embodiments, the step of converting includes a step of estimating a conversion mapping from the rough face position map to the converted face position map by learning the difference between the first set of key points and the second set of key points, and a step of applying the conversion mapping to the rough face position map.

[0083] In some embodiments, the step of refining includes adjusting key points corresponding to the face position map transformed on the occluded side of the face contour to cover the entire face region according to the determination that the 2D face image is tilted. As described above, different 2D face images can be captured at different angles, and this refinement step can correct biases or errors introduced by different image capture conditions and store a more accurate 3D face model of the 2D face image. Further, the transformed face position map can store more detailed face features of the person in the input image than the initial rough face position map, and thus the 3D face model based on the transformed face position map is more accurate.

[0084] In some embodiments, the first set of key points can include 96 key points.

[0085] In some embodiments, the process of constructing the face position map can include face feature classification.

[0086] In some embodiments, the face feature classification is by a deep learning method.

[0087] In some embodiments, the face feature classification is by a multi-task learning or transfer learning method.

[0088] In some embodiments, the face feature classification includes hair prediction classification.

[0089] In some embodiments, the hair prediction classification includes female hair prediction having a plurality of classification tasks that can include curves, length, bangs, and blades.

[0090] In some embodiments, the hair prediction classification includes male hair prediction having a plurality of classification tasks that can include curve / length, bangs, and hair splits.

[0091] In some embodiments, the facial feature classification includes a glasses prediction classification. The glasses prediction classification includes a classification task that may include with glasses and without glasses.

[0092] The methods and systems disclosed herein can generate an accurate 3D face model (i.e., a position map) based on 2D keypoint annotation for 3D ground truth generation. This approach not only avoids the use of BFM and SFM face models, but also better preserves detailed facial features and prevents the loss of these important features caused by face model-based methods.

[0093] In addition to providing keypoints, deep learning-based solutions are used to provide complementary facial features such as hairstyles and glasses, which are essential for personalizing face avatars based on user input face images.

[0094] Although hairstyle and glasses prediction for facial feature classification are disclosed herein as examples, the framework is not limited to these exemplary tasks. The framework and solution are based on multi-task learning and transfer learning, which means that it is easy to extend the framework to include other facial features such as femininity makeup type classification, male beard type classification, and the presence or absence of mask classification. The design of the framework is suitable for extension to more tasks based on the requirements of various computer or mobile games.

[0095] In some embodiments, a lightweight color extraction method based on keypoints is introduced herein. The lightweight image processing algorithm quickly estimates local pixels without segmenting all pixels, resulting in higher efficiency.

[0096] During the training process, the user does not need to have pixel-level labels and only labels a few keypoints such as the corners of the eyes, the mouth, and the eyebrows.

[0097] The lightweight color extraction method disclosed in this specification can be used in various personalized face generation systems for games. To provide more free and personalized character generation, many games have started to adopt a free adjustment method. In addition to adjusting the shape of the face, users can also select different color combinations. For aesthetic purposes, in games, faces often use a predetermined texture instead of the actual face texture. This method and system disclosed in this specification enable users to automatically extract the average color of each part of the face just by uploading a photo. At the same time, the system can automatically correct the texture according to the extracted color. As a result, each part of the personalized face is generated closer to the actual color of the user's photo, improving the user experience. For example, if the user's skin tone is darker than the average skin tone of most people, the skin tone of the character in the game will become darker accordingly. FIG. 11 is a flowchart showing an exemplary color extraction and adjustment process according to some embodiments of the present disclosure.

[0098] To locate various parts of the face, as shown in FIG. 1 above, keypoints are defined for the main feature parts of the face. The above-described algorithm is used for keypoint prediction. Different from the semantic segmentation method, only the keypoints in the image are predicted without the need to classify each pixel, so the cost of labeling the prediction and training data is significantly reduced. With these keypoints, various parts of the face can be roughly located.

[0099] FIG. 12 shows an exemplary skin color extraction method according to some embodiments of the present disclosure. To extract features in the image, as shown in the rotated and aligned image 1204, it is necessary to rotate the face region in the original image 1202 so that the left and right keypoints 1 and 17 of the face are aligned with the corresponding left and right keypoints of the standard face.

[0100] Next, determine the area for skin color pixel inspection. The bottom coordinates of the eye keypoints are selected as the upper boundary of the detection area, the bottom keypoint of the nose is selected as the lower boundary of the detection area, and the left and right boundaries are determined by the face boundary keypoints. Thus, a skin color detection area is obtained as shown by area 1208 on image 1206.

[0101] Not all pixels within this area 1208 are skin pixels, and the pixels may also include some eyelashes, nostrils, nasolabial folds, hair, etc. Therefore, the median value of the R, G, and B values of all pixels within this area is selected as the final predicted average skin color.

[0102] FIG. 13 shows an exemplary eyebrow color extraction method according to some embodiments of the present disclosure. For the average color of the eyebrows, first, the main eyebrow, i.e., the eyebrow closer to the target lens, is selected. In some embodiments, when both eyebrows are the main eyebrows, the eyebrow pixels on both sides are extracted. As shown in FIG. 13, assuming that the left eyebrow is the main eyebrow, a quadrilateral area composed of keypoints 77, 78, 81, and 82 is selected as the eyebrow pixel search area. This is because the outer eyebrow is too thin and the influence of small keypoint errors becomes large. Since the inner eyebrow is sparse and often mixed with the skin color, the central eyebrow area 1302 is selected to collect pixels. Then, each pixel must first be compared with the average skin color, and only pixels having a difference greater than a specific threshold are collected. Finally, similar to the skin color, the median value of the R, G, and B of the collected pixels is selected as the final average eyebrow color.

[0103] FIG. 14 shows an exemplary pupil color extraction method according to some embodiments of the present disclosure. Similar to the eyebrow color extraction, when extracting the pupil color, first, the main eye side closer to the lens is selected. In some embodiments, when both eyes are the main eyes, the pixels on both sides are collected together. In addition to the pupil itself, the enclosed area included inside the key points of the eye can also include eyelashes, sclera, and reflections. These should be removed as much as possible in the pixel collection process to ensure that most of the final pixels come from the pupil itself.

[0104] To remove the eyelash pixels, the eye key points are contracted inward along the y-axis (the vertical direction in FIG. 14) by a specific distance to form the area 1402 shown in FIG. 14. To remove the sclera and reflections (shown by the circle 1404 in FIG. 14), such pixels are further excluded in this area 1402. For example, if all the R, G, and B values of a pixel are greater than a predetermined threshold, the pixel is excluded. The pixels collected in this way can ensure that most of them come from the pupil itself. Similarly, the central color is used as the average pupil color.

[0105] In some embodiments, for lip color extraction, only the pixels within the lower lip region are detected. The upper lip is thin and is often relatively sensitive to key point errors, and the upper lip has a light color, so it cannot well represent the lip color. Therefore, after rotating and correcting the photo, all the pixels in the area enclosed by the key points of the lower lip are collected, and the central color representing the average lip color is used.

[0106] FIG. 15 is a diagram showing an exemplary hair color extraction region used in a hair color extraction method according to some embodiments of the present disclosure. The extraction of hair color is more difficult than previous parts. The main reason is that each person's hairstyle is unique and the background of the photo is complex and diverse. Therefore, it is difficult to identify the positions of hair pixels. One way to accurately find hair pixels is to use a neural network to segment the hair pixels in the image. Since the annotation cost of image segmentation is very high and very high-precision color extraction is not required for game applications, a method based on approximate prediction of key points is used.

[0107] To obtain hair pixels, a detection region is first determined. As shown in FIG. 15, the detection region 1502 is rectangular. The lower boundary is the corners of both eyebrows, and the height (vertical line 1504) is the distance 1506 from the upper edge of the eyebrows to the lower edge of the eyes. The left and right are key points 1 and 17 for extending a fixed distance to the left and right, respectively. The hair pixel detection region 1502 obtained in this way is shown in FIG. 15.

[0108] FIG. 16 shows an exemplary separation between hair pixels and skin pixels within a hair color extraction region according to some embodiments of the present disclosure. Generally, the detection region includes three types of pixels: skin, hair, and background. In more complex cases, headwear is also included. Since the left and right ranges of the detection region are relatively conservative, the hair pixels included are assumed to be much larger than the background pixels in most cases. Therefore, the main process is to divide the pixels in the detection region into hair or skin.

[0109] For each line of pixels in the detection area, the change in skin color is often continuous, for example, from bright to dark, and the skin color and hair junction often have obvious changes. Therefore, the pixel at the center of each row is selected as the starting point 1608, and the left and right skin pixels are detected. First, a relatively conservative threshold is used to find more reliable skin color pixels and then expand them to the left and right. If the colors of adjacent pixels are relatively close, they are also marked as skin color. Such a method takes into account the skin color gradient and can obtain relatively accurate results. As shown in FIG. 16, within the hair color extraction area 1602, darker areas such as 1604 represent skin color pixels, and brighter areas such as 1606 represent hair color pixels. The R, G, B median values of the collected hair color pixels within the hair color area are selected as the final average hair color.

[0110] FIG. 17 shows an exemplary eye shadow color extraction method according to some embodiments of the present disclosure. The extraction of the eye shadow color is a bit different from the previous part. This is because the eye shadow is makeup that may or may not exist. Therefore, when extracting the eye shadow color, it is first necessary to determine whether the eye shadow exists, and if it exists, it is necessary to extract its average color. The extraction of the eye shadow color is performed only on the part close to the lens of the dominant eye, similar to the extraction of the color of the eyebrows and pupils.

[0111] First, it is necessary to determine which pixels belong to the eye shadow. In the case of the detection area of the eye shadow pixels, as shown in FIG. 17, the area 1702 within the lines 1704 and 1706 is used. The left and right sides of the area 1702 are defined as the inner and outer corners of the eye, and the upper and lower sides of the area are defined as the lower edge of the eyebrows and the upper edge of the eyes. In addition to the possible eye shadow pixels within this area 1702, there may also be eyelashes, eyebrows, and skin that need to be excluded when extracting the eye shadow.

[0112] In some embodiments, to eliminate the influence of the eyebrows, the upper edge of the detection region is further moved downward. To reduce the influence of the eyelashes, pixels having a luminance below a specific threshold are excluded. To distinguish between eyeshadow and skin color, the difference between the hue of each pixel and the average skin color is checked. The pixel is collected as a possible eyeshadow pixel only when the difference is greater than a specific threshold. The reason for using hue instead of RGB values is that the average skin color mainly gathers under the eyes, and the skin color above the eyes may have a large change in brightness. Since color is not sensitive to brightness, color is relatively stable. As a result, hue is more suitable for determining whether a pixel is skin.

[0113] Through the above processing, it is possible to determine whether the pixels in each detection region belong to eyeshadow. In some embodiments, when there is no eyeshadow, there may be an error that some pixels may still be recognized as eyeshadow.

[0114] To reduce the above error, each column of the detection region is checked. If the number of eyeshadow pixels in the current column is greater than a specific threshold, the current column is marked as an eyeshadow column. If the ratio of the eyeshadow column to the width of the detection region is greater than a certain threshold, it is considered that there is eyeshadow in the current image, and the central color of the collected eyeshadow pixels is used as the final color. In this way, a small number of pixels misclassified as eyeshadow will not cause an incorrect judgment for the entire eyeshadow.

[0115] Considering the art style, in most games, it is often not possible to freely adjust the colors of all the above parts. For parts where color adjustment is available, usually only a predetermined set of colors can be matched. Taking hair as an example, if a hairstyle can select 5 hair colors, the hairstyles in the resource pack include texture images corresponding to each hair color. At the time of detection, by selecting the texture image of the closest color according to the predicted result of the hair color, the desired hair rendering effect can be obtained.

[0116] In some embodiments, when only one color texture image is provided, the color of the texture image can be reasonably changed according to any detected color. To facilitate color conversion, the commonly used RGB color space representation is converted to the HSV color model. The HSV color model consists of three dimensions: hue H, saturation S, and value V. The hue H is represented as a 360-degree color range within the model, with red at 0 degrees, green at 120 degrees, and blue at 240 degrees. The saturation S represents the mixture of spectral colors and white. The higher the saturation, the brighter the color. As the saturation approaches 0, the color approaches white. The value V represents the lightness of the color, and the value range is from black to white. After color adjustment, the HSV median value of the texture image is expected to match the predicted color. Therefore, the calculation of the hue value for each pixel can be expressed as H i ’=(H i +H’-H)%1, where H i ’ and H i represent the hue of pixel i before and after adjustment, and H and H’ represent the median hue of the texture image before and after adjustment.

[0117] Unlike the hue, which is a continuous space where the ends are connected, the saturation and value have boundary singularities such as 0 and 1. When using a linear processing method similar to the hue adjustment, when the median value of the initial image or the adjusted image is close to 0 or 1, many pixel values may appear to have too high or too low saturation or lightness. This phenomenon causes unnatural colors. To solve this problem, the following non-linear curve is used to fit the saturation and value before and after pixel adjustment.

[0118]

Equation

[0119] In the above equation, x and y are the values of the saturation or value before and after adjustment, respectively. The only uncertain parameter is α, which can be derived as follows.

[0120] α = 1 / (1 + x / (1 - x) × (1 - y) / y)

[0121] This equation can ensure that α falls within the range from 0 to 1. Taking saturation as an example, the initial central saturation S can be simply calculated based on the input image. And the target saturation value S t can be obtained by hair color extraction and color space conversion. Therefore,

Equation

[0122] To make the display effect of the adjusted texture image closer to the actual image, special processing is performed on different parts. For example, to maintain the low saturation of hair,

Equation

[0123] Figure 19 is a flowchart 1900 showing an exemplary process of extracting color from a 2D face image of a real person according to some embodiments of the present disclosure.

[0124] The process of extracting colors from a two-dimensional face image of a real person includes step 1910 of identifying a plurality of key points in the two-dimensional face image based on a key point prediction model.

[0125] The process also includes step 1920 of rotating the two-dimensional face image until a plurality of target key points from the identified key points are aligned with the corresponding target key points of a standard face.

[0126] The process further includes step 1930 of identifying a plurality of parts in the rotated two-dimensional face image, where each part is defined by a respective subset of the identified plurality of key points.

[0127] The process further includes step 1940 of extracting the color of each of the plurality of parts defined by the corresponding subset of key points from the pixel values of the two-dimensional face image.

[0128] The process further includes step 1950 of generating a personalized three-dimensional model of the real person that matches the respective facial feature colors of the two-dimensional face image using the extracted colors of the plurality of parts in the two-dimensional face image.

[0129] Additional embodiments can include one or more of the following features.

[0130] In some embodiments, the key point prediction model of step 1910 for identification is formed based on machine learning from key points manually annotated by a user.

[0131] In some embodiments, the selected key points in rotation step 1920 used for alignment are arranged on the symmetric sides of the two-dimensional face image.

[0132] In some embodiments, in step 1940, extracting the average color for each of the plurality of portions can include selecting the median value of each of the R, G, and B values of all the pixels within each defined region within the corresponding portion as the predicted average color.

[0133] In some embodiments, in step 1940, the step of extracting the average color for each of the plurality of portions can include determining a region for skin color extraction within the skin portion, and selecting the median value of each of the R, G, and B values of all the pixels within the region for skin color extraction as the predicted average color of the skin portion. In some embodiments, the region for skin color extraction within the skin portion is determined as the region above the lower edge of the eyes and under the nose of the face.

[0134] In some embodiments, in step 1940, the step of extracting the average color for each of the plurality of portions can include eyebrow color extraction within the eyebrow portion. The eyebrow color extraction can include selecting the eyebrow as the target eyebrow according to the determination that the eyebrow is on the side closer to the observer of the 2D face image, selecting both eyebrows as the target eyebrows according to the determination that both eyebrows are equally close to the observer of the 2D face image, extracting the central eyebrow region within the target eyebrow, comparing each pixel value within the central region of the eyebrow with the average skin color, collecting the pixels within the intermediate eyebrow region where the pixel value difference from the average skin color exceeds the threshold, and selecting the median value of the R, G, and B values of each of the collected pixels as the predicted average color of the eyebrow portion for eyebrow color extraction.

[0135] In some embodiments, in step 1940, the step of extracting the average color of each of the plurality of parts may include pupil color extraction within the eye part. The pupil color extraction may include the step of selecting the eye as the target eye according to the determination that the eye is on the side closer to the observer of the 2D face image, the step of selecting both eyes as the target eyes according to the determination that both eyes are equally close to the observer of the 2D face image, the step of extracting the area within the target eye without eyelashes, the step of comparing each pixel value within the extracted area with a predetermined threshold value, the step of collecting the pixels having pixel values exceeding the predetermined threshold value within the extracted area, and the step of selecting the median value of each of the R, G, and B values of the pixels collected for pupil color extraction as the predicted average color of the pupil.

[0136] In some embodiments, in step 1940, the step of extracting the average color of each of the plurality of parts may include lip color extraction within the lip part. The lip color extraction may include the step of collecting all the pixels within the area surrounded by the key points of the lower lip, and the step of selecting the median value of each of the R, G, and B values of the pixels collected for lip color extraction as the predicted average color of the lip part.

[0137] In some embodiments, in step 1940, the step of extracting the average color of each of the plurality of parts may include hair color extraction within the hair part. The hair color extraction may include the step of identifying the area including the forehead part extending within the hair parts on both sides, the step of determining the pixel color change exceeding a predetermined threshold value from the center of the area to the left and right boundaries, the step of dividing the area into a hair area and a skin area based on the pixel color change exceeding the predetermined threshold value, and the step of selecting the median value of each of the R, G, and B values of the pixels in the hair area within the area as the predicted average color of the hair part.

[0138] In some embodiments, the region including the frontal portion extending within the hair portions on both sides is identified as a rectangular region having a lower boundary at the corners of both eyebrows, left and right boundaries at a certain distance outward from key points located on the left-right symmetry sides of the 2D face image, and a height equal to the distance from the upper edge of the eyebrows to the lower edge of the eyes.

[0139] In some embodiments, in step 1940, the step of extracting the average color of each of the plurality of portions can include extracting the eyeshadow color within the eyeshadow portion. The eyeshadow color extraction includes a step of selecting the eye as the target eye according to the determination that one eye is on the side closer to the observer of the 2D face image, a step of selecting both eyes as the target eyes according to the determination that both eyes are equally close to the observer of the 2D face image, a step of extracting the central region within the eyeshadow portion close to the target eye, collecting pixels within the extracted central region that have a luminance exceeding a predetermined luminance threshold to exclude eyelashes and have a pixel hue value difference from the average skin hue value exceeding a predetermined threshold, a step of labeling the pixel column as an eyeshadow column according to the determination that the number of collected pixels within one pixel column in the extracted central region is greater than the threshold, and a step of selecting the median value of each of the R, G, B values of the collected pixels for eyeshadow color extraction as the predicted eyeshadow color of the eyeshadow portion according to the determination that the ratio of the eyeshadow column to the width of the extracted central region is greater than a specific threshold.

[0140] In some embodiments, the process of extracting colors from a 2D face image of a real person can further include a step of converting the texture map based on the average color while maintaining the original luminance and color differences of the texture map. The converting step includes a step of converting the average color from the RGB color space representation to the HSV (hue, saturation, value) color space representation, and a step of adjusting the color of the texture map to reduce the difference between the central HSV value of the average color and the central HSV value pixel of the texture map.

[0141] The methods and systems disclosed herein can be used in applications of different scenarios such as character modeling and game character generation. The lightweight method can be flexibly applied to various devices including mobile devices.

[0142] In some embodiments, the definition of facial keypoints in current systems and methods is not limited to the current definition, and other definitions are also possible as long as the contour of each part can be fully represented. Furthermore, in some embodiments, the color directly returned by the scheme may not be used directly, but can be matched with a predetermined color list to achieve further color screening and control.

[0143] The modified method for optimizing the Laplacian operator requires that the mesh be a differentiable manifold. However, in practice, the meshes created by game artists often contain artifacts such as duplicated vertices and unsealed edges that may compromise the properties of the manifold. Therefore, methods such as harmonic deformation can only be used after the mesh has been carefully arranged. The affine deformation method proposed herein does not use the Laplacian operator, so there is no such strong constraint.

[0144] The group of deformation methods represented by harmonic deformation is sometimes troubled by insufficient deformation ability. The harmonic function that solves the Laplacian operator once often fails to achieve a smoothed result due to its low smoothness requirement. The multi-harmonic functions that solve the high-order (>=3) Laplacian operator fail in many meshes due to their high requirements of being at least 6 times differentiable. In most cases, it is observed that only the harmonic deformation that solves the Laplacian operator twice can produce acceptable results. Still, the deformation may still be insufficient due to the lack of its adjustment freedom. The affine deformation proposed herein can achieve delicate deformation adjustment by changing the smoothness parameter, and the range of its deformation results covers the range of using harmonic deformation.

[0145] Figure 20 is a flowchart showing an exemplary head avatar deformation and generation process according to some embodiments of the present disclosure. Using the techniques proposed in the present disclosure, the head mesh can be appropriately deformed without bonding to the bone. Therefore, the workload required from the artist is significantly reduced. These techniques are applicable to meshes of different styles in order to obtain better generality. In the production of game assets, artists can save head models in various formats using tools such as 3DMax or Maya, but the internal representation of these formats is all polygon meshes. The polygon mesh can be easily converted into a pure triangle mesh called a template model. For each template model, 3D key points are manually marked once on the template model. Then, it can be used to deform into a characteristic head avatar according to the detected and reconstructed 3D key points from any human face image.

[0146] Figure 21 is a diagram showing an exemplary head template model configuration according to some embodiments of the present disclosure. The head template model 2102 usually consists of parts such as a face 2110, eyes 2104, eyelashes 2106, teeth 2108, and hair as shown in Figure 21. Without constraining the bones, the mesh deformation depends on the connection structure of the template mesh. Therefore, it is necessary to decompose the template model into its semantic parts and deform the face mesh first. All other parts can be automatically adjusted by setting and following specific key points on the face mesh. In some embodiments, an interactive tool is provided to detect all topologically connected parts, and the user can use it to conveniently export those semantic parts for further deformation.

[0147] In some embodiments, the image keypoints of a human face can be obtained via several detection algorithms or AI models. For the purpose of driving mesh deformation, these keypoints need to be mapped to the vertices on the template model. Due to the randomness of mesh connectivity and the lack of marking data for 3D human keypoints, there is no tool that can accurately and automatically mark the 3D keypoints on any head model. Therefore, an interactive tool has been developed that can quickly mark the keypoints on the 3D model manually. FIG. 22 is a diagram showing some exemplary keypoint markings on realistic-style 3D models such as 2202 and 2204 and comic-style 3D models such as 2206 and 2208 according to some embodiments of the present disclosure.

[0148] In the marking procedure, the positions of the marked 3D keypoints on the 3D model need to match the image keypoints most closely. Since the keypoints are marked on individual vertices of the 3D model mesh, the import of deviation is inevitable. To offset such deviations, one way is to define appropriate rules in position processing. FIG. 23 is a diagram showing an exemplary comparison between template model rendering, manually marked keypoints, and AI-detected keypoints according to some embodiments of the present disclosure. In some embodiments, for those models that are relatively realistic, a keypoint detection and reconstruction algorithm can be applied to the rendering of the template model (2302). For example, the results of 3D keypoints (2306) by artificial intelligence can be further compared with the manually marked ones (2304). Therefore, the deviation between the two groups of keypoints is calculated. When detecting a human image, the calculated deviation is reduced from the detected keypoints in the real image, and the adverse effects of artificial marking are eliminated.

[0149] The method of affine transformation disclosed in this specification is a key-point-driven mathematical modeling that ultimately solves a system of linear equations. The method disclosed herein takes one step of using the detected key points as boundary conditions to deform the template mesh and uses different constraints in the optimization process. FIG. 24 is a diagram showing an exemplary triangular affine transformation according to some embodiments of the present disclosure.

[0150] In some embodiments, the deformation from the template mesh to the predicted mesh is regarded as the assembly of the affine transformation of each triangle. The affine transformation of a triangle can be defined as a 3×3 matrix T and a translation vector d. As shown in FIG. 24, the position of the deformed vertex after affine transformation is v i ’ = Tv i + d, i ∈ 1...4, where v1, v2, v3 represent each vertex of the triangle respectively, and v4 is an additional point introduced in the direction of the normal of the triangle, satisfying the equation v4 = v1 + (v2 - v1) × (v3 - v1) / sqrt(|(v2 - v1) × (v3 - v1)|). In the above equation, the result of the cross product is normalized to be proportional to the length of the edge of the triangle. The reason for introducing v4 is that the coordinates of the three vertices are not sufficient to determine a unique affine transformation. After introducing v4, T = [v’2 - v’1 v’3 - v’1 v’4 - v’1] × [v2 - v1 v3 - v1 v4 - v1] -1 is obtained, and the non-translation part of the matrix T is obtained. The matrix V = [v2 - v1 v3 - v1 v4 - v1] -1 depends only on the template mesh, which is an invariant of other deformation coefficients, and can be pre-calculated as a sparse coefficient matrix for constructing a linear system later.

[0151] So far, the non-transformation part of the affine transformation T in the mathematical formula has been shown. To construct the linear system for optimization, assuming the number of mesh vertices is N and the number of triangles is F, the following four constraints are considered.

[0152] Constraint on the position of the key point: E k=Σ i=1 ||v’ i -c’ i || 2 ,c’ i represents the detected keypoint positions after mesh deformation.

[0153] Adjacency smoothness constraint: E s =Σ i=1 Σ j∈adj(i) ||T i -T j || 2 means that the affine transformations between adjacent triangles should be as similar as possible. To avoid redundant calculations and improve the performance for building the system, the adjacency relationships can be queried and memorized in advance.

[0154] Characteristic constraint: E i =Σ i=1 ||T i -I|| 2 where I represents the identity matrix. This constraint means that the affine transformation should not change as little as possible, which helps to maintain the characteristics of the template mesh.

[0155] Original position constraint: E l =Σ i=1 N||v’ I -c i || 2 where c i represents the position of each vertex on the template mesh before deformation.

[0156] The last constraint is the weighted sum of the above constraints: minE = w k E k +w s E s +w i E i +w l E l where the weights w k ,w s ,w i ,w lThey are ranked from the strongest to the weakest. Using the above constraints, a linear system can finally be constructed, whose size is (F + N) × (F + N), and the weights are multiplied by the corresponding coefficients in the system. The unknowns are the coordinates of each vertex after deformation, in addition to the additional point v'4 of each triangle. Since the former term is useful, the result of v'4 is discarded. In the process of continuous deformation, all constraint matrices except the constraints on the positions of the key points can be reused. The affine transformation can achieve real-time performance of 30 fps on a normal personal computer and an intelligent phone for a mesh with thousands of vertices.

[0157] FIG. 25 is a diagram showing an exemplary comparison of some head model deformation results with and without a blend shape process according to some embodiments of the present disclosure.

[0158] In some embodiments, when deforming the head model of a game avatar, the region of interest is typically only the face. The upper and back parts of the head and neck should remain unchanged; otherwise, there may be mesh penetration between the head and the hair or the body. To avoid this problem, the results of the affine transformation and the template mesh are linearly interpolated in a blend shape manner. The weights for blending can be painted in 3D modeling software or calculated using barycentric or affine transformation with minor changes. For example, the weight of the key points is set to 1s, while more markers (the dark points of 2504 in FIG. 25) are added to the head model, and their weights are set to 0s. In some embodiments, inequality constraints are added to the solving process to enforce all weights to fall within the range of 0 to 1, but this significantly increases the complexity of the solution. Through experiments, good results can be obtained by clipping weights smaller than 0 or larger than 1. As shown in 2504 of FIG. 25, the weight of the darkest model part is 1s, and the weight of the colorless model part is 0s. In the bending weight rendering 2504, there is a natural transition between the bright key points and the dark markers. In the case of blend shape, the back of the deformed model (shown in 2506 of FIG. 25) remains the same as the original (shown in 2502 of FIG. 25). Without blend shape, the back of the deformed model (shown in 2508 of FIG. 25) is not the same as the original (shown in 2502 of FIG. 25).

[0159] In some embodiments, the affine transformation can achieve different deformation effects by manipulating the weights of the constraints, including simulating the results of the barycentric transformation. FIG. 26 is a diagram showing an exemplary comparison of affine transformations with different weights and barycentric transformations according to some embodiments of the present disclosure. As shown in FIG. 26, the smoothness is related to the adjacent smoothness weight w s and the characteristic weight w iIt is a ratio to. The dark points are key points, and the darkness of the color represents the displacement between the deformed position of the vertex and its original position. In all deformation results, one key point remains unchanged while the other moves to the same position. This indicates that when gradually increasing the adjacent smoothness weight with respect to the characteristic weight, the smoothness of the deformed sphere also increases accordingly. Also, the result of the biharmonic deformation can match the result of the affine deformation where the smoothness is somewhere between 10 and 100. This shows that the affine deformation has a greater degree of freedom in deformation compared to the biharmonic deformation.

[0160] Using the workflow described in this specification, the game can easily integrate the function of intelligent generation of the head avatar. For example, FIG. 27 shows some exemplary results automatically generated from several randomly selected female images (not shown in FIG. 27) using a realistic template model according to some embodiments of the present disclosure. All personalized head avatars reflect some characteristics of their corresponding images.

[0161] FIG. 28 is a flowchart 2800 showing an exemplary process for generating a 3D head deformation model from a 2D face image of a real person according to some embodiments of the present disclosure.

[0162] The process of generating a 3D head deformation model from a 2D face image includes step 2810 of receiving a 2D (two - dimensional) face image.

[0163] The process also includes step 2820 of identifying a first set of key points in the 2D face image based on an artificial intelligence (AI) model.

[0164] The process further includes step 2830 of mapping the first set of key points to a second set of key points located at a plurality of vertices of the mesh of the 3D head template model based on a set of user - provided key point annotations located on the 3D head template model.

[0165] This process further includes step 2840 of performing a deformation on the mesh of the 3D head template model to obtain a deformed 3D head mesh model by reducing the difference between a first set of key points and a second set of key points. In some embodiments, there is a correspondence between the key points in the first set and the key points in the second set. After projecting the second set of key points into the same space as the first set of key points, a function is generated to measure the positional difference between each of the first set of key points and the second set of key points. By deforming the mesh of the 3D head template model, the second set of key points in the space is optimized when the function that measures the positional difference (e.g., position, adjacent smoothness, characteristics, etc.) between the first set of key points and the second set of key points is minimized.

[0166] This process further includes step 2850 of applying the blend shape method to the deformed 3D head mesh model to obtain a personalized head model according to the 2D face image.

[0167] Additional embodiments can include one or more of the following features.

[0168] In some embodiments, step 2830 of mapping can further include associating a first set of key points on the 2D face image with a plurality of vertices on the mesh of the 3D head template model, identifying a second set of key points based on a set of user-provided key point annotations on the plurality of vertices on the mesh of the 3D head template model, and mapping the first set of key points and the second set of key points based on the corresponding identified features by each key point on the face.

[0169] In some embodiments, the second set of keypoints is placed by applying a previously computed deviation to a set of user-provided keypoint annotations. In some embodiments, the previously computed deviation is the deviation between a previous set of AI-identified keypoints of the 3D head template model and a previous set of user-provided keypoint annotations on a plurality of vertices of the mesh of the 3D head template model.

[0170] In some embodiments, step 2840 of performing the deformation can include deforming the mesh of the 3D head template model into a deformed 3D head mesh model by using the mapping of the first set of keypoints to the second set of keypoints and using boundary conditions for the deformation with respect to the first set of keypoints.

[0171] In some embodiments, step 2840 of performing the deformation can further include applying different constraints to a process of deformation optimization that includes one or more of the position, adjacent smoothness, characteristics, and original position of the keypoints.

[0172] In some embodiments, step 2840 of performing the deformation can further include applying constraints to a process of deformation that is a weighted sum of one or more of the position, adjacent smoothness, characteristics, and original position of the keypoints.

[0173] In some embodiments, step 2820 of identifying the first set of keypoints includes using a convolutional neural network (CNN).

[0174] In some embodiments, the deformation includes an affine deformation without a Laplacian operator. In some embodiments, the affine deformation achieves deformation adjustment by changing a smoothness parameter.

[0175] In some embodiments, the mesh of the 3D head template model can be deformed without being coupled to the bones. In some embodiments, the face deformation model includes a realistic style model or a cartoon style model.

[0176] In some embodiments, at step 2850, the step of applying the blend shape method to the deformed 3D head mesh model includes specifying each blend weight on the key points of the deformed 3D head mesh model according to the positions of the key points, and applying different levels of deformation to the key points with different blend weights.

[0177] In some embodiments, at step 2850, the step of applying the blend shape method to the deformed 3D head mesh model includes keeping the back surface of the deformed 3D head mesh model in the same shape as the original back surface shape of the 3D head template model before deformation.

[0178] In some embodiments, the semantic parts on the template model are not limited to eyes, eyelashes, or teeth. Decorations such as glasses can potentially be adaptively adjusted by adding and tracking new key points on the face mesh.

[0179] In some embodiments, the key points on the template model are added manually. In some other embodiments, deep learning techniques can also be utilized to automatically add the key points of different template models.

[0180] In some embodiments, the solution procedure for affine deformation can utilize some numerical tricks to further improve its computational performance.

[0181] In some embodiments, the systems and methods disclosed herein form a light-weighted key-point-based face avatar generation system having many advantages such as those listed below.

[0182] The requirements for the input image are low. The system and method do not require the face to face the camera directly, and a certain degree of in-plane rotation, out-of-plane rotation, and occlusion clearly do not affect the performance.

[0183] Applicable to both realistic games and anime games. This system does not limit the game style to realistic ones, but can also be applied to the manga style.

[0184] Lightweight and customizable. Each module of this system is relatively lightweight and suitable for mobile devices. The modules of this system are separated, and users can adopt different combinations according to different game styles to build the final face generation system.

[0185] In some embodiments, for a given single photo, the main face is first detected and keypoint detection is performed. In an actual image, the face may not face the camera, and the actual face is not always completely symmetric. Therefore, the keypoints in the original image are preprocessed to achieve a set of unified symmetric and smooth keypoints. Next, the keypoints are adjusted according to a specific style of the game, such as enlarged eyes and a thin face. After obtaining the stylized keypoints, the stylized keypoints are converted into control parameters of the face model in the game, generally bone parameters or slider parameters.

[0186] In some embodiments, the viewing angle of the actual face may not face the camera directly, and there may be problems such as left-right asymmetry and keypoint detection errors. FIG. 29 is a diagram showing exemplary keypoint processing flow steps according to some embodiments of the present disclosure. The keypoints detected from the original image 2904 cannot be used directly and require specific processing. Here, the process is divided into three steps: normalization, symmetry, and smoothing, as shown in FIG. 29.

[0187] In some embodiments, it is necessary to adjust the in-game standard face model based on the prediction of the actual face key points. The process needs to ensure that the key points of the in-game standard face model and the actual face are aligned in terms of scale, position, and orientation. Therefore, the normalization 2906 of the predicted key points and the key points on the game face model includes the following parts, namely, scale normalization, translation normalization, and angle normalization.

[0188] In some embodiments, all three-dimensional face key points of the original detection are defined as p, and the i-th key point is p i ={x i ,y i ,z i}. For example, the normalized origin is defined as the midpoint of key points No. 1 and No. 17 (refer to the definition of key points in FIG. 1), that is, c = (p1 + p 17 ) / 2. In the case of scale, the distance between the first and 17th key points from the origin is adjusted to 1. As a result, the three-dimensional key points normalized by scale and translation are p' = (p - c) / ||p1 - c||.

[0189] In some embodiments, after normalizing the scale and translation, the face orientation is further normalized. As shown in the image 2902 of FIG. 29, the actual photo face may not face the lens directly and always has a certain deflection, which may exist on the three coordinate axes. The predicted three-dimensional key points of the face along the x, y, and z coordinate axes are sequentially rotated so that the face orientation faces the camera. When rotating along x, the z coordinates of key points 18 and 24 (refer to the definition of key points in FIG. 1) are aligned, that is, to obtain the rotation matrix R X , the depth of the top of the nose bridge is made the same as the depth of the bottom of the nose. When rotating along the y-axis, the z coordinates of key points 1 and 17 are rotated by the rotation matrix R Yis aligned to obtain. When rotating along the z-axis, the y-coordinates of keypoints 1 and 17 are the rotation matrix R Z is aligned to obtain. Therefore, the directions of the keypoints are aligned, and the normalized keypoints are shown as follows. P norm =R Z ×R Y ×R X ×P’

[0190] In some embodiments, the scale, position, and angle of the normalized keypoints are adjusted to be uniform, but the obtained keypoints are often not complete faces. For example, the bridge of the nose may not be a straight line in the center, and the facial features may not be symmetric. This is because the actual face in the photo is not completely symmetric due to expression or its own characteristics, and additional errors are introduced when predicting keypoints. Although the actual face may not be symmetric, if the face model in the game is not symmetric, it will look bad and the user experience will be significantly reduced. Therefore, the symmetry of the keypoints as shown in 2908 is a necessary process.

[0191] Since the keypoints are normalized, in some embodiments, a simple symmetry method is to average the y- and z-coordinates of all left-right symmetric keypoints and replace the original y- and z-coordinates. This method works well in most cases, but when the face rotates at a large angle in the y-axis direction, performance is sacrificed.

[0192] In some embodiments, for example, when using the human face in FIG. 29, if the face is deflected to the left by a large angle, the eyebrow part will disappear from view. At the same time, the left eye becomes smaller than the right eye due to perspective. The 3D keypoints can partially compensate for the effects caused by perspective relationships, but the 2D projections of the 3D keypoints corresponding to the keypoints still need to be retained on the image. Therefore, an overly large angular deflection will result in an obvious difference in the size of the eyes and eyebrows in the 3D keypoint detection results. To address the influence of the angle, when the deflection angle of the face along the y-axis is large, the eye and eyebrow closer to the lens are used as the main eye and main eyebrow, and they are copied to the opposite side to reduce the error caused by the angular deflection.

[0193] In some embodiments, since the prediction error of the keypoints is inevitable, in some individual cases, the symmetrized keypoints may still not match the actual face. Since the shapes of the actual face and facial features are very different, it is difficult to achieve a relatively accurate description using a predetermined parameterized curve. Therefore, when smoothing as shown in 2910, only some regions, such as the contours of the face, eyes, eyebrows, lower lip, etc., are smoothed. These regions basically maintain a monotonic and smooth state, that is, there is no jagged state. In this case, the target curve must always be a convex curve or a concave curve.

[0194] In some embodiments, whether a keypoint satisfies the definition of a convex curve (or a concave curve) is checked one by one for the relevant boundaries. FIG. 30 is a diagram showing an exemplary keypoint smoothing process 2910 according to some embodiments of the present disclosure. As shown in FIG. 30, without loss of generality, the target curve should be convex. For each keypoint 3002, 3004, 3006, 3008, and 3010, it is checked whether its position is on the line of its adjacent left and right keypoints. If the condition is satisfied, it means that the current keypoint meets the convex curve requirement. Otherwise, the current keypoint is moved to the line connecting the left and right keypoints. For example, in FIG. 30, the keypoint 3006 does not meet the limit of the convex curve and is moved to the position 3012. When multiple keypoints are moved, the curve may not be guaranteed to be convex or concave after the movement. Therefore, in some embodiments, multiple smoothings are used to obtain a relatively smooth keypoint curve.

[0195] Different games have different face styles. In some embodiments, the actual face keypoints need to be converted to the styles required by the game. The game faces in a realistic style are similar, but the faces in a comic style are very different. Therefore, it is difficult to have a uniform standard for the stylization of keypoints. The definition of stylization in actual use comes from the game designers who adjust the face characteristics according to a specific game style.

[0196] In some embodiments, a more general face adjustment scheme that most games may require is implemented. For example, custom corrections can be made according to different game art styles such as adjustment of face length, width, face features, adjustment levels, zoom ratios, etc. At the same time, the user can also customize any special style adjustment method, such as changing the shape of the eyes to a rectangle. The system can support any adjustment method.

[0197] In some embodiments, using the key points of the stylized face, the standard game face is deformed such that the key points of the deformed face reach the positions of the target key points. Since most games use control parameters such as bones or sliders to adjust the face, a set of control parameters is required to move the key points to the target positions.

[0198] Since the definitions of bones or sliders in different games can vary and may be modified at any time, it is impossible to directly define a simple parameterized function from key points to bone parameters. In some embodiments, a machine learning method is used to convert key points to parameters via a neural network called a K2P (Key Point-to-Parameter) network. Since the number of general parameters and key points is not large (generally less than 100), in some embodiments, a K-layer fully connected network is used.

[0199] FIG. 31 is a block diagram showing a conversion process from exemplary key points to control parameters (K2P) according to some embodiments of the present disclosure. To use a machine learning method, in some embodiments, first bone or slider parameters are randomly sampled and supplied to the game client 3110, and key points are extracted from the generated game face. In this way, a large amount of training data can be obtained (pairs of parameters 3112 and key points 3114). Next, an autoencoder machine learning method is implemented, which is divided into two steps. The first step is to train the P2K (parameter-to-key point) network 3116 to simulate the process of generating game parameters to key points. In the second step, according to the method described herein, a large number of unlabeled real face images 3102 are used to generate real face key points 3104, and then a large number of stylized key points 3106 are generated. These unlabeled stylized key points 3106 are autoencoder training data. In some embodiments, a set of key points K is input to the K2P network 3108 for learning to obtain the output parameter P. Since the ground truth of the ideal parameters corresponding to these key points is not available, P is further input to the P2K network 3116 trained in the first step to obtain the key point K'. In some embodiments, the K2P network 3108 can be learned by calculating the mean squared error (MSE) loss between K and K'. In some embodiments, during the second step, the P2K network 3116 is fixed and not continuously adjusted. With the help of the P2K network 3116, the process of controlling the parameters of the game client 3110 to key points is simulated using a neural network, thus building the basis for the learning of the K2P network 3108 in the second step. In this way, the final face generated by the parameters remains close to the key points of the generated target stylized face.

[0200] In some embodiments, at the same time, by adjusting the corresponding weights when calculating the MSE loss between K and K’, weights are added to specific keypoints such as the eye keypoints. The definition of the keypoints is pre-defined and not affected by the bones or sliders of the game client, so it is easier to adjust the weights.

[0201] In some embodiments, in actual applications, in order to improve the accuracy of the model, for the parts that can be separated, the neural networks can be trained separately. For example, if some bone parameters only affect the keypoints in the eye region and other parameters do not affect this region, these parameters and this part of the keypoints form a set of independent regions. Separate K2P models 3108 are trained for each group of such regions, and each model can adopt a more lightweight network design. This can not only further improve the accuracy of the model but also reduce the computational complexity.

[0202] FIG. 32 shows some exemplary results of automatic face generation for mobile games according to some embodiments of the present disclosure. As shown in FIG. 32, the results from the original face images (3202 and 3206) to the game face avatar image generation (3204 and 3208) are shown. In some embodiments, when stylizing, the open mouth is closed, and different levels of restrictions and caricaturization are applied to the nose, mouth, face shape, eyes, and eyebrows. The finally generated results still retain the characteristics of a specific human face and meet the aesthetic requirements of the game style.

[0203] FIG. 33 is a flowchart 3300 showing an exemplary process of customizing the standard face of an in-game avatar using a 2D face image of a real person according to some embodiments of the present disclosure.

[0204] The process of customizing the standard face of an avatar in a game using a two-dimensional face image of a real person includes step 3310 of identifying a set of target key points in the two-dimensional face image. As described above, the target can be a real person or a virtual character in a virtual world.

[0205] The process also includes step 3320 of converting the set of target key points into a set of avatar key points associated with the avatar in the game.

[0206] The process further includes step 3330 of generating a set of face control parameters for the standard face of the avatar by applying a key point-to-parameter (K2P) neural network model to the set of avatar key points, where the set of face control parameters is associated with one of a plurality of face features of the standard face respectively. As described above in connection with FIG. 31, the K2P network 3108 is a deep learning neural network model that can predict a set of face control parameters based on the set of input avatar key points because different sets of avatar key points can correspond to different sets of face control parameters such that when the set of face control parameters is applied to the standard face of the avatar, the adjusted standard face can have a set of key points similar to the set of input avatar key points.

[0207] This process further includes step 3340 of adjusting a plurality of face features of the standard face by applying the set of face control parameters to the standard face.

[0208] Additional embodiments can include one or more of the following features.

[0209] In some embodiments, at step 3330, the K2P neural network model obtains a plurality of training 2D face images of real people, generates a set of training game style or avatar key points for each of the plurality of training 2D face images, presents each set of training game style or avatar key points to the K2P neural network model to obtain a set of face control parameters, presents the set of face control parameters to a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a predicted set of game style or avatar key points corresponding to the set of training game style or avatar key points, and updates the K2P neural network model by reducing the difference between the set of training game style or avatar key points and the corresponding predicted set of game style or avatar key points. As described above in connection with FIG. 31, in contrast to the K2P network 3108, the P2K network 3116 is a deep learning neural network model that predicts a set of avatar key points based on a set of input face control parameters, because different sets of face control parameters can result in different sets of avatar key points such that the set of output avatar key points associated with the P2K network 3116 matches the set of input avatar key points associated with the K2P network 3108 when the two neural network models are considered to perform inverse processes with respect to each other.

[0210] In some embodiments, the pre-trained P2K neural network model is configured to receive a set of control parameters including bone or slider parameters associated with an avatar in a game and predict a set of game style key points for the avatar in the game according to the set of control parameters.

[0211] In some embodiments, the difference between a set of training game style keypoints and a corresponding set of predicted game style keypoints is the sum of the mean squared errors between the set of training game style keypoints and a corresponding set of predicted game style keypoints.

[0212] In some embodiments, the trained K2P and pre-trained P2K neural network models are specific to the game.

[0213] In some embodiments, a set of real keypoints in a 2D face image corresponds to the face features of a real person in the 2D face image.

[0214] In some embodiments, the standard face of an avatar in the game can be customized for different characters in the game according to different real person face images.

[0215] In some embodiments, the deformed face of the avatar is a cartoon-style face of a real person. In some embodiments, the deformed face of the avatar is a real face of a real person.

[0216] In some embodiments, in step 3320, the step of converting a set of real keypoints to a set of game style keypoints includes normalizing the set of real keypoints to a canonical space, symmetrizing the normalized set of real keypoints, and adjusting the symmetrized set of real keypoints according to a predetermined style associated with the avatar in the game.

[0217] In some embodiments, the step of normalizing a set of real keypoints to a canonical space includes scaling the set of real keypoints to the canonical space and rotating the scaled set of real keypoints according to the orientation of the set of real keypoints in the 2D face image.

[0218] In some embodiments, the step of converting a set of real keypoints into a set of game-style keypoints further includes the step of smoothing the set of keypoints that has been symmetrized to meet a predetermined convex or concave curve requirement.

[0219] In some embodiments, the step of adjusting a set of real keypoints symmetrized according to a predetermined style associated with an avatar in the game includes one or more of adjusting the length of the face, adjusting the width of the face, adjusting facial features, adjusting the zoom, and adjusting the shape of the eyes.

[0220] The systems and methods disclosed herein can be applied to an automatic face generation system for various games for both real-style games and cartoon-style games. The system has an interface that is easy to integrate and improves the user experience.

[0221] In some embodiments, the systems and methods disclosed herein can be used in a 3D face avatar generation system for various games, and the complex manual adjustment process is automated to improve the user experience. The user can take a selfie or upload an existing photo. The system can extract features from the face in the photo and then automatically generate control parameters (such as bones or sliders) for the game face via an AI face generation system. The game end uses these parameters to generate a face avatar such that the created face has the user's facial features.

[0222] In some embodiments, this system can be easily customized according to various games, including keypoint definition, styling method, skeleton / slider definition, etc. The user can choose to adjust only specific parameters, automatically retrain the model, or add a custom control algorithm. In this way, the present invention can be easily deployed to different games.

[0223] Further embodiments also include various subsets of the above embodiments that are combined or otherwise reconfigured in various other embodiments.

[0224] Here, referring to the description of the accompanying drawings, an image processing apparatus according to an embodiment of the present application is implemented. The image processing apparatus may be realized by different types of computer devices in various forms, such as a server or a terminal (for example, a desktop computer, a notebook computer, or a smartphone). Hereinafter, the hardware structure of the image processing apparatus according to the embodiment of the present application will be further described. It should be understood that FIG. 34 shows only an exemplary structure, not all the structures of the image processing apparatus, and the partial or overall structure shown in FIG. 34 may be implemented according to requirements.

[0225] Referring to FIG. 34, FIG. 34 is a schematic diagram of an optional hardware structure of an image processing apparatus according to an embodiment of the present application, and in an actual application, it can be applied to a server or various terminals that execute an application program. The image processing apparatus 3400 shown in FIG. 34 includes at least one processor 3401, a memory 3402, a user interface 3403, and at least one network interface 3404. The components within the image processing apparatus 3400 are coupled to each other by a bus system 3405. It should be understood that the bus 3405 is configured to implement connections and communications between components. In addition to including a data bus, the bus system 3405 can further include a power bus, a control bus, and a status signal bus. However, for clarity of description, in FIG. 34, all buses are denoted as the bus system 3405.

[0226] The user interface 3403 can include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad, a touch screen, etc.

[0227] The memory 3402 may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory.

[0228] The memory 3402 in the embodiments of the present application is configured to store different types of data to support the operation of the image processing apparatus 3400. Examples of the data include any computer program such as an executable program 34021 and an operating system 34022 used to execute operations on the image processing apparatus 3400, and the program used to execute the image processing method of the embodiments of the present application may be included in the executable program 34021.

[0229] The image processing method disclosed in the embodiments of the present application may be applied to the processor 3401 or may be executed by the processor 3401. The processor 3401 may be an integrated circuit chip and has signal processing capabilities. In the implementation process, each step of the image processing method may be completed using the integrated logic circuit of the hardware in the processor 3401 or instructions in the form of software. The aforementioned processor 3401 may be a general-purpose processor, a digital signal processor (DSP), another programmable logic device, discrete gates, transistor logic devices, discrete hardware components, etc. The processor 3401 can implement or execute the methods, steps, and logic block diagrams provided in the embodiments of the present application. The general-purpose processor may be a microprocessor, any conventional processor, etc. The steps of the method provided in the embodiments of the present application may be directly executed by the hardware decoding processor or may be executed by combining the hardware module and the software module in the decoding processor. The software module may be arranged in a storage medium. The storage medium is arranged in the memory 3402. The processor 3401 reads the information in the memory 3402 and combines the information with its hardware to execute the steps of the image processing method provided in the embodiments of the present application.

[0230] In some embodiments, image processing as well as 3D face and head formation can be achieved on a group of servers or in the cloud on a network.

[0231] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on a computer-readable medium or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium can include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, the computer-readable medium generally can correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the embodiments described in this application. A computer program product can include a computer-readable medium.

[0232] The terms used in the description of the embodiments of this specification are for the sole purpose of describing particular embodiments and are not intended to limit the scope of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0233] Terms such as first, second, etc. may be used herein to describe various elements, but it will also be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the embodiment, the first electrode may be referred to as the second electrode, and similarly, the second electrode may be referred to as the first electrode. The first electrode and the second electrode are both electrodes, but they are not the same electrode.

[0234] The description of this application is presented for purposes of illustration and explanation and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art having the benefit of the teachings presented in the foregoing description and the associated drawings. The embodiments were chosen and described in order to best explain the principles of the invention, the practical application, and to enable others skilled in the art to best utilize the invention in various embodiments with various modifications as are suited to the particular use contemplated. Therefore, it is to be understood that the claims are not limited to the specific examples of the disclosed embodiments and that modifications and other embodiments are intended to be included within the scope of the appended claims.

Explanation of Signs

[0235] 1 Key Point 17 Key Points 18 Key Points 24 Key Points 77 Key Points 78 Key Points 81 Key Points 82 Key Points 202 Input Image 204 Initial Coarse Position Map 206 Key Point 208 2D (Two-Dimensional) Key Point Annotation 210 Spatial Transformation Mapping 212 Transformed Position Map 214 Refinement Process 216 Final Position Map 218 Final Key Point 302 Input Image 304 Initial Coarse Position Map 306 Key Point 308 2D Key Point Annotation 310 Spatial Transformation Mapping 312 Transformed Position Map 402 Circle 404 Circle 406 yen 408 yen 502 image 504 image 506 image 508 image 510 image 702 keypoint 704 keypoint 706 keypoint 708 keypoint 1000 flowchart 1202 image 1204 image 1206 image 1208 area 1302 Central eyebrow area 1402 area 1404 yen 1502 Hair pixel detection area 1504 Vertical height line 1506 distance 1602 Hair color extraction area 1608 starting point 1702 area 1704 line 1706 line 1802 column 1804 column 1806 column 1900 flowchart 2102 Head template model 2104 eye 2106 hair 2108 tooth 2110 face 2302 template model 2306 3D keypoint 2504 Bending weight rendering 2800 flowchart 2902 image 2904 image 2906 normalization 2910 Keypoint smoothing process 3002 keypoint 3004 keypoint 3006 Key Point 3008 Key Point 3010 Key Point 3012 Position 3102 Real Face Image 3104 Real Face Key Point 3106 Stylized Key Point 3108 Key Point-to-Parameter (K2P) Network 3110 Game Client 3112 Parameter 3114 Key Point 3116 Parameter-to-Key Point (P2K) Network 3202 Original Face Image 3204 Game Face Avatar Image Generation 3206 Original Face Image 3208 Game Face Avatar Image Generation 3300 Flowchart 3400 Image Processing Device 3401 Processor 3402 Memory 3403 User Interface 3404 Network Interface 3405 Bus System 34021 Executable Program 34022 Operating System

Claims

Claim 1. A method for customizing a standard face of an avatar using a target two-dimensional (2D) face image applicable to a game, comprising: identifying a set of target key points within the 2D face image; converting the set of target key points into a set of avatar key points associated with the avatar; generating a set of face control parameters for the standard face by applying a keypoint-to-parameter (K2P) neural network model to the set of avatar key points, wherein the set of face control parameters are control parameters for deforming the standard face, and each parameter included in the set of face control parameters is associated with one of a plurality of face features of the standard face; and customizing the plurality of face features of the standard face according to the style of the game by applying the set of face control parameters to the standard face. Claim 2. The K2P neural network model: obtains a plurality of training 2D face images of a target; generates a set of training avatar key points associated with the avatar for each of the plurality of training 2D face images; presents each set of training avatar key points to the K2P neural network model to obtain the set of face control parameters; presents the set of face control parameters to a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a set of predicted avatar key points corresponding to the set of training avatar key points; and is trained by updating the K2P neural network model by reducing a difference between the set of training avatar key points and the corresponding set of predicted avatar key points. The method according to claim 1. Claim 3. The pre-trained P2K neural network model: receives a set of face control parameters including bone or slider parameters associated with the avatar; and is configured to predict a set of avatar key points of the avatar according to the set of face control parameters. The method according to claim 2. Claim 4. The method according to claim 3, wherein the difference between the set of the training avatar key points and the set of the corresponding predicted avatar key points is the sum of the mean squared errors between the set of the training avatar key points and the set of the corresponding predicted avatar key points.

5. The method according to claim 3, wherein the trained K2P and the pre-trained P2K neural network models are associated with a game.

6. The method according to claim 1, wherein the set of the target key points in the 2D face image corresponds to the face features of the target in the 2D face image.

7. The method according to claim 1, wherein the standard face of the avatar is customized for different characters in the game according to face images of different targets.

8. The method according to claim 1, wherein the customized standard face of the avatar is the face of the target in a comic style.

9. The method according to claim 1, wherein the customized standard face of the avatar is the face of the target in a realistic style.

10. The step of converting the set of the target key points into the set of the avatar key points comprises the step of normalizing the set of the target key points into a canonical space, the step of symmetrizing the normalized set of the target key points, and the step of adjusting the symmetrized set of the target key points according to a predetermined style associated with the avatar to obtain the set of the avatar key points, the method according to claim 1.

11. The step of normalizing the set of the target key points into a canonical space comprises the step of scaling the set of the target key points into the canonical space, and the step of rotating the scaled set of the target key points according to the orientation of the set of the target key points in the 2D face image, the method according to claim 10.

12. The step of converting the set of the target key points into the set of the avatar key points further comprises the step of smoothing the symmetrized set of the target key points so as to meet a predetermined convex or concave curve requirement, the method according to claim 10.

13. The step of adjusting the set of the symmetricized target key points according to the predetermined style associated with the avatar includes one or more of face length adjustment, face width adjustment, face feature adjustment, zoom adjustment, and eye shape adjustment, according to claim 10.

14. An electronic device configured to execute the method according to any one of claims 1 to 13.

15. A program including a plurality of instructions that cause an electronic device having one or more processing units to execute the method according to any one of claims 1 to 13 when executed by the electronic device.

Images