Method and device for generating three-dimensional head deformation model

Abstract

This application provides a method and apparatus for generating a 3D deformable head model. The method includes: receiving a two-dimensional (2D) facial image; identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; mapping the first set of key points to a second set of key points located on multiple vertices of a mesh of the 3D head template model based on a set of user-provided key point annotations located on a 3D head template model; deforming the mesh of the 3D head template model by reducing the difference between the first set of key points and the second set of key points to obtain a deformed 3D head mesh model; and applying a hybrid shape method to the deformed 3D head mesh model to obtain a head model based on the 2D facial image.

Description

[0001] Cross-references to related applications

[0002] This application is a continuation-to-file of U.S. Patent Application No. 17 / 202,112, entitled “METHODS AND SYSTEMS FOR PERSONALIZED 3D HEAD MODELDEFORMATION”, filed March 15, 2021, and claims priority to that U.S. Patent Application, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to image processing techniques, and more specifically, to a method and apparatus, electronic device and computer-readable medium for generating a three-dimensional deformable head model. Background Technology

[0004] Commercial face capture systems with multiple sensors (e.g., multi-view cameras, depth sensors, etc.) are used to obtain accurate three-dimensional (3D) facial models of people, with or without explicit markings. These tools capture the geometry and texture information of the human face from multiple sensors and fuse multimodal information into a general 3D facial model. Benefiting from multimodal information from various sensors, the obtained 3D facial model is accurate. However, these commercial systems are expensive and require additional software purchases to process the raw data. Furthermore, these systems are typically deployed in face capture studios, requiring participants or volunteers to collect data, making the data collection process time-consuming and potentially more costly. In short, acquiring 3D facial data using face capture systems is expensive and time-consuming. Conversely, smartphones or camera devices are now widely available, thus potentially providing a large amount of RGB (red, green, blue) images. Using RGB images as input to generate 3D facial models can benefit from a large amount of image data.

[0005] Two-dimensional (2D) RGB images are merely projections of the 3D world onto a 2D plane. Reconstructing 3D geometry from a 2D image is an ill-posed problem, requiring optimization or learning algorithms to fine-tune the reconstruction process. For 3D facial reconstruction, methods based on parametric facial models (3D Morphable Models, 3DMMs) have been developed and used. Specifically, facial models like the Basel Face Model (BFM) and the Surrey Face Model (SFM) are commonly used, but they require commercial licensing. Face model-based methods use a set of scanned 3D human facial models (demonstrating various facial features and expressions) as their foundation, then generate parametric representations of facial features and expressions based on these 3D facial models. New 3D faces can be represented as linear combinations of the underlying 3D facial models based on parametric representations. Due to the nature of these methods, the 3D facial models used to form the foundation and the parameter space limit the expressiveness of face model-based methods. Furthermore, the optimization process of fitting 3DMM parameters based on the input facial image or 2D feature points sacrifices detailed facial features in the facial image. Therefore, facial model-based methods cannot accurately recover 3D facial features and require commercial licensing to use facial models such as BFM and SFM.

[0006] With the popularization of deep learning algorithms, semantic segmentation algorithms have received a lot of attention. Such algorithms can divide each pixel in a facial image into different categories such as background, skin, hair, eyes, nose, and mouth.

[0007] While semantic segmentation methods can achieve relatively accurate results, segmenting all pixels is a highly complex problem, typically requiring intricate network structures and resulting in high computational complexity. Furthermore, training a semantic segmentation network necessitates labeling a large amount of training data, and semantic segmentation itself requires dividing the entire image into pixels—a tedious, time-consuming, and costly process. Therefore, semantic segmentation is unsuitable for scenarios where high efficiency is required but high average color accuracy is not necessary.

[0008] Keypoint-driven deformation methods for optimizing Laplace and other derived operators have been well-studied in academia. The mathematical expression for biharmonic deformation can be represented as Δ 2 x′=0. The key point under constraint, i.e., the boundary condition, can be expressed as x b ′=x bc In the above equation, Δ is the Laplace operator, x′ is the position of the unknown deformable mesh vertex, and x bcIt represents the location of a given key point after deformation. A solution to the two Laplace equations is needed in each dimension. The biharmonic function is a solution to the two Laplace equations, but also a minimizer of the so-called "Laplace energy".

[0009] The essence of energy minimization is mesh smoothing. If the aforementioned minimization is applied directly, all detailed features will be smoothed away. Furthermore, when the positions of keypoints remain unchanged, the deformed mesh is expected to be identical to the original mesh. For these reasons, biharmonic deformation is preferred to solve for the displacements of vertices rather than their positions. In this way, the deformed position can be written as x′=x+d, where d is the displacement of the unknown vertex in each dimension. Naturally, the equations for biharmonic deformation are derived in d. b =x bc -x b The change is Δ 2 d = 0, where d b It is the displacement of the key points after deformation.

[0010] With the rapid development of the gaming industry, customized facial avatar generation has become increasingly popular. For the average player without artistic skills, it is extremely difficult to generate a face that can describe subtle changes by tuning and controlling parameters.

[0011] In some existing face generation systems and methods, such as the Justice Face Generation System, the prediction of the facial model involves predicting 2D information from an image, such as segmenting eyebrows, mouth, nose, and other pixels in a photograph. These 2D segments are susceptible to out-of-plane rotation and partial occlusion, and essentially require a frontal face. Furthermore, since the similarity between the final game facial avatar and the input is determined by a facial recognition system, this method limits its application to only realistic-style games. If the game's style is cartoonish, which is very different from a realistic face, this method cannot be used.

[0012] In some existing face generation systems and methods, such as the Moonlight Blade Face Generation System, a realistic face is reconstructed from an input image. This method is limited to realistic games and cannot be applied to cartoon-style games. Furthermore, the output parameter of this method is a reconstructed game-style facial mesh, and then template matching is performed on each part of the mesh. This method restricts the combination of different facial features. The overall diversity of game faces is closely related to the number of pre-generated templates. If a feature, such as the mouth shape, has few templates, it may produce very few variations, resulting in a lack of diversity in the generated faces.

[0013] In existing technologies, head model generation systems often have high requirements for input images, and head meshes are difficult to deform appropriately without being bound to the skeleton, increasing the workload for artists. In addition, existing methods are difficult to adapt to different styles of meshes, resulting in a trade-off between versatility and customization capabilities. Summary of the Invention

[0014] Learning-based facial reconstruction and keypoint detection methods rely on 3D ground truth data as the gold standard to train models that are as close as possible to the ground truth. Therefore, the 3D ground truth determines the upper limit of learning-based methods. To ensure the accuracy of facial reconstruction and the desired keypoint detection, some implementations use 2D facial keypoint annotations to generate a ground truth 3D facial model without using expensive facial capture systems. The method disclosed in this paper generates a ground truth 3D facial model that preserves detailed facial features of the input image, overcoming the shortcomings of existing facial models that lose facial features (e.g., 3DMM-based methods), and also avoids the use of parametric facial models such as BFM and SFM (both of which require commercial licensing), which are necessary for some existing facial model-based methods.

[0015] In addition to facial landmark detection, some implementations employ multi-task learning and transfer learning solutions for facial feature classification tasks, enabling the extraction of more information from the input facial image, which is complementary to the landmark information. The detected facial landmarks, together with the predicted facial features, are valuable for computer or mobile games used to create facial avatars of players.

[0016] In some implementations, this document discloses lightweight methods for extracting the average color of each part of a human face from a single photograph, including extracting the average color of skin, eyebrows, pupils, lips, hair, and eyeshadow. Simultaneously, algorithms are used to automatically transform texture maps based on the average color, such that the transformed texture retains its original brightness and color difference, but the primary color becomes the target color.

[0017] With the rapid development of computer vision and artificial intelligence (AI) technologies, the capture and reconstruction of 3D human facial key points has reached a high level of accuracy. More and more games are utilizing AI detection to make game characters more lifelike. The method and system disclosed in this paper are based on reconstructed 3D key points to customize 3D head avatars. General key point-driven deformation is applicable to arbitrary meshes. The head avatar customization process and deformation method proposed in this paper can be applied to scenarios such as automated avatar creation and facial expression reproduction.

[0018] This paper discloses a method and system for automatically generating facial avatars in games based on a single photograph. By using deep learning methods for predicting facial key points, automatically processing these key points, and optimizing the prediction model parameters, the system can automatically generate facial avatars in games that: 1) possess the characteristics of a realistic face from a photograph; and 2) conform to the target game style. This system can be applied to facial generation in both realistic and cartoon-style games and can be easily and automatically adjusted according to different game models or skeletal definitions.

[0019] According to a first aspect of this application, a method for constructing a facial location map based on a two-dimensional (2D) facial image of a real person includes: generating a coarse facial location map from the 2D facial image; predicting a first set of key points in the 2D facial image based on the coarse facial location map; identifying a second set of key points in the 2D facial image based on user-provided key point annotations; and updating the coarse facial location map to reduce the difference between the first set of key points and the second set of key points in the 2D facial image.

[0020] In some implementations, the method for constructing a facial location map based on a 2D facial image of a real person further includes: extracting a third set of key points as a final set of key points based on the updated facial location map, wherein the third set of key points has the same position in the facial location map as the first set of key points.

[0021] In some implementations, the method of constructing a facial location map based on a 2D facial image of a real-life person further includes: reconstructing a three-dimensional (3D) facial model of a real-life person based on the updated facial location map.

[0022] According to a second aspect of this application, a method for extracting color from a two-dimensional (2D) facial image of a real-life person includes: identifying multiple keypoints in the 2D facial image based on a keypoint prediction model; rotating the 2D facial image until selected keypoints among the multiple keypoints are aligned; locating multiple parts in the rotated 2D facial image, wherein each part is defined by a corresponding subset of the multiple keypoints; extracting the average color of each of the multiple parts defined by the corresponding subset of keypoints based on pixel values ​​of the 2D facial image; and using the extracted colors of the multiple parts in the 2D facial image to generate a personalized three-dimensional (3D) model of a real-life person, the personalized 3D model being matched with the corresponding facial feature colors of the 2D facial image.

[0023] According to a third aspect of this application, a computer-implemented method for generating a three-dimensional (3D) deformable head model includes: receiving a two-dimensional (2D) facial image; identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; mapping the first set of key points to a second set of key points located on a plurality of vertices of a mesh of the 3D head template model based on a set of user-provided key point annotations located on the 3D head template model; performing deformation on the mesh of the 3D head template model by reducing the difference between the first set of key points and the second set of key points to obtain a deformed 3D head mesh model; and applying a blendshape method to the deformed 3D head mesh model to obtain a head model based on the 2D facial image.

[0024] According to a fourth aspect of this application, a method for customizing the standard face of an avatar in a game using a two-dimensional (2D) facial image of a real-life person includes: identifying a set of real-life keypoints in the 2D facial image; transforming the set of real-life keypoints into a set of game-style keypoints associated with the avatar in the game; generating a set of control parameters for the standard face of the avatar in the game by applying the set of game-style keypoints to a keypoint-to-parameter (K2P) neural network model; and deforming the standard face of the avatar in the game based on the set of control parameters, wherein the deformed face of the avatar has facial features of the 2D facial image.

[0025] According to a fifth aspect of this application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When executed by one or more processing units, the programs cause the electronic device to perform one or more methods as described above.

[0026] According to a sixth aspect of this application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. When executed by one or more processing units, the programs cause the electronic device to perform one or more methods as described above.

[0027] Additionally, this application provides an apparatus for generating a 3D deformable head model, characterized in that the apparatus comprises: a receiving module for receiving a 2D facial image; a recognition module for recognizing a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; a mapping module for mapping the first set of key points to a second set of key points located on a plurality of vertices of a mesh of the 3D head template model based on a set of user-provided key point annotations located on a 3D head template model; a deformation module for deforming the mesh of the 3D head template model by reducing the difference between the first set of key points and the second set of key points to obtain a deformed 3D head mesh model; and a blending shape module for applying a blending shape method to the deformed 3D head mesh model to obtain a head model based on the 2D facial image.

[0028] Note that the various embodiments described above can be combined with any other embodiments described herein. Not all features and advantages described in the specification are included, and in particular, many additional features and advantages will be apparent to those skilled in the art in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and guidance purposes, and not for depicting or limiting the subject matter of the invention.

[0029] In summary, this application provides a method and apparatus for generating a three-headed deformable model, enabling the head mesh to deform appropriately without being bound to the skeleton. This significantly reduces the workload required by artists and allows for adaptation to different mesh styles for better versatility. The above method and apparatus have low requirements for input images, are suitable for both real-world and cartoon games, and also offer good lightweight and customization capabilities. Attached Figure Description

[0030] To gain a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and should not be considered limiting, as other features that produce practical effects may also be included in the specification.

[0031] Figure 1 This is a diagram illustrating exemplary key point definitions according to an embodiment of this application.

[0032] Figure 2 This is a block diagram illustrating an exemplary keypoint generation process according to an embodiment of this application.

[0033] Figure 3 This is a diagram illustrating an exemplary process of transforming an initial coarse position map according to an embodiment of this application.

[0034] Figure 4 This is a diagram illustrating an exemplary transformed positional view that does not cover the entire facial area according to an embodiment of this application.

[0035] Figure 5 This is a diagram illustrating an exemplary process of refining a transformed position map to cover the entire facial region according to an embodiment of this application.

[0036] Figure 6 This is a diagram illustrating some exemplary results of a location map refinement algorithm according to an embodiment of this application.

[0037] Figure 7A and Figure 7B Some exemplary comparisons of a final location map and an initial rough location map according to an embodiment of this application are shown.

[0038] Figure 8A This is a diagram illustrating an exemplary eyeglasses classification network structure according to an embodiment of this application.

[0039] Figure 8B This is a diagram illustrating an exemplary female hair prediction network structure according to an embodiment of this application.

[0040] Figure 8C This is a diagram illustrating an exemplary male hair prediction network structure according to an embodiment of this application.

[0041] Figure 9A Some exemplary eyeglass classification prediction results according to an embodiment of this application are shown.

[0042] Figure 9B Some exemplary female hair prediction results according to an embodiment of this application are shown.

[0043] Figure 9C Some exemplary male hair prediction results according to an embodiment of this application are shown.

[0044] Figure 10 This is a flowchart illustrating an exemplary process of constructing a facial position map based on a 2D facial image of a real person according to an embodiment of this application.

[0045] Figure 11This is a flowchart illustrating an exemplary color extraction and adjustment process according to an embodiment of this application.

[0046] Figure 12 An exemplary skin color extraction method according to an embodiment of this application is shown.

[0047] Figure 13 An exemplary eyebrow color extraction method according to an embodiment of this application is shown.

[0048] Figure 14 An exemplary pupil color extraction method according to an embodiment of this application is shown.

[0049] Figure 15 An exemplary hair color extraction area is shown in a hair color extraction method according to an embodiment of this application.

[0050] Figure 16 An exemplary separation between hair pixels and skin pixels within a hair color extraction region is shown according to an embodiment of this application.

[0051] Figure 17 An exemplary eyeshadow color extraction method according to an embodiment of this application is shown.

[0052] Figure 18 Some exemplary color adjustment results according to an embodiment of this application are shown.

[0053] Figure 19 This is a flowchart illustrating an exemplary process for extracting colors from a 2D facial image of a real person according to an embodiment of this application.

[0054] Figure 20 This is a flowchart illustrating an exemplary head avatar transformation and generation process according to an embodiment of this application.

[0055] Figure 21 This is a diagram illustrating an exemplary header template model composition according to an embodiment of this application.

[0056] Figure 22 This is a diagram showing some exemplary key point markings on a realistic-style 3D model and a cartoon-style 3D model according to an embodiment of this application.

[0057] Figure 23 This is a diagram illustrating an exemplary comparison between template model rendering, manually marked key points, and AI-detected key points according to an embodiment of this application.

[0058] Figure 24 This is a diagram illustrating an affine transformation of an exemplary triangle according to an embodiment of this application.

[0059] Figure 25 This is an exemplary comparison of the deformation results of some head models with and without a shape blending process according to an embodiment of this application.

[0060] Figure 26 This is a diagram illustrating an exemplary comparison of affine deformation and biharmonic deformation with different weights according to an embodiment of this application.

[0061] Figure 27 The present invention illustrates some exemplary results generated automatically from randomly selected images of women using a realistic template model according to an embodiment of the present application.

[0062] Figure 28 This is a flowchart illustrating an exemplary process for generating a 3D deformable head model based on a 2D facial image of a real person according to an embodiment of this application.

[0063] Figure 29 This is a diagram illustrating exemplary key point processing steps according to an embodiment of this application.

[0064] Figure 30 This is a diagram illustrating an exemplary keypoint smoothing process according to an embodiment of this application.

[0065] Figure 31 This is a block diagram illustrating an exemplary keypoint to control parameter (K2P) conversion process according to an embodiment of this application.

[0066] Figure 32 Some exemplary results of automatic face generation in a mobile game according to an embodiment of this application are shown.

[0067] Figure 33 This is a flowchart illustrating an exemplary process of customizing the standard face of an avatar in a game using a 2D facial image of a real-life person, according to an embodiment of this application.

[0068] Figure 34 This is a schematic diagram of an exemplary hardware structure of an image processing apparatus according to an embodiment of this application.

[0069] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily enlarged or reduced. Furthermore, some drawings may not depict all parts of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0070] Reference will now be made to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices.

[0071] Before further describing the embodiments of this application in detail, the names and terms involved in the embodiments of this application are described, and the names and terms involved in the embodiments of this application are explained as follows.

[0072] Facial key points: Predefined feature points that determine the shape of certain facial features, such as the corners of the eyes, chin, tip of the nose, and corners of the mouth.

[0073] Facial features: facial contours, eyes, eyebrows, nose, mouth, and other areas.

[0074] Facial reconstruction: Reconstructing the 3D geometry of the human face and common representations including mesh models, point clouds, or depth maps.

[0075] RGB image: a three-channel image format consisting of red, green, and blue channels.

[0076] Location map: The red, green, and blue channels in a standard image format are used to store the x, y, and z coordinates of the facial region, which is a 3D representation of the human face.

[0077] Facial feature classification: including hairstyle classification and whether or not glasses are worn.

[0078] Convolutional neural network (CNN): A type of deep neural network, most commonly used for analyzing visual images.

[0079] Base network: A CNN-like network that is used by one or more downstream tasks to act as a feature extractor.

[0080] Laplace operator: A differential operator given by the divergence of the gradient of a function on Euclidean space.

[0081] Differentiable manifold: A topological space that is locally similar to a linear space to allow for calculus.

[0082] Biharmonic function: A quaternary differentiable function defined on a differentiable manifold whose squared Laplace operator equals 0.

[0083] Keypoint-driven deformation: a class of methods that deform a mesh by changing the position of certain vertices.

[0084] Double harmonic deformation: A deformation method that optimizes the double harmonic function by applying certain boundary conditions.

[0085] Affine deformation: The keypoint-driven deformation method proposed in this disclosure optimizes the affine transformation of triangles to achieve mesh deformation.

[0086] Facial model: A predefined mesh of standard faces in the target game.

[0087] Bone / Slider: Control parameters used to deform the facial model.

[0088] As mentioned above, even when both the input 2D image and 2D keypoints are fed into the optimization process to fit the 3DMM parameters, the optimization must find a balance between fitting the 3D facial model based on the foundation (i.e., the ensemble of 3D facial models) and the fidelity of the 2D keypoints. This optimization results in a 3D facial model that ignores the 2D input keypoints, sacrificing the detailed facial information provided by the input 2D keypoints. In existing 3D facial reconstruction methods, facial capture solutions can produce accurate reconstructions, but these are expensive and time-consuming, and the obtained data also shows limited variation in facial features (a limited number of participants). On the other hand, facial model-based methods can use 2D images or 2D feature point annotations as input, but the resulting 3D model is inaccurate. To meet the demands of the rapid development of computer / mobile games, there is a need to produce the desired 3D model accuracy while reducing the required cost and time. To meet these requirements, the novel 3D ground truth face model generation algorithm disclosed in this paper takes a 2D image, 2D keypoint annotations, and a coarse 3D face model (location map format) as input, transforms the coarse 3D model based on the 2D keypoints, and finally produces a 3D face model in which detailed facial features are well preserved.

[0089] Beyond addressing key challenges in facial reconstruction and keypoint prediction, this paper discloses a multi-task learning and transfer learning-based approach for facial feature classification, partially built upon the facial reconstruction and keypoint prediction framework. Specifically, glasses classification (with or without glasses) is achieved through multi-task learning by reusing the underlying network for facial reconstruction and keypoint prediction. A linear classifier trained on top of the existing facial reconstruction and keypoint prediction framework significantly reuses the existing model and avoids introducing another large network for image feature extraction. Furthermore, another shared underlying network is used for male and female hairstyle classification. Hairstyle is an important facial feature that complements facial keypoints or 3D facial models. Adding hairstyle and glasses prediction during the creation of 3D avatars for users better reflects their facial features and provides a more personalized experience.

[0090] Facial landmark prediction has been a research topic in computer vision for decades. With the development of artificial intelligence and deep learning in recent years, convolutional neural networks (CNNs) have facilitated progress in facial landmark prediction. 3D face reconstruction and facial landmark detection are two intertwined problems, so solving one can simplify the other. The conventional approach is to first solve 2D facial landmark detection and then further infer a 3D facial model based on the estimated 2D facial landmarks. However, when the face in the image is tilted (nodding or shaking), some facial landmarks are occluded, leading to incorrect 2D facial landmark estimations. Therefore, the 3D facial model built on these incorrect 2D facial landmarks becomes inaccurate.

[0091] Ground-value data sets the upper limit for deep learning-based methods, but existing 3D face model datasets are not only limited in number but also only available for academic research. On the other hand, face model-based methods require the use of Basel Face Models (BFM) or Surrey Face Models (SFM), both of which require commercial licensing. High accuracy and a large volume of 3D ground-value data become paramount when training any face reconstruction or keypoint estimation model.

[0092] Besides facial landmark prediction, facial feature classification is a crucial aspect of creating a user's 3D avatar. Based on predicted facial landmarks, style transfer can only be performed on the user's facial features (i.e., eyes, eyebrows, nose, mouth, and facial contours). However, to better reflect the user's facial features, matching the user's hairstyle to the input image and adding glasses if the user is wearing them is very helpful. Based on these requirements, a facial feature classification method based on multi-task learning and transfer learning was developed to achieve male / female hairstyle prediction and glasses prediction (with or without glasses). This allows for a more personalized facial avatar, improving the user experience.

[0093] In some implementations, to represent the three-dimensional shape of the main facial features, methods such as... Figure 1 The key points are shown below. Figure 1 This is a diagram illustrating exemplary keypoint definitions according to some implementations of this disclosure. Keypoints are numbered sequentially to define specific facial features. In other words, there is a mapping between the keypoint numbers and specific locations on the face. For example, number 9 corresponds to the base of the chin, and number 21 corresponds to the tip of the nose, etc. Keypoints are concentrated on the boundaries of major facial features (e.g., facial contours, eye contours, and eyebrow contours). More keypoints mean greater difficulty in prediction, but more accurate shape representation. In some implementations, in Figure 1 The code uses a definition of 96 key points. In some implementations, users can modify the number and specific definitions of key points according to their own needs.

[0094] Many algorithms can predict the 3D coordinates of key points on a human face. Methods with better performance use deep learning algorithms based on large amounts of offline 3D training data. However, in some implementations, any 3D keypoint prediction algorithm can be used. In some implementations, the definition of keypoints is not fixed, and users can customize the definition according to their needs.

[0095] To address the problem of generating 3D ground truth face models, the following automated algorithm was developed that takes 2D RGB images, 2D keypoint annotations, and a coarse location map as input. Figure 2 This is a block diagram illustrating an exemplary keypoint generation process according to some implementations of this disclosure. For example, a 2D RGB image of a face is used as input image 202, and the 2D RGB image has a corresponding initial coarse location map 204, where each pixel in the initial coarse map represents the spatial coordinates of the corresponding facial point in the 2D RGB image. 2D keypoint annotations 208 represent a set of user-provided keypoints used to correct a set of keypoints 206 detected from the initial coarse map 204.

[0096] Figure 3 This is a diagram illustrating an exemplary process of transforming an initial rough positional diagram according to some implementations of this disclosure.

[0097] In some implementations, a 3D reconstruction method is used to convert the input facial image into a location map containing 3D depth information of facial features. For example, the location map can be a 2D three-color (RGB) channel map with a 256×256 matrix array, and each array element has coordinates (x, y, z) representing a 3D position on the facial model. The 3D position coordinates (x, y, z) are represented by the RGB pixel values ​​of each array element on the location map. Specific facial features are located at fixed 2D positions within the 2D location map. For example, the tip of the nose can be identified by the positions of 2D array elements at X=128 and Y=128 within the location map. Similarly, specific keypoints for identifying specific facial features on the face can be located at the same array element positions on the 2D location map. However, for the location map, depending on different input facial images, specific keypoints can have different 3D position coordinates (x, y, z).

[0098] In some implementations, such as Figure 2 and Figure 3 As shown, a 3D reconstruction method is used to obtain an initial coarse location map (204, 304) based on the input image (202, 302). Then, the (x, y) coordinates of the corresponding keypoints (206, 306) in the initial location map are adjusted using the input 2D keypoint annotations (208, 308) to ensure that the adjusted (x, y) coordinates of the keypoints in the adjusted location map are the same as those of the annotated 2D keypoints. Specifically, first, a set of 96 keypoints from the initial location map P is obtained. Based on the keypoint index, this set of 96 keypoints is called K = k_i, where each k_i is the 2D coordinate (x, y) of the keypoint, and i = 0, ..., 95. A second set of 96 keypoints A = a_i, which are 2D (x, y) coordinates, are obtained based on the 2D keypoint annotations (208, 308), and i = 0, ..., 95. Secondly, the spatial transformation mapping (210, 310) is estimated from K to A, defined as T: Ω → Ω, where Then, the obtained transformation T is applied to the initial position map P to obtain the transformed position map P'(212, 312). In this way, the transformed position map P'(212, 312) preserves the detailed facial features of the person in the input image (202, 302), and at the same time, the transformed position map P'(212, 312) has reasonable 3D depth information. Therefore, the solution disclosed in this paper provides an accurate and practical alternative solution for generating 3D ground truth information to avoid using expensive and time-consuming face capture systems.

[0099] In some implementations, since the 96 facial key points only cover a portion of the entire facial area (i.e., below the eyebrows, inside the facial contour), for example, in Figure 3In the image, keypoints from the ear to the chin follow the jawline but are not on the visible facial contour. When the face in the input image is tilted, the entire facial region is not covered by the contours of the connected keypoints. Furthermore, when performing manual keypoint annotation, regardless of whether the face in the image is tilted, keypoints can only be marked along the visible facial contour (i.e., occluded keypoints cannot be accurately annotated). Therefore, in the transformed position map P'(212, 312), this region has no valid values ​​because the transformation map T(210, 310) has no estimate in a portion of the facial region. Additionally, the forehead region is above the eyebrows, so T also has no estimate in this region. All these issues cause the transformed position map P'(212, 312) to have no valid values ​​in certain regions. Figure 4 This is a diagram illustrating an exemplary transformed positional diagram that does not cover the entire facial region, according to some implementations of this disclosure.

[0100] exist Figure 4 In the middle, the top circle (402, 406) highlights the forehead area, and the right circle (404, 408) indicates the area where the key point outline is smaller than the visible facial outline.

[0101] In some implementations, to address the aforementioned problems and make the algorithm robust to tilted faces, which are commonly found in facial images, methods such as... Figure 2 The refinement process 214 is shown. Based on the head pose and a coarse 3D facial model, keypoints from the transformed location map are shifted along the facial contour to match the visible facial contour. Subsequently, missing values ​​in the facial contour regions can be filled in the obtained location map. However, values ​​in the forehead region remain missing. To cover the forehead region, control points are expanded by adding eight feature points from the four corners of the image to two keypoint sets K and A.

[0102] Figure 5 This diagram illustrates an exemplary process for refining a transformed position map to cover the entire facial region, according to some implementations of this disclosure. Figure 5 The image shows a refined version of the location map.

[0103] In some implementations, first, the head pose is determined based on a coarse position map P to determine whether the head is tilted to the left or right, and the left or right is defined in the 3D facial model space (e.g., as shown in the image). Figure 5 As shown, the face is tilted to the left. Based on the determination of whether the face is tilted to the left or right, the key points on the corresponding side of the facial contour are adjusted. The key points on the right side of the facial contour have indices from 1 to 8, and the key points on the left side of the facial contour have indices from 10 to 17. Using a face tilted to the left as an example, the 2D projection of the initial position map P is calculated to obtain... Figure 5 The depth map of image 502 is shown in the figure. The left facial contour keypoints k_i, i = 10, ..., 17 are shifted to the right until they reach the boundary of the depth map. Then, the new coordinates are used to replace the original keypoint positions. Similarly, when the face is tilted to the right, the processed keypoints are indexed by k_i, i = 1, ..., 8, and the search direction is to the left. After adjusting the facial contour keypoints, the updated keypoints are visualized as follows: Figure 5 Image 504 in the image, and the updated overlay of the location map is shown as... Figure 5 Image 506 in the image. The updated location map shows better facial coverage in the facial contour region, but the forehead region still has missing values.

[0104] In some implementations, to cover the forehead region, two anchor points are added at each corner of the image domain Ω as additional keypoints k_i, i = 96, ..., 103, to obtain an updated set of keypoints K' (e.g., ...). Figure 5 (See image 508). The same operation is performed on the manually annotated keypoint set a_i, i = 96, ..., 103 to obtain the updated A'. Using the updated keypoint sets K' and A', the transformation map T' is re-estimated and then applied to the initial location map P to obtain the final location map P"( Figure 2 (216 in the middle), thus covering the entire facial area (such as 216). Figure 5 (As shown in image 510). The final key point 218 is derived from the final position diagram 216.

[0105] Figure 6 These are diagrams illustrating some exemplary results of a position map refinement algorithm according to some implementations of the present disclosure. 602 is a diagram of the initial transformed position map. 604 is a diagram of the updated position map after correcting the facial contours. 606 is a diagram of the final position map.

[0106] Figure 7A and Figure 7B Some exemplary comparisons are shown between final location maps and initial coarse location maps of some implementations according to this disclosure. Figure 7A In one example, the nose in the initial location map and its associated 3D model and keypoint 702 is incorrect and does not reflect the person's facial features at all (highlighted by the arrow). However, after applying the method described in this paper, the nose is perfectly aligned with the image in the final location map and its associated 3D model and keypoint 704 (highlighted by the arrow). Figure 7BIn the second example, several inaccuracies exist in the initial location map and its associated 3D model and keypoint 706, such as mismatched facial contours, open mouth, and nose shape (indicated by arrows). In the final location map and its associated 3D model and keypoint 708, all these errors are corrected (indicated by arrows).

[0107] Hairstyles and glasses categories are important for the facial avatar creation process in mobile game applications. In some implementations, this paper presents a solution based on multi-task learning and transfer learning to address these issues.

[0108] In some implementations, four different classification tasks (head) are performed for predicting female hair. The classification categories and parameters are shown below:

[0109] Category Head 1: Curved

[0110] Straight (0); Curved (1)

[0111] Category Header 2: Length

[0112] Short (0); Long (1)

[0113] Category Header 3: Bangs

[0114] No bangs or no parting (0); Left parting (1); Right parting (2); M-shape (3); Straight bangs (4); Natural bangs (5); Airy bangs (6)

[0115] Category 4: Braids

[0116] A single braid (0); two or more braids (1); a single bun (2); two or more buns (3); other (4).

[0117] In some implementations, three different classification tasks (head) are performed for male hair prediction. The classification categories and parameters are shown below:

[0118] Classification of head 1: Very short (0), curly (1), other (2)

[0119] Category 2: No bangs (0), Middle-parted bangs (1), Natural bangs (2)

[0120] Category 3: Left-parted bangs (0), and right-parted bangs (1)

[0121] In some implementations, glasses classification is a binary classification task. The classification parameters are as follows:

[0122] No glasses (0); with glasses (1).

[0123] Among various deep learning image classification models, those that achieve state-of-the-art accuracy on ImageNet (e.g., EfficientNet, Noisy Student, and FixRes) typically have large model sizes and complex architectures. When deciding which architecture to use as the base network for the feature extractor, a balance must be struck between prediction accuracy and model size. In practice, a 1% improvement in classification accuracy may not yield a noticeable change for the end user, but the model size will increase exponentially. Given that trained models may need to be deployed on the client side, a smaller base network allows for flexible deployment on both the server and client sides. Therefore, MobileNetV2, for example, is used as the base network for transfer learning on different classification heads. The MobileNetV2 architecture is based on an inverted residual structure, where the input and output of the residual block are thin bottleneck layers, the opposite of traditional residual models that use expanded representations in the input. MobileNetV2 uses lightweight depthwise convolutions to filter features in intermediate expanded layers.

[0124] For glasses classification, a multi-task learning approach is used. The keypoint prediction network is used again as the base network, and the parameters are frozen. A binary classifier is trained using feature vectors with cross-entropy loss in the bottleneck layer of the U-shaped network. Figure 8A This is a diagram illustrating an exemplary glasses classification network structure according to some implementations of this disclosure. Figure 8B This is a diagram illustrating an exemplary female hair prediction network structure according to some implementations of this disclosure. Figure 8C This is a diagram illustrating an exemplary male hair prediction network structure according to some implementations of this disclosure.

[0125] Figure 9A Some exemplary glasses classification prediction results are shown in some implementations of this disclosure. Figure 9B Some exemplary female hair prediction results are shown in some implementations of this disclosure. Figure 9C Some exemplary male hair prediction results are shown in some implementations of this disclosure.

[0126] Figure 10This is a flowchart 1000 illustrating an exemplary process for constructing a facial position map from a 2D facial image of a real-life person, according to some implementations of the present disclosure. In real life, different people have different facial features, such that the same key points corresponding to the same facial features (e.g., the position of eyebrows on a person's face) can have very different spatial coordinates. Since the 2D facial images used to generate 3D facial models are captured at different angles and under different lighting conditions, the problem of facial detection becomes more challenging, and research in this area has been a very active topic in the field of computer vision. In this application, various methods have been proposed for improving the efficiency and accuracy of facial key point detection from any 2D facial image of a subject ranging from real-life people to cartoon characters. In some embodiments, a set of user-provided facial key points from the same facial image is provided as a reference for correcting or improving a set of facial key points initially detected by a computer-implemented method. For example, since there is a one-to-one mapping between user-provided facial key points and computer-generated facial key points based on their respective indices, the refinement of computer-generated facial key points is defined as an optimization problem of reducing the differences between the two sets of facial key points (e.g., measured by their corresponding spatial coordinates in the location map).

[0127] The process of constructing a facial location map includes step 1010: generating a rough facial location map from a 2D facial image.

[0128] The process also includes step 1020: predicting the first set of key points in the 2D facial image based on a rough facial location map.

[0129] The process also includes step 1030: identifying a second set of key points in the 2D facial image based on user-provided key point annotations.

[0130] The process further includes step 1040: updating the coarse facial location map to reduce the difference between the first set of keypoints and the second set of keypoints in the 2D facial image. For example, by reducing the difference between the first set of keypoints and the second set of keypoints in the 2D facial image in terms of their corresponding spatial coordinates, the first set of keypoints in the 2D facial image based on the coarse facial location map is modified to be more similar to the second set of keypoints in the 2D facial image based on user-provided keypoint annotations, which is generally considered more accurate, and the modification of the first set of facial keypoints automatically triggers an update of the initial coarse facial location map from which the first set of keypoints was generated. The updated coarse facial location map can then be used to predict a more accurate set of keypoints based on the 2D facial image. It should be noted that the second set of keypoints in the 2D facial image based on user-provided keypoint annotations does not mean that it was done manually. Rather, the user can employ another computer-implemented method for performing annotations. In some implementations, although the number of key points in the second set (e.g., 10 to 20) is only a fraction of the number of key points in the first set (e.g., 96 or even higher), the fact that the second set of key points is more accurate contributes to the overall improvement of the first set of key points.

[0131] In one implementation, the process further includes step 1050: extracting a third set of keypoints as the final set of keypoints based on the updated facial location map / final location map, wherein the third set of keypoints has the same position in the facial location map as the first set of keypoints. In some implementations, the positions of the keypoints in the facial location map are represented by the 2D coordinates of array elements in the location map. As described above, the updated facial location map benefits from the second set of keypoints in the 2D facial image based on user-provided keypoint annotations, thus the third set of keypoints is more accurate and can be used in fields such as computer vision for more accurate face detection or in fields such as computer graphics for more accurate 3D facial modeling.

[0132] In one implementation, in addition to or as an alternative to step 1050, the process further includes step 1060: reconstructing a 3D facial model of a real-life person based on the updated facial location map. In one example, the 3D facial model is a 3D depth model.

[0133] Additional implementations may include one or more of the following features.

[0134] In some implementations, the updated step 1040 may include: transforming the coarse facial location map into a transformed facial location map, and refining the transformed facial location map. As described above, compared to the initial coarse facial location map, the transformed facial location map can preserve more detailed facial features of the person in the input image, and therefore, the 3D facial model based on the transformed facial location map is more accurate.

[0135] In some implementations, the transformation includes: estimating a transformation mapping from a rough facial location map to a transformed facial location map based on the differences between a first set of keypoints and a second set of keypoints; and applying the transformation mapping to the rough facial location map.

[0136] In some implementations, the refinement includes adjusting key points on the occluded side of the facial contour corresponding to the transformed facial position map, based on the determination that the 2D facial image is tilted, to cover the entire facial region. As mentioned above, different 2D facial images can be captured at different angles, and this refinement step can correct for deviations or errors introduced by different image capture conditions, preserving a more accurate 3D facial model from the 2D facial image. Furthermore, compared to the initial coarse facial position map, the transformed facial position map can preserve more detailed facial features of the person in the input image, and therefore, the 3D facial model based on the transformed facial position map is more accurate.

[0137] In some implementations, the first set of key points may include 96 key points.

[0138] In some implementations, the process of constructing a facial location map may include facial feature classification.

[0139] In some implementations, facial feature classification is performed using deep learning methods.

[0140] In some implementations, facial feature classification is performed using multi-task learning or transfer learning methods.

[0141] In some implementations, facial feature classification includes hair prediction classification.

[0142] In some implementations, hair prediction classification includes female hair prediction with multiple classification tasks, which may include: curl, length, bangs, and braids.

[0143] In some implementations, hair prediction classification includes male hair prediction with multiple classification tasks, which may include: curl, length, bangs, and hair parting.

[0144] In some implementations, facial feature classification includes glasses prediction classification. Glasses prediction classification includes a classification task that may include: wearing glasses and not wearing glasses.

[0145] The method and system disclosed in this paper can generate accurate 3D facial models (i.e., location maps) based on 2D keypoint annotations used for 3D ground truth generation. This method not only avoids the use of BFM and SFM facial models, but also better preserves detailed facial features, thus preventing the loss of these important features caused by facial model-based methods.

[0146] In addition to providing key points, using deep learning-based solutions to provide supplementary facial features (such as hairstyle and glasses) is crucial for personalizing facial avatars based on user-input facial images.

[0147] While this paper presents hairstyle and glasses prediction as examples for facial feature classification, the framework is not limited to these example tasks. The framework and solution are based on multi-task learning and transfer learning, meaning it can be easily extended to include other facial features, such as female makeup type classification, male beard type classification, and masked / unmasked classification. The framework is designed to be easily extended to more tasks based on the needs of various computer or mobile games.

[0148] In some implementations, a keypoint-based light-weighted color extraction method is introduced in this paper. Light-weighted image processing algorithms quickly estimate local pixels without segmenting all pixels, resulting in higher efficiency.

[0149] During the training process, users do not need to have pixel-level markings, but only mark a few key points, such as the corners of the eyes, the borders of the mouth, and the eyebrows.

[0150] The light-weighted color extraction method disclosed in this paper can be used in personalized facial generation systems for various games. To provide more freedom in personalized character generation, many games have begun to adopt customizable methods. In addition to adjusting the shape of the face, users can also choose different color combinations. For aesthetic purposes, game faces often use predefined textures instead of real facial textures. The method and system disclosed in this paper allow users to automatically extract the average color of each part of the face simply by uploading a photo. Simultaneously, the system can automatically modify the texture based on the extracted colors to generate personalized facial features that more closely resemble the real colors in the user's photo, thereby improving the user experience. For example, if a user's skin is darker than the average skin tone, the character's skin in the game will be darkened accordingly. Figure 11 This is a flowchart illustrating an exemplary color extraction and adjustment process according to some implementations of this disclosure.

[0151] To locate the various parts of the face, as described above Figure 1As shown, keypoints are defined for the main feature areas of the face. The algorithm described above is used for keypoint prediction. Unlike semantic segmentation methods, it predicts keypoints only in the image without classifying each pixel, greatly reducing prediction costs and the need for labeled training data. Using these keypoints, various parts of the face can be roughly located.

[0152] Figure 12 An exemplary skin color extraction method according to some implementations of this disclosure is shown. In order to extract features from an image, the facial region in the original image 1202 needs to be rotated so that key points 1 and 17 on the left and right sides of the face are aligned with corresponding key points on the left and right sides of a standard face, as shown in the rotated and aligned image 1204.

[0153] Next, the area used for skin color pixel inspection is determined. The bottom coordinates of the key points of the eyes are selected as the upper boundary of the detection area, the bottom key points of the nose are selected as the lower boundary of the detection area, and the left and right boundaries are determined by the facial border key points. In this way, the skin color detection area is obtained as shown in region 1208 of image 1206.

[0154] Not all pixels in this region of 1208 are skin pixels; pixels may also include eyelashes, nostrils, nasolabial folds, hair, etc. Therefore, the median R, G, B values ​​of all pixels in this region are selected as the final predicted average skin color.

[0155] Figure 13 Exemplary eyebrow color extraction methods according to some implementations of this disclosure are illustrated. For the average eyebrow color, the primary eyebrow, i.e., the eyebrow closer to the camera side, is first selected as the target. In some implementations, if both eyebrows are primary eyebrows, then the eyebrow pixels on both sides are extracted. Assuming the left eyebrow is the primary eyebrow, as... Figure 13 As shown, a quadrilateral region consisting of keypoints 77, 78, 81, and 82 is selected as the eyebrow pixel search area. This is because the eyebrows near the outer edges are too sparse, and the impact of small keypoint errors will be amplified. Since the eyebrows near the inner edges are often sparse and blend with the skin color, the middle eyebrow region 1302 is selected to collect pixels. Each pixel must first be compared with the average skin color, and only pixels with a difference greater than a certain threshold are collected. Finally, similar to skin color, the median R, G, and B values ​​of the collected pixels are selected as the final average eyebrow color.

[0156] Figure 14Exemplary pupil color extraction methods according to some implementations of this disclosure are illustrated. Similar to eyebrow color extraction, when extracting pupil color, the side closer to the primary eye in the lens is first selected. In some implementations, if both eyes are primary eyes, pixels on both sides are collected together. In addition to the pupil itself, the closed region contained within the keypoints of the eye may also include eyelashes, sclera (white of the eye), and reflections. These should be removed as much as possible during pixel collection to ensure that the majority of the final pixels originate from the pupil itself.

[0157] To remove eyelash pixels, the key points of the eye are along the y-axis ( Figure 14 (Vertical direction) contracting inward by a certain distance to form Figure 14 The area shown is 1402. This is to remove the whites of the eyes and reflections (such as those caused by...). Figure 14 (As shown in circle 1404), pixels in region 1402 are further excluded. For example, if the R, G, and B values ​​of a pixel are all greater than a predetermined threshold, the pixel is excluded. Pixels collected in this way ensure that most of them originate from the pupil itself. Similarly, the median color is used as the average pupil color.

[0158] In some implementations, for lip color extraction, only the pixels in the lower lip region are detected. The upper lip is typically thin and relatively sensitive to keypoint errors, and because its color is lighter, it does not represent lip color well. Therefore, after rotating and correcting the image, all pixels in the region surrounded by the keypoints of the lower lip are collected, and the median color is used to represent the average lip color.

[0159] Figure 15 Exemplary hair color extraction regions used in hair color extraction methods according to some implementations of this disclosure are shown. Hair color extraction is more difficult than previous part extraction. The main reason is that everyone's hairstyle is unique, and the background of the photo is complex and diverse. Therefore, it is difficult to locate the pixels of the hair. In one way to accurately locate the hair pixels, a neural network is used to segment the hair pixels of the image. Since the annotation cost of image segmentation is very high, and game applications do not require very high accuracy of color extraction, a keypoint-based approximate prediction method is used.

[0160] To obtain hair pixels, the detection area is first determined. For example... Figure 15 As shown, the detection area 1502 is rectangular. The lower boundary is the corner points of the eyebrows on both sides, and the height (vertical line 1504) is the distance 1506 from the upper edge of the eyebrow to the lower edge of the eye. The left and right boundaries are keypoints 1 and 17, extending fixed distances to the left and right respectively. Figure 15 The image shows the hair pixel detection region 1502 obtained as a result.

[0161] Figure 16 The diagram illustrates exemplary separation of hair pixels and skin pixels within a hair color extraction region according to some implementations of this disclosure. Typically, the detection region contains three types of pixels: skin, hair, and background. In some more complex cases, headwear is also included. Because the left and right extents of the detection region are relatively conservative, it is assumed in most cases that the number of hair pixels included far exceeds the number of background pixels. Therefore, the main process is to divide the pixels of the detection region into hair or skin.

[0162] For each row of pixels in the detection region, skin color changes are typically continuous, e.g., from light to dark, and there is usually a noticeable change where skin color meets hair. Therefore, the middle pixel of each row is chosen as the starting point 1608, and skin pixels are detected to the left and right. First, a relatively conservative threshold is used to find more reliable skin color pixels, and then this is expanded to the left and right. If the colors of adjacent pixels are relatively similar, they are also marked as skin color. This method takes into account the grayscale of skin color and can obtain relatively accurate results. Figure 16 As shown, within the hair color extraction region 1602, darker areas such as 1604 represent skin color pixels, while brighter areas such as 1606 represent hair color pixels. The median R, G, and B values ​​of the collected hair color pixels within the hair color region are selected as the final average hair color.

[0163] Figure 17 An exemplary method for extracting eyeshadow color according to some implementations of this disclosure is shown. The extraction of eyeshadow color differs slightly from previous methods. This is because eyeshadow is a cosmetic substance that may or may not be present. Therefore, when extracting eyeshadow color, it is necessary to first determine whether eyeshadow is present, and if so, extract its average color. Similar to the extraction of eyebrow and pupil color, eyeshadow color extraction is performed only on the area closest to the main eye in the frame.

[0164] First, it's necessary to determine which pixels belong to the eyeshadow. For example... Figure 17 As shown, for the detection area of ​​eyeshadow pixels, region 1702 within lines 1704 and 1706 is used. The left and right sides of region 1702 are defined as the inner and outer corners of the eye, and the upper and lower sides of the region are the lower edge of the eyebrow and the upper edge of the eye. In addition to the possible eyeshadow pixels in region 1702, there may also be eyelashes, eyebrows, and skin that need to be excluded when extracting eyeshadow.

[0165] In some implementations, to eliminate the influence of eyebrows, the upper edge of the detection area is moved further downwards. To reduce the influence of eyelashes, pixels with brightness below a certain threshold are excluded. To distinguish between eyeshadow and skin color, the difference between the hue of each pixel and the average skin hue is checked. Only pixels with a difference greater than a certain threshold are collected as potential eyeshadow pixels. Hue is used instead of RGB values ​​because the average skin color is primarily collected below the eyes, while the skin color above the eyes can vary significantly in brightness. Since color is not sensitive to brightness, it is relatively stable. Therefore, hue is more suitable for determining whether a pixel is skin.

[0166] The above process determines whether a pixel in each detection area belongs to eyeshadow. In some implementations, if eyeshadow is not present, some pixels may still be incorrectly identified as eyeshadow.

[0167] To reduce the above errors, each column of the detection region is checked. If the number of eyeshadow pixels in the current column is greater than a certain threshold, the current column is marked as an eyeshadow column. If the ratio of the eyeshadow column to the width of the detection region is greater than a certain threshold, eyeshadow is considered to exist in the current image, and the median color of the collected eyeshadow pixels is used as the final color. In this way, a few pixels misclassified as eyeshadow will not lead to incorrect judgments of eyeshadow as a whole.

[0168] Considering art style, most games typically don't allow free color adjustments for all of the above-mentioned parts. For parts where color adjustments are allowed, usually only a predefined set of colors can be matched. Taking hair as an example, if a hairstyle allows five hair colors, the hairstyle in the resource pack will contain texture images corresponding to each hair color. During detection, the desired hair rendering effect can be obtained by simply selecting the texture image with the closest color based on the hair color prediction result.

[0169] In some implementations, when only a color texture image is provided, the color of the texture image can be reasonably changed based on any detected color. To facilitate color conversion, the commonly used RGB color space representation is converted to the HSV color model. The HSV color model includes three dimensions: hue (H), saturation (S), and lightness (V). Hue (H) is represented in the model as a 360-degree color range, where red is 0 degrees, green is 120 degrees, and blue is 240 degrees. Saturation (S) represents the mixture of spectral colors and white. The higher the saturation, the brighter the color. When the saturation is close to 0, the color is close to white. Lightness (V) represents the brightness of the color, and its value ranges from black to white. After color adjustment, it is expected that the HSV values ​​of the texture image match the predicted color. Therefore, the hue value calculation for each pixel can be represented as follows: H i =(H i+H′-H)%1, where H i ′ and H i H represents the hue of pixel i before and after adjustment, and H and H1 represent the median of the hue of the texture image before and after adjustment.

[0170] Unlike hue, which exists as a continuous, end-to-end connected space, saturation and brightness have boundary singularities similar to 0 and 1. If a linear processing method similar to hue adjustment is used, many pixel values ​​will exhibit excessively high or low saturation or brightness when the median of the initial or adjusted image is close to 0 or 1. This phenomenon results in unnatural colors. To address this issue, the following non-linear curves are used to fit saturation and brightness before and after pixel adjustment:

[0171] y = 1 / (1+(1-α)(1- x ) / (α x )), α∈(0,1)

[0172] In the above equation, x and y represent the saturation or brightness values ​​before and after adjustment, respectively. The only uncertain parameter is α, which can be derived as...

[0173] α = 1 / (1+x / (1-x)×(1-y) / y)

[0174] This equation guarantees that α falls within the range of 0 to 1. Taking saturation as an example, the initial median saturation can be easily calculated based on the input image. S Furthermore, the target saturation value S can be obtained through hair color extraction and color space conversion. t Therefore, α = 1 / (1 + S / (1- S )×(1-S t ) / S t For each pixel S in the default texture image. i Then it can be done through the equation: S i =1 / (1+(1-α)(1-S)) i ) / (αS i The adjusted value is calculated using this method. The same calculation applies to brightness.

[0175] To make the adjusted texture image display more closely resemble a realistic image, special processing is applied to different parts. For example, to maintain low saturation in the hair, [the following settings are used]. S′ = S′ × V′ ^0.3. Figure 18The illustration shows some exemplary color adjustment results according to some implementations of this disclosure. Column 1802 shows some default texture images provided by a specific game, column 1804 shows some texture images adjusted from the corresponding default texture images in the same row based on the real image shown at the top of column 1804, and column 1806 shows some texture images adjusted from the corresponding default texture images in the same row based on the real image shown at the top of column 1806.

[0176] Figure 19 This is a flowchart 1900 illustrating an exemplary process for extracting colors from a 2D facial image of a real person, according to some implementations of the present disclosure.

[0177] The process of extracting color from a 2D facial image of a real person includes step 1910: identifying multiple key points in the 2D facial image based on a key point prediction model.

[0178] The process also includes step 1920: rotating the 2D facial image until the key points selected from multiple key points are aligned.

[0179] The process also includes step 1930: locating multiple parts in the rotated 2D facial image, with each part defined by a corresponding subset of multiple key points.

[0180] The process also includes step 1940: extracting the average color of each of a plurality of parts defined by corresponding subsets of key points from the pixel values ​​of the 2D facial image.

[0181] The process also includes step 1950: using extracted colors from multiple parts of the 2D facial image to generate a personalized 3D model of a real-life person that matches the corresponding facial feature colors of the 2D facial image.

[0182] Other implementations may include one or more of the following features.

[0183] In some implementations, the keypoint prediction model in step 1910 is formed by machine learning based on keypoints manually annotated by the user.

[0184] In some implementations, the selected key points in the rotation step 1920 for alignment are located on the symmetrical left and right sides of the 2D facial image.

[0185] In some implementations, in step 1940, extracting the average color of each of the multiple regions may include selecting the median of the R, G, B values ​​of all pixels in a corresponding defined region within the corresponding region as the predicted average color.

[0186] In some embodiments, step 1940, extracting the average color of each of the multiple locations, may include determining a region within the skin location for skin color extraction, and selecting the median R, G, B values ​​of all pixels within the region for skin color extraction as the predicted average color of the skin location. In some embodiments, the region within the skin location for skin color extraction is determined to be the area below the eyes and above the lower edge of the nose on the face.

[0187] In some implementations, step 1940, extracting the average color of each of the multiple regions, may include eyebrow color extraction within the eyebrow region. Eyebrow color extraction includes: selecting an eyebrow as a target eyebrow based on the determination that the eyebrow is located closer to the viewer of the 2D facial image; selecting two eyebrows as target eyebrows based on the determination that both eyebrows are equally close to the viewer of the 2D facial image; extracting a middle eyebrow region within the target eyebrows; comparing each pixel value within the middle eyebrow region with the average skin color; collecting pixels within the middle eyebrow region whose pixel values ​​differ from the average skin color by a threshold; and selecting the median of the R, G, and B values ​​of the pixels collected for eyebrow color extraction as the predicted average color of the eyebrow region.

[0188] In some implementations, step 1940, extracting the average color of each of the multiple regions may include pupil color extraction within the eye region. Pupil color extraction includes: selecting an eye as a target eye based on determining that the eye is located closer to the viewer of the 2D facial image; selecting two eyes as target eyes based on determining that both eyes are equally close to the viewer of the 2D facial image; extracting a region within the target eye without eyelashes; comparing each pixel value within the extracted region to a predetermined threshold; collecting pixels within the extracted region that have pixel values ​​exceeding the predetermined threshold; and selecting the median of the R, G, and B values ​​of the pixels collected for pupil color extraction as the predicted average color of the pupil.

[0189] In some implementations, in step 1940, extracting the average color of each of the multiple regions may include lip color extraction within the lip region, which includes: collecting all pixels in the region surrounded by keypoints of the lower lip, and selecting the median of the R, G, B values ​​of the pixels collected for the lip color extraction as the predicted average color of the lip region.

[0190] In some implementations, step 1940, extracting the average color of each of the multiple regions, may include hair color extraction within a hair region. The hair color extraction includes: identifying a region including a forehead region extending into the hair regions on both sides; determining pixel color variations exceeding a predetermined threshold from the middle of the region to its left and right boundaries; dividing the region into a hair region and a skin region based on the pixel color variations exceeding the predetermined threshold; and selecting the median of the R, G, B values ​​of the pixels in the hair region within the region as the predicted average color of the hair region.

[0191] In some implementations, the area including the forehead extending into the hairline on both sides is identified as a rectangular region, wherein the lower boundary is at the two eyebrow corners, the left and right boundaries are at fixed distances outward from key points located on the symmetrical left and right sides of the 2D facial image, and the height is at the distance from the upper edge of the eyebrow to the lower edge of the eye.

[0192] In some implementations, step 1940, extracting the average color of each of the multiple regions may include eyeshadow color extraction within an eyeshadow region. Eyeshadow color extraction includes: selecting an eye as a target eye based on determining that the eye is located closer to the viewer of the 2D facial image; selecting two eyes as target eyes based on determining that both eyes are equally close to the viewer of the 2D facial image; extracting a central region within the eyeshadow region near the target eye; collecting pixels within the extracted central region whose brightness is higher than a predetermined brightness threshold to exclude eyelashes and whose difference from the average skin pixel hue value exceeds a predetermined threshold; labeling a pixel column as an eyeshadow column based on determining that the number of pixels collected in a pixel column within the extracted central region is greater than a threshold; and selecting the median of the R, G, and B values ​​of the pixels collected for eyeshadow color extraction as the predicted eyeshadow color for the eyeshadow region based on determining that the ratio of the eyeshadow column to the width of the extracted central region is greater than a certain threshold.

[0193] In some implementations, the process of extracting colors from a 2D facial image of a real person may additionally include converting a texture map based on an average color while preserving the original brightness and color difference of the texture map. The texture map conversion includes converting the average color from an RGB color space representation to an HSV (hue, saturation, brightness) color space representation and adjusting the colors of the texture map to reduce the difference between the median HSV value of the average color and the median HSV value of the texture map pixels.

[0194] The methods and systems disclosed in this paper can be used in various applications, such as character modeling and game character generation. The lightweight approach allows for flexible application to different devices, including mobile devices.

[0195] In some implementations, the definition of facial key points in the current system and method is not limited to the current definition, and other definitions are also possible, as long as the contour of each part can be fully expressed. Additionally, in some implementations, instead of directly using the colors returned from the scheme, they can be matched against a predefined color list to achieve further color filtering and control.

[0196] Deformation methods that optimize the Laplacian operator require the mesh to be a differentiable manifold. However, in practice, meshes created by game artists often contain artifacts such as repeating vertices and unclosed edges, which can corrupt the properties of the manifold. Therefore, methods such as biharmonic deformation can only be used after careful mesh cleaning. The affine deformation method proposed in this paper does not use the Laplacian operator and therefore does not have such a strong constraint.

[0197] A series of deformation methods based on biharmonic deformation have limitations in some cases. Solving the harmonic function of the Laplace operator once often fails to achieve smooth results due to its low smoothness requirement. Solving the multiharmonic function of higher-order (>=3) Laplace operators fails on many meshes due to its high requirement of at least 6th-order differentiability. In most cases, biharmonic deformation, which involves solving the Laplace operator only twice, is observed to achieve acceptable results. Even so, the deformation remains unsatisfactory due to the lack of tuning degrees of freedom. The affine deformation proposed in this paper allows for fine deformation tuning by changing the smoothness parameter, and its deformation results cover the range of applications using biharmonic deformation.

[0198] Figure 20 This is a flowchart illustrating an exemplary head avatar deformation and generation process according to some implementations of this disclosure. Using the techniques proposed in this disclosure, the head mesh can be appropriately deformed without being bound to a skeleton. Therefore, the workload required by artists is greatly reduced. These techniques adapt to different styles of meshes for better versatility. When creating game assets, artists can save head models in various formats using tools such as 3ds Max or Maya, but the internal representation of these formats is a polygonal mesh. Polygonal meshes can be easily converted into pure triangular meshes called stencil models. For each stencil model, 3D keypoints are manually marked once on the stencil model. After this, the stencil model can be used to deform into a characteristic head avatar based on 3D keypoints detected and reconstructed from any human facial image.

[0199] Figure 21 This is a diagram illustrating exemplary header template models comprising some implementations of the present disclosure. For example... Figure 21As shown, the head template model 2102 typically includes parts such as the face 2110, eyes 2104, eyelashes 2106, teeth 2108, and hair. Without a bound skeleton, mesh deformation depends on the connection structure of the template mesh. Therefore, the template model needs to be decomposed into these semantic parts, and the facial mesh needs to be deformed first. By setting and following certain key points on the facial mesh, all other parts can be automatically adjusted. In some implementations, an interactive tool is provided to detect all topologically connected parts, and the user can use this interactive tool to conveniently export these semantic parts for further deformation.

[0200] In some implementations, key points of human facial images can be obtained through detection algorithms or AI models. These key points need to be mapped to vertices on a template model for the purpose of driving mesh deformation. Due to the randomness of mesh connections and the lack of 3D human key point annotation data, there is no tool that can accurately and automatically annotate 3D key points on any head model. Therefore, an interactive tool has been developed that allows for the rapid manual annotation of key points on 3D models. Figure 22 This is a diagram illustrating some exemplary keypoint markings on realistic-style 3D models such as 2202 and 2204 and cartoon-style 3D models such as 2206 and 2208, according to some implementations of this disclosure.

[0201] During the labeling process, the positions of the labeled 3D keypoints on the 3D model should match the keypoints in the image as closely as possible. Since keypoints are labeled on discrete vertices on the 3D model mesh, some bias is unavoidable. To counteract such bias, one approach is to define appropriate rules in the pose processing. Figure 23 This diagram illustrates an exemplary comparison between template model rendering, manually labeled keypoints, and AI-detected keypoints in some implementations of this disclosure. In some embodiments, for models that are rendered relatively realistically, keypoint detection and reconstruction algorithms can be applied to the rendering of the template model (2302), and the results of 3D keypoints obtained, for example, through artificial intelligence (2306), can be further compared with manually labeled keypoints (2304), and thus the deviation between the two sets of keypoints is calculated. In the case of detecting human images, keypoints detected from real-life images reduce the calculated deviation, and the undesirable effects of manual labeling are eliminated.

[0202] The affine deformation method disclosed in this paper is a keypoint-driven mathematical modeling approach that ultimately solves a system of linear equations. The method disclosed herein employs a step to deform the template mesh using detected keypoints as boundary conditions, and incorporates different constraints during the optimization process. Figure 24This is a diagram illustrating an exemplary affine transformation of a triangle according to some implementations of this disclosure.

[0203] In some implementations, the deformation from the template mesh to the predicted mesh is considered as a set of affine transformations for each triangle. The affine transformation of a triangle can be defined as a 3×3 matrix T and a translation vector d. Figure 24 As shown, the position of the deformed vertex after the affine transformation is denoted as v. i ′=Tv i +d, i∈1…4, where v1, v2, v3 represent each vertex of the triangle, and v4 is an additional point introduced along the triangle's normal direction, satisfying the equation v4=v1+(v2-v1)×(v3-v1) / sqrt( / (v2-v1)×(v3-v1) / ). In this equation, the cross product is normalized to be proportional to the length of the triangle's sides. v4 is introduced because the coordinates of the three vertices are insufficient to determine a unique affine transformation. After introducing v4, we obtain the derived equation: T=[v′2-v′1 v′3-v′1 v′4-v′1 / ×[v2-v1 v3-v1 v4-v1] -1 Furthermore, the non-translated portion of matrix T was determined. Since matrix V = [v2-v1 v3-v1 v4-v1] -1 It depends only on the template mesh and is independent of other deformation factors, so it can be pre-computed as a sparse coefficient matrix for later construction of the linear system.

[0204] So far, the non-translational part of the affine transformation T in the mathematical formula has been represented. To construct an optimized linear system, assuming the number of mesh vertices is N and the number of triangles is F, consider the following four constraints:

[0205] Constraints on key point locations: E k =∑ i＝1 ||v′ i -c′ i || 2 , c′ i This represents the location of key points detected after mesh deformation.

[0206] Constraints on adjacency smoothness: E s =∑ i＝1 ∑ j∈adj(i) ||T i -T j || 2 This constraint means that the affine transformations between adjacent triangles should be as similar as possible. Adjacency relationships can be queried and stored in advance to avoid redundant calculations and improve the performance of the constructed system.

[0207] Constraints on characteristics: E i =∑i-1 ||T i -I|| 2 , where I represents the identity matrix. This constraint means that the affine transformation should be as invariant as possible, which helps to preserve the properties of the template mesh.

[0208] Constraints of the original position: E l =∑ i＝1 ||v′ i -c i || 2 , where c i This indicates the position of each vertex on the template mesh before deformation.

[0209] The final constraint is a weighted sum of the above constraints: minE = w k E k +w s E s +w i E i +w l W l Where the weight w k w s w i The values ​​of w1 are arranged from strongest to weakest. Using these constraints, a linear system of size (F+N)×(F+N) can be constructed, with the weights multiplied by their corresponding coefficients in the system. The unknowns, except for the extra point v′4 for each triangle, are the coordinates of each vertex after the deformation. Since the preceding terms are useful, the results for v′4 are discarded. During successive deformations, all constraint matrices except those for keypoint locations can be reused. For meshes with thousands of vertices, the affine transformation can achieve real-time performance of 30fps on ordinary personal computers and smartphones.

[0210] Figure 25 This is a diagram illustrating an exemplary comparison of the deformation results of some head models with and without a blending shape process according to some implementations of this disclosure.

[0211] In some implementations, when deforming the head model of a game avatar, the region of interest is typically only the face. The top, back, and neck of the head should remain unchanged, otherwise mesh penetration between the head and hair or torso will occur. To avoid this problem, the result of the affine deformation and the stencil mesh are linearly interpolated in a blended shape manner. The blending weights can be drawn in 3D modeling software or calculated using affine deformation or biharmonic deformation with minor changes. For example, the weights on keypoints are set to 1s, while more markers are added to the head model ( Figure 25(Dark spots in 2504), and set the weight of the marker to 0s. In some implementations, inequality constraints are added to the solution process to force all weights to fall within the range of 0 to 1, but doing so greatly increases the complexity of the solution. Experiments have shown that good results can be obtained by removing weights less than 0 or greater than 1. Figure 25 As shown in 2504, the model portion with the darkest color has a weight of 1s, and the colorless model portion has a weight of 0s. In the blended weight rendering 2504, there is a natural transition between the bright keypoints and the dark keypoints. In the case of blended shapes, the back side of the deformed model (such as...) Figure 25 (as shown in 2506) and the original (as shown in 2506) Figure 25 (As shown in 2502) remains the same. Without a blended shape, the back side of the deformed model (as shown in 2502) remains the same. Figure 25 (as shown in 2508) and the original (as shown in 2508) Figure 25 (As shown in 2502) does not remain the same.

[0212] In some implementations, affine deformation can achieve different deformation effects by manipulating the weights of the constraints, including the results of simulating double harmonic deformation. Figure 26 This is a diagram illustrating an exemplary comparison of affine deformation and biharmonic deformation with different weights according to some implementations of this disclosure. (See diagram for example.) Figure 26 As shown, smoothness is the adjacent smoothness weight w s and feature weight w i The ratio. Dark points are keypoints, and the darkness of the color represents the displacement of the vertex's deformed position from its initial position. In all deformation results, one keypoint remains unchanged, while another keypoint moves to the same position. This indicates that as the adjacent smoothness weight gradually increases relative to the characteristic weight, the smoothness of the deformed sphere also increases accordingly. Furthermore, the results of biharmonic deformation can be matched with those of affine deformation where the smoothness falls somewhere between 10 and 100. This indicates that affine deformation has more degrees of freedom in deformation compared to biharmonic deformation.

[0213] Using the workflow described in this article, games can easily integrate the functionality of intelligently generating head avatars. For example, Figure 27 This disclosure illustrates some implementations of a realistic template model based on randomly selected images of women. Figure 27 (Not shown in the image) Some examples of automatically generated results. All personalized head avatars reflect some characteristics of their corresponding images.

[0214] Figure 28This is a flowchart 2800 illustrating an exemplary process for generating a 3D deformable head model from a 2D facial image of a real person, according to some implementations of the present disclosure.

[0215] The process of generating a 3D deformable head model from a 2D facial image includes step 2810: receiving a two-dimensional (2D) facial image.

[0216] The process also includes step 2820: identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model (e.g., a convolutional neural network).

[0217] The process also includes step 2830: mapping a first set of keypoints to a second set of keypoints on multiple vertices of the mesh of the 3D head template model based on a set of user-provided keypoint annotations located on the 3D head template model.

[0218] The process also includes step 2840: deforming the mesh of the 3D head template model by reducing the difference between the first set of keypoints and the second set of keypoints to obtain a deformed 3D head mesh model. In some implementations, there is a correspondence between the keypoints in the first set and the keypoints in the second set. After projecting the second set of keypoints into the same space as the first set of keypoints, a function is generated to measure the positional difference between each keypoint in the first set and the second set of keypoints. By deforming the mesh of the 3D head template model, the second set of keypoints in space is optimized when the function measuring the positional difference (e.g., position, adjacency smoothness, properties, etc.) between each keypoint in the first set and the second set of keypoints is minimized.

[0219] The process also includes step 2850: applying the blendshape method to the deformed 3D head mesh model to obtain a personalized head model based on the 2D facial image.

[0220] Other implementations may include one or more of the following features.

[0221] In some implementations, the mapping step 2830 may further include: associating a first set of key points on a 2D facial image with a plurality of vertices on a mesh of a 3D head template model; identifying a second set of key points based on a set of user-provided key point annotations on a plurality of vertices on a mesh of a 3D head template model; and mapping the first set of key points and the second set of key points based on features identified by corresponding key points on the face.

[0222] In some implementations, step 2840 of performing the deformation may include: deforming the mesh of the 3D head template model into a deformed 3D head mesh model by using a mapping from a first set of key points to a second set of key points and by using boundary conditions of the deformation associated with the first set of key points.

[0223] In some implementations, step 2840 of performing the deformation may further include applying different constraints during the deformation optimization process, including one or more of the following: the location of key points, adjacency smoothness, characteristics, and initial location.

[0224] In some implementations, step 2840 of performing the deformation may further include applying a constraint to the deformation process, the constraint being a weighted sum of one or more of the key point's location, adjacency smoothness, characteristics, and original location.

[0225] In some implementations, step 2820 of identifying the first set of key points includes using a convolutional neural network (CNN).

[0226] In some implementations, the deformation includes affine deformation without a Laplacian operator. In some implementations, affine deformation is tuned by changing the smoothness parameter.

[0227] In some implementations, the mesh of the 3D head template model can deform without being bound to the skeleton. In some implementations, the facial deformation model includes a realistic style model or a cartoon style model.

[0228] In some implementations, applying the blending shape method to the deformed 3D head mesh model in step 2850 includes: assigning corresponding blending weights to key points based on the positions of key points in the deformed 3D head mesh model; and applying different levels of deformation to key points with different blending weights.

[0229] In some implementations, applying the hybrid shape method to the deformed 3D head mesh model in step 2850 includes maintaining the back side of the deformed 3D head mesh model in the same shape as the original back side of the 3D head template model before deformation.

[0230] In some implementations, the semantic elements on the template model are not limited to eyes, eyelashes, or teeth. By adding new keypoints to the facial mesh and tracking them, decorations such as glasses can potentially be adaptively adjusted.

[0231] In some implementations, keypoints on the template model are added manually. In other implementations, deep learning techniques can be used to automatically add keypoints to different template models.

[0232] In some implementations, the solution process for affine deformation can utilize numerical techniques to further improve its computational performance.

[0233] In some implementations, the systems and methods disclosed herein form a facial avatar generation system based on lightweight keypoints, which has many advantages, such as those listed below:

[0234] The system and method have low requirements for the input image. The face does not need to be directly facing the camera device, and a certain degree of in-plane rotation, out-of-plane rotation, and occlusion will not significantly affect performance.

[0235] It works for both realistic and cartoon games. This system does not restrict the game style to realistic, and the game style can also be applied to cartoon styles.

[0236] Lightweight and customizable. Each module of this system is relatively lightweight and suitable for mobile devices. The modules in the system are decoupled, and users can use different combinations to build the final face generation system according to different game styles.

[0237] In some implementations, for a given single photograph, the main face is detected first, followed by keypoint detection. In real-world images, faces may not be facing the camera, and real faces are not always perfectly symmetrical. Therefore, keypoints in the original image are preprocessed to obtain a uniform, symmetrical, and smooth set of keypoints. These keypoints are then adjusted according to the game's specific style (e.g., enlarged eyes, thin face). After obtaining the stylized keypoints, they are converted into control parameters for the facial model in the game, typically skeletal or slider parameters.

[0238] In some implementations, the viewpoint of the real face may not be directly facing the camera device, and there may be problems such as left-right asymmetry and key point detection errors. Figure 29 This is a diagram illustrating exemplary keypoint processing steps according to some implementations of this disclosure. Keypoints 2904 detected from the original image cannot be used directly and require some processing. Here, as... Figure 29 As shown, the process consists of three steps: normalization, symmetry, and smoothing.

[0239] In some implementations, it is necessary to adjust the standard facial model in the game based on the prediction of real facial key points. This process needs to ensure that the key points of the standard facial model in the game are aligned with the real face in terms of scale, position, and orientation. Therefore, the normalization 2906 of the predicted key points and the key points on the game facial model includes the following components: scale normalization, translation normalization, and angle normalization.

[0240] In some implementations, all the originally detected 3D facial key points are defined as p, where the i-th key point is p. i ={x i y i , z i For example, the normalization origin is defined as the midpoint between key points 1 and 17 (see reference). Figure 1 (Definition of key points), that is, c = (p1 + p) 17 ) / 2. For the scale, the distance between the 1st and 17th keypoints and the origin is adjusted to 1, so that the 3D keypoints normalized by scale and translation are p′=(pc) / ||p1-c||.

[0241] In some implementations, after normalizing the scale and translation, the facial orientation is further normalized. For example... Figure 29 As shown in image 2902, the face in the actual photograph may not be directly facing the lens and will always have some degree of deflection, which may exist on the three coordinate axes. The predicted 3D keypoints of the face are rotated sequentially along the x, y, and z coordinate axes so that the face faces the camera. When rotating along the x-axis, the z-coordinates of keypoints 18 and 24 (refer to...) are... Figure 1 (Definition of key points) Alignment, that is, making the depth of the uppermost part of the bridge of the nose the same as the depth of the bottom of the nose, in order to obtain the rotation matrix R. X When rotating along the y-axis, the z-coordinates of keypoints 1 and 17 are aligned to obtain the rotation matrix R. Y When rotating along the z-axis, the y-coordinates of keypoints 1 and 17 are aligned to obtain the rotation matrix R. z Therefore, the direction of the keypoints is aligned, and the normalized keypoints are as follows:

[0242] P norm =R Z ×R Y ×R X ×P′

[0243] In some implementations, the scale, position, and angle of the normalized keypoints have been adjusted to be uniform; however, the resulting keypoints are often not perfect facial features. For example, the bridge of the nose is not a straight line relative to the center, and facial features may be asymmetrical. This is because real faces in photographs are not perfectly symmetrical due to expressions or their inherent characteristics, and this introduces additional errors when predicting keypoints. While real faces may be asymmetrical, if the facial model in a game is asymmetrical, it will result in an unattractive appearance and will significantly degrade the user experience. Therefore, making keypoints symmetrical, as shown in 2908, is a necessary process.

[0244] Since the keypoints have been normalized, in some implementations, a simple symmetry approach is to average the y and z coordinates of all left-symmetric and right-symmetric keypoints to replace the original y and z coordinates. This method works well in most cases, but performance is sacrificed when the face is rotated at a large angle along the y-axis.

[0245] In some implementations, with Figure 29 Taking a human face as an example, when the face is turned a large angle to the left, part of the eyebrows will be invisible. Simultaneously, due to perspective, the left eye will appear smaller than the right eye. Although 3D keypoints can partially compensate for the effects of perspective, the 2D projections of the corresponding 3D keypoints still need to be preserved in the image. Therefore, excessive angular deflection will lead to significant differences in the size of the eyes and eyebrows in the 3D keypoint detection results. To address the effects of angle, when the face is turned a large angle along the y-axis, the eye and eyebrow closest to the camera are used as the primary eye and primary eyebrow, and they are copied to the other side to reduce errors caused by angular deflection.

[0246] In some implementations, because prediction errors of keypoints are unavoidable, symmetrical keypoints may still not match the real face in some individual cases. Since the shapes of the real face and facial features are very different, it is difficult to achieve a relatively accurate description using predefined parametric curves. Therefore, when smoothing as shown in 2910, only some areas, such as the contours of the face, eyes, eyebrows, lower lip, etc., are smoothed. These areas remain essentially monotonically smooth, that is, without jagged edges. In this case, the target curve should always be either a convex or concave curve.

[0247] In some implementations, key points on the relevant boundaries are checked one by one to see if they satisfy the definition of a convex curve (or concave curve). Figure 30 This is a diagram illustrating an exemplary keypoint smoothing process 2910 according to some implementations of this disclosure. (See diagram for example.) Figure 30 As shown, without loss of generality, the target curve should be convex. For each keypoint 3002, 3004, 3006, 3008, and 3010, check if its position is above the line connecting its adjacent left and right keypoints. If this condition is met, it means the current keypoint satisfies the convex curve requirement. Otherwise, the current keypoint will move upwards to the line connecting the left and right keypoints. For example, in Figure 30 In this case, keypoint 3006 does not meet the constraints of a convex curve, and keypoint 3006 will be moved to position 3012. If multiple keypoints are moved, it may not be possible to guarantee whether the curve will be convex or concave after the movement. Therefore, in some implementations, multi-round smoothing is used to obtain a relatively smooth keypoint curve.

[0248] Different games have different facial styles. In some implementations, it's necessary to transform the key features of a realistic face into the style required by the game. Realistic game faces are similar, but cartoon faces are very different. Therefore, it's difficult to have a unified standard for stylizing key features. In practice, the definition of stylization comes from the game designers, who adjust facial features according to the specific game style.

[0249] In some implementations, more general facial adjustment schemes that most games might require are implemented. These include adjustments to face length, width, and facial features. Custom corrections can be made based on different game art styles, adjustment levels, scaling ratios, etc. Users can also customize adjustment methods for any specific style, such as changing the eye shape to a rectangle. The system supports any type of adjustment.

[0250] In some implementations, key points of a stylized face are used to deform a standard game face so that the key points of the deformed face reach the position of a target key point. Since most games use control parameters such as bones or sliders to adjust the face, a set of control parameters is needed to move the key points to the target position.

[0251] Since the definitions of bones or sliders may differ across games and are subject to change, directly defining a simple parameterized function from keypoints to bone parameters is impractical. In some implementations, machine learning methods are used to convert keypoints into parameters via a neural network, known as a K2P (keypoint-to-parameter) network. Because the number of parameters and keypoints is generally small (typically less than 100), K-layer fully connected networks are used in some implementations.

[0252] Figure 31This is a block diagram illustrating an exemplary keypoint-to-control-parameter (K2P) conversion process according to some implementations of this disclosure. To utilize machine learning methods, in some implementations, bone or slider parameters are first randomly sampled and fed to a game client 3110, and keypoints are extracted from the generated game face. In this way, a large amount of training data (parameter 3112 and keypoint 3114 pairs) can be obtained. A self-supervised machine learning method is then implemented, which consists of two steps: the first step is to train a P2K (parameter-to-keypoint) network 3116 to simulate the process of generating game parameter-to-keypoints. In the second step, according to the method described herein, a large number of unlabeled real face images 3102 are used to generate real facial keypoints 3104, and then a large number of stylized keypoints 3106 are generated. These unlabeled stylized keypoints 3106 are self-supervised learning training data. In some implementations, a set of keypoints K is input into a K2P network 3108 for learning to obtain output parameters P. Since the true values ​​of the ideal parameters corresponding to these keypoints are unavailable, P is further fed into the P2K network 3116 trained in the first step to obtain keypoints K′. In some implementations, the K2P network 3108 can be learned by calculating the mean square error (MSE) loss between K and K′. In some implementations, the P2K network 3116 is fixed during the second step and will not be further adjusted. With the aid of the P2K network 3116, the control process of translating the parameters of the game client 3110 to keypoints is simulated using a neural network, thus laying the foundation for the learning of the K2P network 3108 in the second step. In this way, the final face generated by the parameters maintains a close resemblance to the keypoints of the generated target stylized face.

[0253] In some implementations, weights are also added to certain keypoints (e.g., keypoints in the eye) by adjusting the corresponding weights when calculating the MSE loss between K and K′. Since the keypoints are predefined and will not be affected by the game client's skeleton or slider, the weights are easier to adjust.

[0254] In some implementations, to improve model accuracy in practical applications, decoupled parts can be trained separately for neural networks. For example, if some skeletal parameters only affect keypoints in the eye region, while other parameters have no effect on that region, then these parameters and keypoints form a separate set of regions. A separate K2P model 3108 is trained for each such set of regions, and each model can employ a lighter network design. This not only further improves model accuracy but also reduces computational complexity.

[0255] Figure 32The following diagram illustrates some exemplary results of automatic face generation in mobile games based on some implementations of this disclosure. For example... Figure 32 The diagram illustrates the results generated from the original facial images (3202 and 3206) to the game's facial avatar images (3204 and 3208). In some implementations, during stylization, an open mouth is made closed, and different levels of restriction and cartoonishness are applied to the nose, mouth, facial shape, eyes, and eyebrows. The final generated result still retains certain human facial features and meets the aesthetic requirements of the game style.

[0256] Figure 33 This is a flowchart 3300 illustrating an exemplary process of customizing the standard face of an avatar in a game using a 2D facial image of a real-life person according to some implementations of this disclosure.

[0257] The process of using 2D facial images of real-life people to customize the standard face of the avatar in the game includes step 3310: identifying a set of real-life key points in the 2D facial image.

[0258] The process also includes step 3320: transforming the set of real-life key points into a set of game-style key points associated with the avatar in the game.

[0259] The process also includes step 3330: generating a set of control parameters for the standard face of the avatar in the game by applying this set of game-style keypoints to a Keypoint To Parameter (K2P) neural network model. (The above is combined...) Figure 31 The K2P network 3108 is a deep learning neural network model that predicts a set of facial control parameters based on a set of input avatar keypoints. Because different sets of avatar keypoints can correspond to different sets of facial control parameters, when this set of facial control parameters is applied to the standard face of the avatar, the adjusted keypoints of the standard face can have a set of keypoints similar to the input avatar keypoints. (As described above...) Figure 31 In contrast to the K2P network 3108, the P2K network 3116 is a deep learning neural network model that predicts a set of avatar keypoints based on the set of input facial control parameters. Since different sets of facial control parameters can lead to different sets of avatar keypoints, when the two neural network models are considered to perform opposite processes, the set of output avatar keypoints associated with the P2K network 3116 should match the set of input avatar keypoints associated with the K2P network 3108.

[0260] The process also includes step 3340: adjusting multiple facial features of the standard face by applying the set of facial control parameters to the standard face.

[0261] Other implementations may include one or more of the following features.

[0262] In some implementations, in step 3330, the K2P neural network model is trained by the following steps: obtaining multiple training two-dimensional facial images of real-life people; generating a set of training game style keypoints for each of the multiple training two-dimensional facial images; feeding each set of training game style keypoints into the K2P neural network model to obtain a set of control parameters; feeding the set of control parameters into a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a set of predicted game style keypoints corresponding to the set of training game style keypoints; and updating the K2P neural network model by reducing the difference between the set of training game style keypoints and the corresponding set of predicted game style keypoints.

[0263] In some implementations, the pre-trained P2K neural network model is configured to: receive a set of control parameters, including bone or slider parameters associated with an avatar in the game; and a set of game style keypoints for predicting the avatar in the game based on the set of control parameters.

[0264] In some implementations, the difference between the training game style keypoints and the predicted game style keypoints of the corresponding group is the sum of the mean squared errors between the training game style keypoints and the predicted game style keypoints of the corresponding group.

[0265] In some implementations, the trained K2P and pre-trained P2K neural network models are game-specific.

[0266] In some implementations, a set of real-life key points in a 2D facial image corresponds to the facial features of a real-life person in the 2D facial image.

[0267] In some implementations, the standard face of an avatar in the game can be customized into different game characters based on facial images of different real-life people.

[0268] In some implementations, the avatar's deformed face is a cartoon-style face of a real person. In other implementations, the avatar's deformed face is a realistic-style face of a real person.

[0269] In some implementations, transforming the set of real-life key points into the set of game-style key points in step 3320 includes: normalizing the set of real-life key points to a normalized space; making the normalized set of real-life key points symmetrical; and adjusting the symmetrical set of real-life key points according to a predefined style associated with the avatar in the game.

[0270] In some implementations, normalizing the set of real-life key points into a normalized space includes: scaling the set of real-life key points into the normalized space; and rotating the scaled set of real-life key points according to the orientation of the set of real-life key points in the 2D facial image.

[0271] In some implementations, transforming the set of real-life key points into the set of game-style key points also includes smoothing the set of symmetrical key points to meet predefined convex or concave curve requirements.

[0272] In some implementations, adjusting the set of realistic key points of symmetry according to a predefined style associated with the avatar in the game includes one or more of the following: face length adjustment, face width adjustment, facial feature adjustment, scaling adjustment, and eye shape adjustment.

[0273] The system and method disclosed in this paper can be applied to automated face generation systems for various games, including both realistic and cartoon-style games. The system features an easy-to-integrate interface, improving the user experience.

[0274] In some implementations, the systems and methods disclosed herein can be used in 3D facial avatar generation systems for various games, and the complex manual adjustment process is automated to improve the user experience. Users can take selfies or upload existing photos. The system can extract features from the face in the photo and then automatically generate control parameters (such as bones or sliders) for the game face via an AI facial generation system. The game client uses these parameters to generate a facial avatar, resulting in a face that reflects the user's facial features.

[0275] In some implementations, the system can be easily customized for different games, including keypoint definitions, stylization methods, and skeleton / slider definitions. Users can choose to adjust only certain parameters, automatically retrain the model, or add custom control algorithms. In this way, the invention can be easily deployed to different games.

[0276] Other implementations include various subsets of the above implementations that are combined or otherwise rearranged in various other implementations.

[0277] The image processing apparatus of embodiments of this application is described herein with reference to the accompanying drawings. The image processing apparatus can be implemented in various forms, for example, as different types of computer devices such as servers or terminals (e.g., desktop computers, laptop computers, or smartphones). The hardware structure of the image processing apparatus of embodiments of this application is further described below. It will be understood that... Figure 34 Only an exemplary structure of the image processing apparatus is shown, not all structures, and it can be implemented as needed. Figure 34 Some or all of the structures shown in the figure.

[0278] Reference Figure 34 , Figure 34 This is a schematic diagram of an optional hardware structure of an image processing apparatus according to an embodiment of this application. In practical applications, the optional hardware structure can be applied to servers or various terminals running applications. Figure 34 The image processing apparatus 3400 shown includes at least one processor 3401, a memory 3402, a user interface 3403, and at least one network interface 3404. The components in the image processing apparatus 3400 are coupled together via a bus system 3405. It will be understood that the bus 3405 is configured to enable connection and communication between the components. In addition to a data bus, the bus system 3405 may also include a power bus, a control bus, and a status signal bus. However, for the purpose of clarity, all buses are shown in... Figure 34 The system is marked as Bus System 3405.

[0279] User interface 3403 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, touch screen, etc.

[0280] It is understood that memory 3402 can be volatile memory or non-volatile memory, or it can include both volatile memory and non-volatile memory.

[0281] In embodiments of this application, memory 3402 is configured to store different types of data to support the operation of image processing apparatus 3400. Examples of data include: any computer program for performing operations on image processing apparatus 3400, such as executable program 34021 and operating system 34022, and a program for performing the image processing method of embodiments of this application may be included in executable program 34021.

[0282] The image processing method disclosed in the embodiments of this application can be applied to processor 3401, or can be executed by processor 3401. Processor 3401 can be an integrated circuit chip and has signal processing capabilities. In implementation, each step of the image processing method can be completed by using integrated logic circuits in the hardware of processor 3401 or instructions in software form. The aforementioned processor 3401 can be a general-purpose processor, a digital signal processor (DSP), another programmable logic device, discrete gates, transistor logic devices, discrete hardware components, etc. Processor 3401 can implement or execute the methods, steps, and logic block diagrams provided in the embodiments of this application. The general-purpose processor can be a microprocessor, any conventional processor, etc. The steps in the method provided in the embodiments of this application can be directly executed by a hardware decoding processor, or can be executed by combining hardware and software modules in the decoding processor. The software modules can be located in a storage medium. The storage medium is located in memory 3402. Processor 3401 reads information from memory 3402 and executes the steps of the image processing method provided in the embodiments of this application by combining the information with its hardware.

[0283] In some implementations, image processing and 3D face and head formation can be performed on a server cluster on a network or in the cloud.

[0284] In one or more examples, the described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored on or transmitted via a computer-readable medium as one or more instructions or code, and executed by a hardware-based processing unit. The computer-readable medium can include a computer-readable storage medium, which corresponds to a tangible medium (e.g., a data storage medium) or a communication medium, including any medium that facilitates, for example, the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium can generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the implementations described in this application. The computer program product can include a computer-readable medium.

[0285] The terminology used in the description of the implementations herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. As used in the description of the implementations and the appended claims, the singular forms (“a”, “an”) and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the relevant listed items. It should also be understood that, when used in this specification, the terms “comprises” and / or “comprising” specify the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0286] It should also be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the implementation, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.

[0287] The description in this application is presented for purposes of illustration and description and is not intended to be exhaustive or limited to the disclosed forms of the invention. Many modifications, variations, and alternative implementations will be apparent to those skilled in the art from the teachings presented in the foregoing description and the accompanying drawings. The embodiments were chosen and described to best explain the principles of the invention, its practical application, and to enable others skilled in the art to understand the various implementations of the invention, and preferably to utilize the basic principles and various implementations with modifications suitable for the intended particular use. Therefore, it should be understood that the scope of the claims is not limited to the specific examples of the disclosed implementations, and that modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. A method for generating a three-dimensional deformable head model, characterized in that, The method includes: Receive 2D facial images; Based on an artificial intelligence (AI) model, the first set of key points in the 2D facial image is identified. In the 2D rendering of the 3D head template model, the first set of key points is corrected by applying a pre-calculated deviation; wherein the deviation is calculated based on the difference between a set of key points in the 2D rendering of the 3D head template model obtained by AI and a set of key points manually marked on multiple vertices of the 3D head template model mesh. Based on a set of user-provided keypoint annotations located on a 3D head template model, the corrected first set of keypoints is mapped to a second set of keypoints located on multiple vertices of the mesh of the 3D head template model. By reducing the difference between the first set of key points and the second set of key points, the mesh of the 3D head template model is deformed to obtain a deformed 3D head mesh model; and A hybrid shape method is applied to the deformed 3D head mesh model to obtain a head model based on the 2D facial image.

2. The method according to claim 1, characterized in that, Mapping the first set of key points to the second set of key points also includes: Associating the first set of key points on the 2D facial image with the plurality of vertices on the mesh of the 3D head template model; and Based on the corresponding features identified through the corresponding key points on the face, the first set of key points is mapped to the second set of key points.

3. The method according to claim 1, characterized in that, The deformation process includes: deforming the mesh of the 3D head template model into the deformed 3D head mesh model by using the mapping from the first set of key points to the second set of key points and by using the boundary conditions of the deformation related to the first set of key points.

4. The method according to claim 3, characterized in that, Performing deformation also includes applying different constraints during deformation optimization, which include one or more of the following: the location of key points, adjacent smoothness, characteristics, and original location.

5. The method according to claim 3, characterized in that, Performing deformation also includes applying constraints to the deformation process, said constraints being a weighted sum of one or more of the following: the location of key points, adjacent smoothness, characteristics, and original location.

6. The method according to claim 1, characterized in that, Identifying the first set of key points includes using a convolutional neural network (CNN).

7. The method according to claim 1, characterized in that, The deformations include affine deformations that do not have a Laplacian operator.

8. The method according to claim 7, characterized in that, The affine deformation is achieved by changing the smoothness parameter to achieve deformation tuning.

9. The method according to claim 1, characterized in that, The mesh of the 3D head template model is deformed without being bound to the skeleton.

10. The method according to claim 1, characterized in that, Facial deformation models include realistic style models or cartoon style models.

11. The method according to claim 1, characterized in that, Applying the hybrid shape method to the deformed 3D head mesh model includes: Based on the location of the corresponding key points, assign a corresponding blending weight to each of the second set of key points in the deformed 3D head mesh model; and Different levels of deformation are applied to the second set of key points with different mixing weights.

12. The method according to claim 1, characterized in that, Applying the hybrid shape method to the deformed 3D head mesh model includes maintaining the shape of the back side of the deformed 3D head mesh model the same as the original back side shape of the 3D head template model before deformation.

13. An apparatus for generating a three-dimensional 3D deformable head model, characterized in that, The device includes: The receiving module is used to receive two-dimensional facial images; The recognition module is used to identify a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; In the 2D rendering of the 3D head template model, the first set of key points is corrected by applying a pre-calculated deviation; wherein the deviation is calculated based on the difference between a set of key points in the 2D rendering of the 3D head template model obtained by AI and a set of key points manually marked on multiple vertices of the 3D head template model mesh. The mapping module is used to map a corrected first set of key points to a second set of key points located on multiple vertices of the mesh of the 3D head template model, based on a set of user-provided key point annotations located on the 3D head template model. A deformation module is configured to deform the mesh of the 3D head template model by reducing the difference between the first set of key points and the second set of key points to obtain a deformed 3D head mesh model; and A blend shape module is used to apply a blend shape method to the deformed 3D head mesh model to obtain a head model based on the 2D facial image.

14. The apparatus according to claim 13, characterized in that, The mapping module is used for: Associating the first set of key points on the 2D facial image with multiple vertices on the mesh of the 3D head template model; The second set of key points is identified based on the set of user-provided key point annotations on the plurality of vertices of the mesh of the 3D head template model; as well as Based on the corresponding features identified through the corresponding key points on the face, the first set of key points is mapped to the second set of key points.

15. The apparatus according to claim 13, characterized in that, The deformation module is used to: deform the mesh of the 3D head template model into the deformed 3D head mesh model by using the mapping from the first set of key points to the second set of key points and by using the boundary conditions of the deformation related to the first set of key points.

16. The apparatus according to claim 15, characterized in that, The deformation module is used to apply different constraints during the deformation optimization process, the different constraints including one or more of the following: the position of key points, adjacent smoothness, characteristics, and original position.

17. The apparatus according to claim 15, characterized in that, Performing deformation also includes applying constraints to the deformation process, said constraints being a weighted sum of one or more of the following: the location of key points, adjacent smoothness, characteristics, and original location.

18. The apparatus according to claim 13, characterized in that, Identifying the first set of key points includes using a convolutional neural network (CNN).

19. The apparatus according to claim 13, characterized in that, The deformations include affine deformations that do not have a Laplacian operator.

20. The apparatus according to claim 19, characterized in that, The affine deformation is achieved by changing the smoothness parameter to achieve deformation tuning.

21. The apparatus according to claim 13, characterized in that, The mesh of the 3D head template model is deformed without being bound to the skeleton.

22. The apparatus according to claim 13, characterized in that, Facial deformation models include realistic style models or cartoon style models.

23. The apparatus according to claim 13, characterized in that, The hybrid shape module is used for: Based on the location of the corresponding key points, assign a corresponding blending weight to each of the second set of key points in the deformed 3D head mesh model; and Different levels of deformation are applied to the second set of key points with different mixing weights.

24. The apparatus according to claim 13, characterized in that, Applying the hybrid shape method to the deformed 3D head mesh model includes maintaining the shape of the back side of the deformed 3D head mesh model the same as the original back side shape of the 3D head template model before deformation.

25. An electronic device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when executed by the processor, the computer-readable instructions cause the electronic device to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, characterized in that, When the plurality of programs are executed by the one or more processing units, the electronic device performs the method according to any one of claims 1 to 12.

Images