Method of extracting color from a two-dimensional face image of a subject, electronic device and medium

Abstract

An electronic device performs a method of extracting colors from a two-dimensional (2D) face image of a subject (e.g., a real-life person), the method including: identifying a plurality of keypoints in the 2D face image based on a keypoint prediction model; rotating the 2D face image until a plurality of target keypoints from the identified plurality of keypoints align with corresponding target keypoints of a standard face; locating a plurality of parts in the rotated 2D face image, each part defined by a respective subset of the identified plurality of keypoints; extracting a color for each part of the plurality of parts from pixel values of the 2D face image; and generating a three-dimensional (3D) model of the real-life person using the extracted colors, the 3D model matching respective facial feature colors of the 2D face image.

Description

[0001] Cross-references to related applications

[0002] This application is a continuation-into-file of U.S. Patent Application No. 17 / 202,116, filed March 15, 2021, entitled “METHODS AND SYSTEMS FOR EXTRACTINGCOLOR FROM FACIAL IMAGE”, and claims priority to that U.S. Patent Application, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to image technology, specifically to image processing and head / face model formation methods and systems, and particularly to methods, electronic devices, and media for extracting colors from two-dimensional facial images of objects. Background Technology

[0004] Commercial facial capture systems with multiple sensors (such as multi-view cameras, depth sensors, etc.) are used to obtain accurate 3D facial models of people, with or without distinctive features. These tools capture the geometry and texture information of the face from multiple sensors and fuse multimodal information into a general 3D facial model. Benefiting from multimodal information from various sensors, the resulting 3D facial model is relatively accurate. However, these commercial systems are expensive and require additional software purchases to process the raw data. Furthermore, these systems are typically deployed in facial capture studios, requiring participants or volunteers to collect data, making the data collection process time-consuming and potentially more costly. In short, facial capture systems are expensive and time-consuming for acquiring 3D facial data. Conversely, smartphones or cameras are now widely available, so there is likely a large amount of RGB (red, green, blue) images available. Using RGB images as input to generate 3D facial models benefits from a large amount of image data.

[0005] Two-dimensional (2D) RGB images are merely projections of the 3D world onto a 2D plane. Recovering 3D geometry from a 2D image is an ill-posed problem requiring optimization or learning algorithms to regularize the reconstruction process. For 3D face reconstruction, methods based on parametric facial models (3D Deformable Models, 3DMMs) have been developed and used. Specifically, facial models such as the Basel Face Model (BFM) and the Sarri Face Model (SFM) are commonly used and require commercial licensing. Facial model-based methods take a scanned set of 3D face models (showing various facial features and expressions) as their foundation, and then generate parametric representations of facial features and expressions based on the 3D facial models. New 3D faces can be represented as linear combinations of the underlying 3D facial models based on parametric representation. Due to the nature of these methods, the 3D facial models used to form the foundation and parameter spaces limit the expressiveness of facial model-based methods. Furthermore, the optimization process of fitting 3DMM parameters from input facial images or 2D landmarks further sacrifices detailed facial features in the facial images. Therefore, face model-based methods cannot accurately recover 3D facial features and require commercial licenses to use face models such as BFM and SFM.

[0006] With the popularization of deep learning algorithms, semantic segmentation algorithms have received a lot of attention. Such algorithms can classify each pixel in a facial image into different categories, such as background, skin, hair, eyes, nose, and mouth.

[0007] While semantic segmentation methods can achieve relatively accurate results, semantic segmentation of all pixels is a highly complex problem, typically requiring intricate network structures and resulting in high computational complexity. Furthermore, training a semantic segmentation network necessitates labeling a large amount of training data, and semantic segmentation requires dividing the entire image into pixels, which is tedious, time-consuming, and costly. Therefore, it is unsuitable for scenarios that require high efficiency but not high average color accuracy.

[0008] Keypoint-driven deformation methods that optimize the Laplacian operator and other derived operators have been well studied in academia. The mathematical expression for biharmonic deformation can be written as: Δ 2 x'＝0 The key constrained points, i.e., the boundary conditions, can be expressed as follows: x b '＝x bc In the above equation, Δ is the Laplace operator. x' The location of the unknown deformed mesh vertices, and x bcThese are the locations of key points given after deformation. The double Laplace equation needs to be solved in each dimension. The double harmonic function is a solution to the double Laplace equation, but it is also the smallest element of the so-called "Laplace energy".

[0009] The property of energy minimization is mesh smoothing. Applying the aforementioned minimum element directly would eliminate all detail features. Furthermore, when the positions of key points remain unchanged, the deformed mesh is expected to be identical to the original mesh. Among these considerations, the preferred use of biharmonic deformation is for solving for the displacements of vertices other than their positions. In this way, the deformed positions can be written as... x'=x+d , where d is the displacement of the unknown vertex in each dimension. Naturally, the equations for biharmonic deformation are subject to... d b =x bc x b Constraints Δ 2 d＝0 ,in d b It is the displacement of the key points after deformation.

[0010] With the rapid development of the gaming industry, custom face avatar generation has become increasingly popular. For the average player without artistic skills, it's difficult to adjust the control parameters to generate faces that can depict subtle variations.

[0011] In some existing face generation systems and methods, such as the Justice face generation system, facial model prediction involves predicting 2D information in an image, such as eyebrows, mouth, nose, and segmentation of other pixels in the photo. These 2D segments are susceptible to out-of-plane rotation and partial occlusion, and essentially require a frontal view. Furthermore, since the similarity between the final game face avatar and the input is determined by a face recognition system, this method limits its application to only realistic-style games. If the game's style is a cartoonish style significantly different from realistic faces, this method cannot be used.

[0012] In some other existing face generation systems and methods, such as a game face generation system, a realistic face is reconstructed from an input image. This method is limited to realistic-style games and cannot be applied to cartoon-style games. Furthermore, the output parameter of this method is a reconstructed game-style facial mesh, and then template matching is performed on each part of the mesh. This method restricts the combination of different facial parts. The overall diversity of game faces is closely related to the number of pre-generated templates. If a part (e.g., the mouth shape) has a small number of templates, it can produce very few different variations, resulting in a lack of diversity in the generated faces. Summary of the Invention

[0013] Learning-based face reconstruction and keypoint detection methods rely on 3D ground truth data as the gold standard to train models that approximate the ground truth as closely as possible. Therefore, 3D ground truth determines the upper limit of learning-based methods. To ensure the accuracy of face reconstruction and the desired keypoint detection, in some embodiments, 2D facial keypoint annotations are used to generate the ground truth of the 3D face model, without the need for expensive face capture systems. The method disclosed in this paper generates a 3D ground truth face model that preserves the detailed facial features of the input image, overcoming the shortcomings of existing face models (e.g., 3DMM-based methods that lose facial features), and also avoiding the use of parametric face models (e.g., BFM and SFM, both of which require commercial licenses) required by some existing face model-based methods.

[0014] In addition to facial landmark detection, some embodiments implement multi-task learning and transfer learning solutions for facial feature classification tasks, enabling the extraction of more information from input facial images supplemented with landmark information. For computer or mobile games used to create player facial avatars, detected facial landmarks, along with predicted facial features, are valuable.

[0015] In some embodiments, this document discloses a lightweight method for extracting the average color of each part of a face from a single photograph, including the average color of skin, eyebrows, pupils, lips, hair, and eyeshadow. Simultaneously, an algorithm is used to automatically transform a texture map based on the average color, such that the transformed texture retains the original brightness and color difference, but the dominant color becomes the target color.

[0016] With the rapid development of computer vision and artificial intelligence (AI) technologies, the capture and reconstruction of 3D facial key points has reached a high level of accuracy. More and more games are utilizing AI detection to make game characters more lifelike. The method and system disclosed in this paper customize 3D head avatars based on reconstructed 3D key points. Key point-driven deformations are typically applicable to arbitrary meshes. The head avatar customization and deformation methods proposed in this paper can find applications in scenarios such as automatic avatar creation and facial expression reproduction.

[0017] This paper discloses a method and system for automatically generating facial avatars in games based on a single photograph. Through the prediction of facial key points, automatic processing of these key points, and the use of deep learning methods to predict model parameters, the system automatically generates facial avatars in games that: 1) possess the characteristics of a realistic face in the photograph; and 2) conform to the target game style. This system can be applied to facial generation in both realistic and cartoon-style games and can be easily and automatically adjusted according to different game models or skeletal definitions.

[0018] According to a first aspect of this application, a method for constructing a facial location map from a two-dimensional (2D) facial image of a real person includes: generating a coarse facial location map from the 2D facial image; predicting a first set of key points in the 2D facial image based on the coarse facial location map; identifying a second set of key points in the 2D facial image based on key point annotations provided by a user; and updating the coarse facial location map to reduce the difference between the first set of key points and the second set of key points in the 2D facial image.

[0019] In some embodiments, the method for constructing a facial location map from a 2D facial image of a real person further includes: extracting a third set of key points based on the updated facial location map as a final set of key points, wherein the third set of key points has the same position as the first set of key points in the facial location map.

[0020] In some embodiments, the method for constructing a facial location map from a 2D facial image of a real person further includes: reconstructing a three-dimensional (3D) facial model of a real person based on the updated facial location map.

[0021] According to a second aspect of this application, a method for extracting color from a two-dimensional (2D) facial image of an object includes: identifying a plurality of keypoints in the 2D facial image based on a keypoint prediction model; rotating the 2D facial image until a plurality of target keypoints from the identified plurality of keypoints are aligned with corresponding target keypoints of a standard face; locating a plurality of portions in the rotated 2D facial image, wherein each portion is defined by a corresponding subset of the identified plurality of keypoints; extracting the color of each of the plurality of portions based on pixel values ​​of the 2D facial image, wherein each portion is defined by a corresponding subset of keypoints; and generating a three-dimensional (3D) model of the object using the colors extracted from the plurality of portions of the 2D facial image, the 3D model matching the corresponding facial feature colors of the 2D facial image.

[0022] According to a third aspect of this application, a method for generating a three-dimensional (3D) deformable head model includes: receiving a two-dimensional (2D) facial image; identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; mapping the first set of key points to a second set of key points based on a set of user-provided key point annotations located on a mesh of a 3D head template model; performing deformation on the mesh of the 3D head template model by reducing the difference between the first set of key points and the second set of key points to obtain a deformed 3D head mesh model; and applying a blendshape method to the deformed 3D head mesh model to obtain a personalized head model based on the 2D facial image.

[0023] According to a fourth aspect of this application, a method for customizing the standard face of an avatar in a game using a two-dimensional (2D) facial image of a real-life person includes: identifying a set of real-life key points in the 2D facial image; converting this set of real-life key points into a set of game style key points associated with the avatar in the game; generating a set of control parameters for the standard face of the avatar in the game by applying this set of game style key points to a keypoint-to-parameter (K2P) neural network model; and deforming the standard face of the avatar in the game based on this set of control parameters, wherein the deformed face of the avatar has facial features of the 2D facial image.

[0024] According to a fifth aspect of this application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When the programs are executed by the one or more processing units, the programs cause the electronic device to perform one or more methods as described above.

[0025] According to a sixth aspect of this application, a non-transitory computer-readable storage medium stores a plurality of programs, which are executed by an electronic device having one or more processing units. When the programs are executed by the one or more processing units, the programs cause the electronic device to perform one or more methods as described above.

[0026] It should be noted that the above embodiments can be combined with any other embodiments described herein. The features and advantages described in the specification are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, particularly when considering the drawings, specification, and claims. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and guidance purposes and may not have been chosen to depict or limit the subject matter of this disclosure. Attached Figure Description

[0027] To gain a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and should not be considered limiting, as the description may accommodate other valid features.

[0028] Figure 1 This is a diagram illustrating exemplary key point definitions according to some implementations of this disclosure.

[0029] Figure 2 This is a block diagram illustrating an exemplary keypoint generation process according to some implementations of this disclosure.

[0030] Figure 3 This is a diagram illustrating an exemplary process for transforming an initial coarse location map according to some implementations of this disclosure.

[0031] Figure 4 This is a diagram illustrating an exemplary transformed positional diagram that does not cover the entire facial region according to some implementations of this disclosure.

[0032] Figure 5 This is a diagram illustrating an exemplary process of refining a transformed position map to cover the entire facial region according to some implementations of this disclosure.

[0033] Figure 6 This is a diagram illustrating some exemplary results of a location map refinement algorithm according to some implementations of this disclosure.

[0034] Figure 7A and Figure 7B Some exemplary comparisons of final location maps and initial rough location maps according to some implementations of this disclosure are shown.

[0035] Figure 8A This is a diagram illustrating an exemplary eyeglasses classification network structure according to some implementations of this disclosure.

[0036] Figure 8B This is a diagram illustrating an exemplary female hair prediction network structure according to some implementations of this disclosure.

[0037] Figure 8C This is a diagram illustrating an exemplary male hair prediction network structure according to some implementations of this disclosure.

[0038] Figure 9A Some exemplary glasses classification prediction results are shown according to some implementations of this disclosure.

[0039] Figure 9B Some exemplary female hair prediction results are shown according to some implementations of this disclosure.

[0040] Figure 9C Some exemplary male hair prediction results are shown according to some implementations of this disclosure.

[0041] Figure 10 This is a flowchart illustrating an exemplary process for constructing a facial location map from a 2D facial image of a real person, according to some implementations of this disclosure.

[0042] Figure 11 This is a flowchart illustrating an exemplary color extraction and adjustment process according to some implementations of this disclosure.

[0043] Figure 12 An exemplary skin color extraction method according to some implementations of this disclosure is shown.

[0044] Figure 13 An exemplary eyebrow color extraction method according to some implementations of this disclosure is shown.

[0045] Figure 14 An exemplary pupil color extraction method according to some implementations of this disclosure is shown.

[0046] Figure 15 An exemplary hair color extraction region used in a hair color extraction method according to some implementations of this disclosure is shown.

[0047] Figure 16 An exemplary separation between hair pixels and skin pixels within a hair color extraction region is shown according to some implementations of this disclosure.

[0048] Figure 17 An exemplary eyeshadow color extraction method according to some implementations of this disclosure is shown.

[0049] Figure 18 Some exemplary color adjustment results are shown according to some implementations of this disclosure.

[0050] Figure 19 This is a flowchart illustrating an exemplary process for extracting colors from a 2D facial image of a real person according to some implementations of this disclosure.

[0051] Figure 20 This is a flowchart illustrating an exemplary head avatar deformation and generation process according to some implementations of this disclosure.

[0052] Figure 21 This is a diagram illustrating exemplary header template model components according to some implementations of this disclosure.

[0053] Figure 22This is a diagram illustrating some exemplary key points marked on a realistic 3D model and a cartoon-style 3D model according to some implementations of this disclosure.

[0054] Figure 23 This is a diagram illustrating an exemplary comparison between template model rendering, manually marked keypoints, and AI-detected keypoints according to some implementations of this disclosure.

[0055] Figure 24 This is a diagram illustrating an exemplary affine transformation of a triangle according to some implementations of this disclosure.

[0056] Figure 25 This is a diagram illustrating an exemplary comparison of deformation results of some head models with and without a hybrid deformation process according to some implementations of this disclosure.

[0057] Figure 26 This is a diagram illustrating an exemplary comparison of affine and biharmonic deformations with different weights according to some implementations of this disclosure.

[0058] Figure 27 The following are some exemplary results shown, generated automatically from a set of randomly selected images of women using a realistic template model according to some implementations of this disclosure.

[0059] Figure 28 This is a flowchart illustrating an exemplary process for generating a 3D deformable head model from a 2D facial image of a real person, according to some implementations of this disclosure.

[0060] Figure 29 This is a diagram illustrating exemplary key point processing steps according to some implementations of this disclosure.

[0061] Figure 30 This is a diagram illustrating an exemplary keypoint smoothing process according to some implementations of this disclosure.

[0062] Figure 31 This is a block diagram illustrating an exemplary key-to-control parameter (K2P) conversion process according to some implementations of this disclosure.

[0063] Figure 32 Some exemplary results of automatic face generation in mobile games according to some implementations of this disclosure are shown.

[0064] Figure 33 This is a flowchart illustrating an exemplary process of customizing a standard face for an avatar in a game using a 2D facial image of a real-life person, according to some implementations of this disclosure.

[0065] Figure 34This is a schematic diagram of an exemplary hardware structure of an image processing apparatus according to some implementations of the present disclosure.

[0066] By convention, features shown in the figures may not be drawn to scale. Therefore, for clarity, the dimensions of individual features may be arbitrarily enlarged or reduced. Furthermore, some figures may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and figures to indicate similar features. Detailed Implementation

[0067] Now, reference will be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives can be used without departing from the scope of the claims, and that the subject matter can be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices.

[0068] It is understood that in the embodiments of this application, data such as facial information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0069] Before further describing the embodiments of this application in detail, the names and terms involved in the embodiments of this application are described, and the names and terms involved in the embodiments of this application have the following interpretations.

[0070] Facial key points: Predefined landmarks that define the shape of certain facial features (such as the corners of the eyes, chin, tip of the nose, and corners of the mouth).

[0071] Facial features: facial borders, eyes, eyebrows, nose, mouth, and other parts.

[0072] Facial reconstruction: Reconstructing the 3D geometry of a human face, typically represented by mesh models, point clouds, or depth maps.

[0073] RGB image: a three-channel image format consisting of red, green, and blue channels.

[0074] Location map: Using red, green, and blue channels in a standard image format, the x, y, and z coordinates of the facial region are stored. The location map is a representation of the 3D face.

[0075] Facial feature classification: including hairstyle classification and whether glasses are worn or not.

[0076] Convolutional Neural Networks (CNNs): A type of deep neural network, most commonly used for analyzing visual images.

[0077] Base network: A network such as a CNN, which is used by one or more downstream tasks to act as a feature extractor.

[0078] Laplace operator: A difference operator given by the divergence of the gradient of a function on Euclidean space.

[0079] Differentiable manifolds: a class of topological spaces that are locally similar to linear spaces to allow for calculus.

[0080] Biharmonic function: A quartic differentiable function defined on a differentiable manifold, whose square Laplace operator is equal to 0.

[0081] Keypoint-driven deformation: A class of methods that deforms a mesh by changing the position of certain vertices.

[0082] Double harmonic deformation: A deformation method is adopted to optimize the double harmonic function with some boundary conditions.

[0083] Affine deformation: The keypoint-driven deformation method proposed in this disclosure optimizes the affine transformation of triangles to achieve mesh deformation.

[0084] Facial model: A standard face in a mesh within a predetermined objective game.

[0085] Bone / Slider: Control parameters used to deform the facial model.

[0086] As mentioned above, even when both the input 2D image and 2D keypoints are fed into the optimization process to fit the 3DMM parameters, the optimization must strike a balance between fitting the 3D facial model based on the base (i.e., the 3D facial model set) and the fidelity of the 2D keypoints. This optimization results in an 3D facial model that is adversarial to the 2D input keypoints, thereby sacrificing the detailed facial information provided by the input 2D keypoints. In existing 3D facial reconstruction methods, facial capture schemes can produce accurate reconstructions, but they are expensive and time-consuming, and the obtained data also shows limited variation in facial features (limited number of participants). On the other hand, facial model-based methods can use 2D images or 2D landmark annotations as input, but the resulting 3D models are inaccurate. To meet the demands of the rapid development of computer / mobile games, it is necessary to produce the desired 3D model accuracy while reducing the required cost and time. To meet these requirements, the novel 3D ground truth face model generation algorithm disclosed in this paper takes 2D images, 2D key point annotations, and a coarse 3D face model (location map format) as input. It transforms the coarse 3D model based on the 2D key points to finally generate a 3D face model in which detailed facial features are well preserved.

[0087] Beyond addressing key challenges in facial reconstruction and keypoint prediction, this paper discloses multi-task learning and transfer learning-based methods for facial feature classification, partially built upon the facial reconstruction and keypoint prediction framework. Specifically, the underlying network for facial reconstruction and keypoint prediction is reused to classify glasses (whether glasses are worn or not) through multi-task learning. Training a linear classifier on top of the existing facial reconstruction and keypoint prediction framework effectively reuses the existing model, avoiding the introduction of another large network for image feature extraction. Furthermore, another shared underlying network is used for male and female hairstyle classification. Hairstyles are an important facial feature that complements facial keypoints or 3D facial models. Adding hairstyle and glasses prediction during the creation of 3D avatars for users better reflects their facial features and provides a more personalized experience.

[0088] For decades, facial landmark prediction has been a research topic in computer vision. In recent years, with the development of artificial intelligence and deep learning, convolutional neural networks (CNNs) have become increasingly useful for facial landmark prediction. 3D face reconstruction and facial landmark detection are two closely related problems; solving one simplifies the other. The traditional approach is to first solve 2D facial landmark detection and then use the estimated 2D facial landmarks to further infer a 3D facial model. However, when the face in an image is tilted (nodding or shaking), some facial landmarks are occluded, leading to incorrect 2D facial landmark estimations. Therefore, the 3D facial model built on incorrect 2D facial landmarks becomes inaccurate.

[0089] Because ground truth data sets the upper limit for deep learning-based methods, existing 3D face model datasets are not only limited in quantity but also only available for academic research. On the other hand, face model-based methods require the use of Basel Face Models (BFM) and Surrey Face Models (SFM), both of which require commercial licenses. High accuracy and a large volume of 3D ground truth data become paramount when training any face reconstruction or keypoint estimation model.

[0090] Besides facial landmark prediction, facial feature classification is a crucial aspect of creating user 3D avatars. For the predicted facial landmarks, style transfer can be performed only on the user's facial features (i.e., eyes, eyebrows, nose, mouth, and facial contours). However, to better reflect the user's facial features, matching the user's hairstyle and adding glasses (if the user is wearing glasses in the input image) is very helpful. Based on these requirements, a facial feature classification method based on multi-task learning and transfer learning was developed to achieve male / female hairstyle prediction and glasses prediction (with or without), making the created avatar more personalized and improving the user experience.

[0091] In some embodiments, key points are used to represent the three-dimensional shape of the main parts of the face, such as... Figure 1 As shown. Figure 1 This is a diagram illustrating exemplary keypoint definitions according to some implementations of this disclosure. Keypoints are numbered sequentially to define specific facial features. In other words, there is a mapping between the keypoint numbers and specific locations on the face. For example, number 9 corresponds to the base of the chin, number 21 corresponds to the tip of the nose, and so on. Keypoints focus on the boundaries of major facial features, such as the facial contours, the eye contours, and the eyebrow contours. More keypoints mean greater prediction difficulty, but a more accurate shape representation. In some embodiments, Figure 1 The code uses a definition of 96 key points. In some embodiments, users can modify specific definitions and the number of key points according to their needs.

[0092] Many algorithms can predict the 3D coordinates of facial keypoints. Methods with better performance use deep learning algorithms based on large amounts of offline 3D training data. However, in some embodiments, any 3D keypoint prediction algorithm can be used. In some embodiments, the definition of keypoints is not fixed, and users can customize the definition according to their necessity.

[0093] To address the problem of generating ground truth facial models in 3D, the following automated algorithm was developed, which takes 2D RGB images, 2D keypoint annotations, and a coarse location map as input. Figure 2This is a block diagram illustrating an exemplary keypoint generation process according to some implementations of this disclosure. For example, a 2D RGB image of a face is used as input image 202, and the 2D RGB image has a corresponding initial coarse location map 204, where each pixel in the initial coarse map represents the spatial coordinates of the corresponding facial point in the 2D RGB image. 2D keypoint annotation 208 represents a set of user-provided keypoints used to correct the set of keypoints 206 detected from the initial coarse map 204.

[0094] Figure 3 This is a diagram illustrating an exemplary process for transforming an initial coarse location map according to some implementations of this disclosure.

[0095] In some embodiments, the 3D reconstruction method is used to convert an input facial image into a location map containing 3D depth information of facial features. For example, the location map can be a 2D three-color (RGB) channel map with a 256×256 matrix array, where each array element has coordinates (x, y, z) representing a 3D position on the facial model. The 3D position coordinates (x, y, z) are represented by the RGB pixel values ​​on the location map for each array element. Specific facial features are located at fixed 2D positions within the 2D location map. For example, the tip of the nose can be identified by the position of a 2D array element located at x=128 and y=128 within the location map. Similarly, specific keypoints identified for specific facial features on the face can be located at the same array element positions on the 2D location map. However, depending on the different input facial images for the location map, specific keypoints can have different 3D position coordinates (x, y, z).

[0096] In some embodiments, such as Figure 2 and Figure 3 As shown, an initial coarse location map (204, 304) is obtained from the input image (202, 302) using a 3D reconstruction method. Then, the (x, y) coordinates of the corresponding keypoints (206, 306) in the initial location map are adjusted using the input 2D keypoint annotations (208, 308) to ensure that the adjusted (x, y) coordinates of the keypoints in the adjusted location map are the same as those of the annotated 2D keypoints. Specifically, first, a set of 96 keypoints is obtained from the initial location map P. Based on the keypoint index, this set of 96 keypoints is called K = k_i, where each k_i is the 2D coordinate (x, y) of the keypoint, i = 0, ..., 95. From the 2D keypoint annotations (208, 308), a second set of 96 keypoints A = a_i is obtained, where a_i is the 2D (x, y) coordinate, i = 0, ..., 95. Secondly, the spatial transformation mapping from K to A is estimated (210. 310), which is defined as T: Ω → Ω, where Ω R^2. The obtained transformation T is then applied to the initial position map P to obtain the transformed position map P'(212, 312). In this way, the transformed position map P'(212, 312) preserves the detailed facial features of the person in the input image (202, 302), while also possessing reasonable 3D depth information. Therefore, the solution disclosed in this paper provides an accurate and practical alternative for generating 3D ground truth information, avoiding the use of expensive and time-consuming face capture systems.

[0097] In some embodiments, since the 96 facial key points only cover a portion of the entire facial area (i.e., below the eyebrows, within the facial contour), for example in Figure 3 In the image, keypoints from the ear to the chin follow the jawline but are not on the visible facial contour. When the face in the input image is tilted, the entire facial region is not covered by the contours of the connected keypoints. Furthermore, when performing manual keypoint annotation, regardless of whether the face in the image is tilted, keypoints can only be labeled along the visible facial contour (i.e., there is no way to accurately label occluded keypoints). Therefore, in the transformed location map P'(212, 312), a portion of the facial region has no valid values ​​because the transformation map T(210, 310) does not estimate in this region. Additionally, the forehead region is located above the eyebrows, so T also does not estimate in this region. All these issues result in the transformed location map P'(212, 312) having no valid values ​​in certain regions. Figure 4 This is a diagram illustrating an exemplary transformed positional diagram that does not cover the entire facial region according to some implementations of this disclosure.

[0098] exist Figure 4 In the middle, the top circle (402, 406) highlights the forehead area, and the right circle (404, 408) indicates the area where the key point outline is smaller than the visible facial contour.

[0099] In some embodiments, to address the aforementioned issues and make the algorithm robust to tilted faces, which are commonly found in facial images, methods such as... Figure 2 The refinement process 214 is shown. Based on the head pose and a rough 3D facial model, keypoints from the transformed location map are shifted along the facial contour to match the visible facial contour. Missing values ​​in the facial contour regions can then be filled in the obtained location map. However, values ​​in the forehead region remain missing. To cover the forehead region, control points are expanded by adding eight landmarks from the four corners of the image to two keypoint sets K and A.

[0100] Figure 5This diagram illustrates an exemplary process for refining a transformed position map to cover the entire facial region, according to some implementations of this disclosure. The position map refining process is performed in... Figure 5 As shown in the image.

[0101] In some embodiments, the head pose is first determined based on a coarse position map P to determine whether the head is tilted to the left or right, which is defined in the 3D facial model space (e.g., as shown in the image). Figure 5 As shown, the face is tilted to the left. Based on determining whether the face is tilted to the left or right, the key points on the corresponding side of the facial contour are adjusted. The key points on the right side of the facial contour have indices from 1 to 8, and the key points on the left side have indices from 10 to 17. Using a face tilted to the left as an example, the 2D projection of the initial position map P is calculated to obtain the depth map. Figure 5 Image 502 is shown. The left facial contour keypoints k_i, i = 10, ..., 17 are individually shifted to the right until they reach the boundary of the depth map. The original keypoint positions are then replaced with the new coordinates. Similarly, when the face is tilted to the right, the processed keypoints are indexed by k_i, i = 1, ..., 8, with the search direction to the left. After adjusting the facial contour keypoints, the updated keypoints are visualized as follows: Figure 5 Image 504 in the image, the updated overlay of the location map is shown as follows Figure 5 Image 506 in the image. The updated location map shows better facial coverage in the facial contour region, but the forehead region still has missing values.

[0102] In some embodiments, to cover the forehead region, two anchor points are added at each corner of the image domain Ω as additional keypoints k_i, i = 96, ..., 103, to obtain an updated keypoint set K' (e.g., ...). Figure 5 (See image 508). For manually annotated keypoint sets a_i, i = 96, ..., 103, the same process is performed to obtain the updated A'. Using the updated keypoint sets K' and A', the transformation map T' is re-estimated and then applied to the initial location map P to obtain the final location map P''. Figure 2 216) to cover the entire facial area (e.g. Figure 5 (As shown in image 510). The final keypoint 218 is derived from the final position diagram 216.

[0103] Figure 6 These are diagrams illustrating some exemplary results of a position map refinement algorithm according to some implementations of this disclosure. 602 is a diagram of the initial transformed position map. 604 is a diagram of the updated position map after fixing the facial contours. 606 is a diagram of the final position map.

[0104] Figure 7A and Figure 7B Some exemplary comparisons of final location maps and initial coarse location maps according to some implementations of this disclosure are shown. Figure 7A In one example, the nose and its associated 3D model and keypoint 702 in the initial location image are incorrect and do not reflect human facial features at all (highlighted by the arrow). However, after applying the method described herein, the nose is well aligned with the image and its associated 3D model and keypoint 704 in the final location image (highlighted by the arrow). Figure 7B In the second example, several inaccuracies exist in the initial location map and its associated 3D model and keypoint 706, such as mismatched facial contours, open mouth, and nose shapes (indicated by arrows). In the final location map and its associated 3D model and keypoint 708, all these errors have been corrected (indicated by arrows).

[0105] For mobile game applications used in the facial avatar creation process, hairstyle and glasses classification are particularly important. In some embodiments, this paper implements a solution based on multi-task learning and transfer learning to address these issues.

[0106] In some embodiments, four different classification tasks (head) are implemented for predicting female hair. The classification categories and parameters are as follows: Category Header 1: Curve Straight (0); Curved (1) Category Header 2: Length Short (0); Long (1) Category Header 3: Bangs No bangs or split (0); left part (1); right part (2); M-shape (3); straight bangs (4); natural bangs (5); airy bangs (6) Category 4: Braids A single braid (0); two or more braids (1); a single bun (2); two or more buns (3); other (4).

[0107] In some embodiments, for male hair prediction, three different classification tasks (head) are implemented. The classification categories and parameters are as follows: Category Head 1: Very Short (0), Curly (1), Other (2) Category 2: No bangs (0), Parted bangs (1), Natural bangs (2) Category 3: Left-parted bangs (0), Right-parted bangs (1).

[0108] In some embodiments, glasses classification is a binary classification task. The classification parameters are as follows: No glasses (0); wearing glasses (1).

[0109] Among various deep learning image classification models, those that achieve state-of-the-art accuracy on ImageNet typically have large model sizes and complex architectures, such as EfficientNet, Noisy Student, and FixRes. When deciding which architecture to use as the base network for the feature extractor, a balance needs to be struck between prediction accuracy and model size. In practice, a 1% improvement in classification accuracy may not have a noticeable impact on the end user, but the model size can increase exponentially. Considering that the trained model may need to be deployed on the client side, a smaller base network allows for flexible deployment on both the server and client sides. Therefore, for example, MobileNetV2 can be used as the base network for transfer learning on different classification heads. The MobileNetV2 architecture is based on an inverse residual structure, where the input and output of the residual block are fine-grained bottleneck layers, unlike traditional residual models that use extended representations in the input. MobileNetV2 uses lightweight deep convolutions to filter features in the intermediate extended layers.

[0110] For glasses classification, a multi-task learning approach is used. The keypoint prediction network is reused as the base network with its parameters frozen. A binary classifier is trained using feature vectors with cross-entropy loss in the bottleneck layer of the U-shaped network. Figure 8A This is a diagram illustrating an exemplary eyeglasses classification network structure according to some implementations of this disclosure. Figure 8B This is a diagram illustrating an exemplary female hair prediction network structure according to some implementations of this disclosure. Figure 8C This is a diagram illustrating an exemplary male hair prediction network structure according to some implementations of this disclosure.

[0111] Figure 9A Some exemplary glasses classification prediction results are shown according to some implementations of this disclosure. Figure 9B Some exemplary female hair prediction results are shown according to some implementations of this disclosure. Figure 9C Some exemplary male hair prediction results are shown according to some implementations of this disclosure.

[0112] Figure 10This is a flowchart 1000 illustrating an exemplary process for constructing a facial location map from 2D facial images of real-life people according to some implementations of this disclosure. In real life, different people have different facial features, causing the same keypoints corresponding to the same facial features (e.g., the position of eyebrows on a face) to have vastly different spatial coordinates. Because the 2D facial images used to generate 3D facial models are captured at different angles and under different lighting conditions, the face detection problem becomes more challenging, and research in this field is a very active area in the field of computer vision. In this application, various methods have been proposed to improve the efficiency and accuracy of facial keypoint detection from any 2D facial image ranging from real-life people to cartoon characters. In some embodiments, a set of user-provided facial keypoints from the same facial image is provided as a reference for correcting or improving the set of facial keypoints initially detected by a computer-implemented method. For example, since there is a one-to-one mapping between user-provided facial key points and computer-generated facial key points based on their respective indexes, the refinement of computer-generated facial key points is defined as an optimization problem of reducing the differences between the two sets of facial key points (e.g., measured by the spatial coordinates corresponding to the facial key points in the location map).

[0113] The process of constructing a facial location map includes step 1010: generating a rough facial location map from a 2D facial image.

[0114] The process also includes step 1020: predicting the first set of key points in the 2D facial image based on a rough facial location map.

[0115] The process additionally includes step 1030: identifying a second set of key points in the 2D facial image based on user-provided key point annotations.

[0116] The process further includes step 1040: updating the coarse facial location map to reduce the difference between the first set of keypoints and the second set of keypoints in the 2D facial image. For example, by reducing the difference between the first and second sets of keypoints in the 2D facial image in terms of the spatial coordinates corresponding to the facial keypoints, the first set of keypoints in the 2D facial image based on the coarse facial location map is modified based on user-provided keypoint annotations to make it more similar to the second set of keypoints in the 2D facial image, which is generally considered more accurate. The modification of the first set of facial keypoints automatically triggers an update of the initial coarse facial location map, from which the first set of keypoints is generated. The updated coarse facial location map can then be used to predict a more accurate set of keypoints from the 2D facial image. It should be noted that the second set of keypoints in the 2D facial image based on user-provided keypoint annotations does not mean that the second set of keypoints was manually annotated. Instead, the user can use another computer-implemented method to perform the annotation. In some embodiments, although the number of the second set of keypoints (e.g., 10 to 20) is only a fraction of the number of the first set of keypoints (e.g., 96 or even more), the fact that the second set of keypoints is more accurate contributes to the overall improvement of the first set of keypoints.

[0117] In one implementation, the process further includes step 1050: extracting a third set of keypoints as a final keypoint set based on the updated facial location map / final location map, wherein the third set of keypoints has the same position as the first set of keypoints in the facial location map. In some embodiments, the positions of the keypoints in the facial location map are represented by the 2D coordinates of array elements in the location map. As described above, the updated facial location map benefits from the second set of keypoints in the 2D facial image annotated based on user-provided keypoints, thus the third set of keypoints is more accurate and can be used in fields such as computer vision for more accurate face detection, or in computer graphics for more accurate 3D facial modeling.

[0118] In one implementation, as an alternative to or addition to step 1050, the process further includes step 1060: reconstructing a 3D facial model of a real-life person based on the updated facial location map. In one example, the 3D facial model is a 3D depth model.

[0119] Additional implementations may include one or more of the following features.

[0120] In some embodiments, the update step 1040 may include: converting the coarse facial location map into a transformed facial location map, and refining the transformed facial location map. As described above, compared to the initial coarse facial location map, the transformed facial location map can retain more detailed facial features of the person in the input image, thus the 3D facial model based on the transformed facial location map is more accurate.

[0121] In some embodiments, the transformation includes: estimating a transformation mapping from a rough facial location map to a transformed facial location map based on learning the differences between a first set of keypoints and a second set of keypoints; and applying the transformation mapping to the rough facial location map.

[0122] In some embodiments, refinement includes: adjusting key points on the occluded side of the facial contour, corresponding to the transformed facial position map, to cover the entire facial region, corresponding to the determination that the 2D facial image is tilted. As mentioned above, different 2D facial images can be captured at different angles, and this refinement step can correct for deviations or errors introduced by different image capture conditions, preserving a more accurate 3D facial model from the 2D facial image. Furthermore, compared to the initial coarse facial position map, the transformed facial position map can preserve more detailed facial features of the person in the input image, thus resulting in a more accurate 3D facial model based on the transformed facial position map.

[0123] In some embodiments, the first set of key points may include 96 key points.

[0124] In some embodiments, the process of constructing a facial location map may include facial feature classification.

[0125] In some embodiments, facial feature classification is performed using deep learning methods.

[0126] In some embodiments, facial feature classification is performed using multi-task learning or transfer learning methods.

[0127] In some embodiments, facial feature classification includes hair prediction classification.

[0128] In some embodiments, hair prediction classification includes female hair prediction with multiple classification tasks, which may include: curve, length, bangs, and braids.

[0129] In some embodiments, hair prediction classification includes male hair prediction with multiple classification tasks, which may include: curve / length, bangs, and parting.

[0130] In some embodiments, facial feature classification includes glasses prediction classification. Glasses prediction classification includes a classification task, which may include: wearing glasses, and not wearing glasses.

[0131] The method and system disclosed in this paper can generate accurate 3D facial models (i.e., location maps) based on 2D keypoint annotations for use in 3D ground truth generation. This method not only avoids the use of BFM and SFM facial models, but also better preserves detailed facial features, preventing the loss of these important features caused by facial model-based methods.

[0132] In addition to providing key points, using deep learning-based solutions to provide supplementary facial features such as hairstyle and glasses is essential for personalizing facial avatars based on user-input facial images.

[0133] While this paper discloses hairstyle and glasses prediction for facial feature classification as examples, the framework is not limited to these exemplary tasks. The framework and solution are based on multi-task learning and transfer learning, meaning it is easy to extend the framework to include other facial features, such as female makeup type classification, male beard type classification, and classification with or without a mask. Based on the requirements of various computer or mobile games, the framework is well-suited for expansion to a wider range of tasks.

[0134] In some embodiments, this paper introduces a keypoint-based lightly weighted color extraction method. The lightly weighted image processing algorithm quickly estimates local pixels without segmenting all pixels, thus achieving higher efficiency.

[0135] During the training process, users do not need to have pixel-level labels, but only mark a few key points, such as the corners of the eyes, the borders of the mouth, and the eyebrows.

[0136] The lightly weighted color extraction method disclosed in this paper can be used in personalized facial generation systems for various games. To provide more freedom in personalized character generation, many games have begun to adopt free adjustment methods. In addition to adjusting facial shape, users can also choose different color combinations. For aesthetic purposes, game faces often use pre-defined textures instead of real facial textures. The method and system disclosed in this paper allow users to automatically extract the average color of each part of the face simply by uploading a photo. Simultaneously, the system can automatically modify the texture based on the extracted colors, so that each part of the personalized face is generated with colors closer to the real colors in the user's photo, improving the user experience. For example, if a user's skin color is darker than the average skin color of most people, the skin color of the character in the game will be darkened accordingly. Figure 11 This is a flowchart illustrating an exemplary color extraction and adjustment process according to some implementations of this disclosure.

[0137] To locate the various parts of the face, key points are defined for the main facial features, as described above. Figure 1As shown, the algorithm described above is used for keypoint prediction. Unlike semantic segmentation methods, it predicts keypoints only in the image without classifying each pixel, significantly reducing the cost of labeling training data and prediction. Using these keypoints, various parts of the face can be roughly located.

[0138] Figure 12 An exemplary skin color extraction method according to some implementations of this disclosure is shown. In order to extract features from an image, the facial region in the original image 1202 needs to be rotated so that key points 1 and 17 on the left and right sides of the face are aligned with corresponding key points on the left and right sides of a standard face, as shown in the image after initial alignment 1204.

[0139] Next, the area for skin color pixel inspection is determined. The bottom coordinates of the key points of the eyes are selected as the upper boundary of the detection area, and the bottom key points of the nose are selected as the lower boundary of the detection area. The left and right boundaries are determined by the facial boundary key points. In this way, the skin color detection area is obtained, as shown in region 1208 on image 1206.

[0140] Within region 1208, not all pixels are skin pixels; some may include eyelashes, nostrils, nasolabial folds, hair, etc. Therefore, the median R, G, and B values ​​of all pixels in this region are selected as the final predicted average skin color.

[0141] Figure 13 An exemplary eyebrow color extraction method according to some implementations of this disclosure is illustrated. For the average eyebrow color, the primary eyebrow (i.e., the eyebrow closer to the camera side) is first selected as the target. In some embodiments, if both eyebrows are primary eyebrows, the pixels of both eyebrows are extracted. Assuming the left eyebrow is the primary eyebrow, then... Figure 13 As shown, a quadrilateral region consisting of keypoints 77, 78, 81, and 82 is selected as the eyebrow pixel search area. This is because the eyebrows near the outer edges are too thin and will amplify the impact of small keypoint errors. Since the eyebrows near the inner edges can typically be sparser and blend with the skin color, the middle eyebrow region 1302 is selected to collect pixels. Each pixel must first be compared with the average skin color, and only pixels with a difference greater than a certain threshold are collected. Finally, similar to skin color, the median R, G, and B values ​​of the collected pixels are selected as the final average eyebrow color.

[0142] Figure 14Exemplary pupil color extraction methods according to some implementations of this disclosure are illustrated. Similar to eyebrow color extraction, when extracting pupil color, the side closer to the primary eye in the lens is first selected. In some embodiments, if both eyes are primary eyes, pixels from both sides are collected together. In addition to the pupil itself, the enclosed region contained within the keypoints of the eye may also include eyelashes, sclera (white of the eye), and reflections. These should be removed as much as possible during pixel collection to ensure that the majority of pixels in the final image are derived from the pupil itself.

[0143] To remove eyelash pixels, the key points of the eye are along the y-axis ( Figure 14 (Vertical direction) contracting inward by a certain distance to form Figure 14 The area shown is 1402. This is to remove white eyes and reflections (such as...). Figure 14 (As shown in circle 1404 in the diagram), such pixels are further excluded in region 1402. For example, if the R, G, and B values ​​of a pixel are all greater than a predetermined threshold, the pixel is excluded. Pixels collected in this way ensure that the majority of pixels in the image originate from the pupil itself. Similarly, intermediate colors are used as the average pupil color.

[0144] In some embodiments, for lip color extraction, only pixels in the lower lip region are detected. The upper lip is typically thinner and relatively sensitive to keypoint errors, and because the upper lip is illuminated by colored light, it cannot accurately represent lip color. Therefore, after rotating and correcting the image, all pixels in the region surrounded by the keypoints of the lower lip are collected, and an intermediate color is used to represent the average lip color.

[0145] Figure 15 Exemplary hair color extraction regions used in hair color extraction methods according to some implementations of this disclosure are shown. Hair color extraction is more difficult than extraction in the previous sections. The main reason is that each person's hairstyle is unique, and the background of the photo is complex and diverse. Therefore, it is difficult to locate the pixels of the hair. In one way to accurately locate the hair pixels, a neural network is used to segment the hair pixels of the image. Since the annotation cost of image segmentation is high, and game applications do not require very high-precision color extraction, a keypoint-based approximate prediction method is used.

[0146] To obtain hair pixels, the detection area is first determined. For example... Figure 15 As shown, the detection area 1502 is rectangular. The lower boundary is the corners of the eyebrows on both sides, and the height (vertical line 1504) is the distance 1506 from the upper edge of the eyebrow to the lower edge of the eye. The left and right sides are keypoints 1 and 17 extending fixed distances to the left and right, respectively. The resulting hair pixel detection area 1502... Figure 15 As shown in the image.

[0147] Figure 16Exemplary separation between hair pixels and skin pixels within a hair color extraction region according to some implementations of this disclosure is illustrated. Typically, the detection region contains three types of pixels: skin, hair, and background. In some more complex cases, the detection region also includes a hat. Because the left and right extents of our detection region are relatively conservative, it is assumed in most cases that the included hair pixels far outnumber the background pixels. Therefore, the main process is to divide the pixels of the detection region into hair or skin.

[0148] For each row of pixels in the detection area, skin color changes are typically continuous, such as from light to dark, and there are usually noticeable variations between skin color and hair color. Therefore, the middle pixel of each row is chosen as the starting point 1608, and skin pixels are detected to the left and right. First, a relatively conservative threshold is used to find more reliable skin color pixels, and then the detection expands to the left and right. If the colors of adjacent pixels are similar, then the colors of those adjacent pixels are also labeled as skin colors. This method considers the gradation of skin color and yields relatively accurate results. Figure 16 As shown, within the hair color extraction region 1602, darker areas such as 1604 represent skin color pixels, and brighter areas such as 1606 represent hair color pixels. The median R, G, and B values ​​of the hair color pixels collected within the hair color region are selected as the final average hair color.

[0149] Figure 17 An exemplary eyeshadow color extraction method according to some implementations of this disclosure is shown. The extraction of eyeshadow color differs slightly from that described in the previous section. This is because eyeshadow is a cosmetic that may or may not be present. Therefore, when extracting eyeshadow color, it is necessary to first determine whether eyeshadow is present; if eyeshadow is present, the average color of the eyeshadow is extracted. Similar to eyebrow and pupil color extraction, eyeshadow color extraction is performed only on the main eye portion closest to the camera lens.

[0150] First, it's necessary to determine which pixels belong to the eyeshadow. For the detection area of ​​eyeshadow pixels, such as... Figure 17 As shown, region 1702 is used within lines 1704 and 1706. The left and right sides of region 1702 are defined as the inner and outer corners of the eye, and the upper and lower sides of this region are the lower edge of the eyebrow and the upper edge of the eye. In addition to the possible eyeshadow pixels in region 1702, there may also be eyelashes, eyebrows, and skin, which are required when extracting eyeshadow.

[0151] In some embodiments, to eliminate the influence of eyebrows, the upper edge of the detection area is moved further downwards. To reduce the influence of eyelashes, pixels with brightness below a certain threshold are excluded. To distinguish eyeshadow from skin color, the difference between the hue of each pixel and the average skin color hue is examined. Pixels are collected as potential eyeshadow pixels only if this difference is greater than a certain threshold. The reason for using hue instead of RGB values ​​is that the average skin color is mainly collected below the eyes, while the skin color above the eyes can have greater brightness variations. Since color is not sensitive to brightness, it is relatively stable. Therefore, hue is more suitable for determining whether a pixel is skin.

[0152] The process described above determines whether a pixel in each detection area belongs to eyeshadow. In some embodiments, if no eyeshadow is present, some pixels may still be incorrectly identified as eyeshadow.

[0153] To reduce the aforementioned errors, each column of the detection region is examined. If the number of eyeshadow pixels in the current column exceeds a certain threshold, the current column is marked as an eyeshadow column. If the ratio of the eyeshadow column to the width of the detection region exceeds a certain threshold, eyeshadow is considered to exist in the current image, and the intermediate color of the collected eyeshadow pixels is used as the final color. In this way, a few pixels misclassified as eyeshadow will not lead to an incorrect judgment of the entire eyeshadow.

[0154] Considering art style, most games typically don't allow free color adjustments for all the aforementioned components. For those components where color adjustments are allowed, they are usually only permitted to match a predetermined set of colors. Taking hair as an example, if a hairstyle allows five hair colors, the hairstyle in the resource pack will contain texture images corresponding to each hair color. During detection, the desired hair rendering effect can be obtained by simply selecting the texture image with the closest color based on the hair color prediction result.

[0155] In some embodiments, when only a color texture image is provided, the color of the texture image can be reasonably altered based on any detected color. For ease of color conversion, the commonly used RGB color space representation is converted to the HSV color model. The HSV color model includes three dimensions: hue (H), saturation (S), and lightness (V). Hue (H) is represented in the model as a 360-degree color range, where red is 0 degrees, green is 120 degrees, and blue is 240 degrees. Saturation (S) represents a mixture of spectral colors and white. Higher saturation results in a brighter color. When saturation approaches 0, the color is close to white. Lightness (V) represents the brightness of the color, with values ​​ranging from black to white. After color adjustment, the expected HSV median value of the texture image matches the predicted color. Therefore, the hue value for each pixel can be calculated as follows: H i ′=(H i +H′ H)%1, where Hi 'and H i H represents the hue of pixel i before and after adjustment, and H and H' represent the median hue of the texture image before and after adjustment.

[0156] Unlike hue, which is a continuous space connected end-to-end, saturation and brightness have boundary singularities, such as 0 and 1. If a linear processing method similar to hue adjustment is used, many pixel values ​​will appear too high or too low in saturation or brightness when the median values ​​of the initial or adjusted image are close to 0 or 1. This phenomenon results in unnatural colors. To address this issue, the following non-linear curve is used to fit saturation and brightness before and after pixel adjustment: y = 1 / (1+(1) (1) x ) / ( x )), ∈(0,1) In the above formula, x and y represent the saturation or brightness values ​​before and after adjustment, respectively. The only uncertain parameter is α, which can be derived as follows: α=1 / (1 + x / (1 - x) × (1 - y) / y) This equation guarantees that α falls within the range of 0 to 1. Using saturation as an example, the initial intermediate saturation can be easily calculated based on the input image. S Target saturation value S t This can be obtained through hair color extraction and color space conversion. Therefore, α = 1 / (1 + S / (1 - S ) × (1 - S t ) / S t ) For each pixel S in the default texture image i Then it can be solved by equation S i =1 / (1+(1) α)(1 S i ) / (α S i The adjusted value is calculated using this method. The same calculation is applied to brightness.

[0157] To make the adjusted texture image appear closer to a realistic image, special processing is applied to different parts. For example, to maintain low saturation in the hair, [the settings are as follows]. S' = S' × V' ^0.3. Figure 18 The illustration shows some exemplary color adjustment results according to some implementations of this disclosure. Column 1802 shows some default texture images provided by a specific game, column 1804 shows some texture images adjusted from the corresponding default texture images in the same row based on the actual image shown at the top of column 1804, and column 1806 shows some texture images adjusted from the corresponding default texture images in the same row based on the actual image shown at the top of column 1806.

[0158] Figure 19 This is a flowchart 1900 illustrating an exemplary process for extracting colors from a 2D facial image of a real person according to some implementations of this disclosure.

[0159] The process of extracting color from a 2D facial image of a real person includes step 1910: identifying multiple key points in the 2D facial image based on a key point prediction model.

[0160] The process also includes step 1920: rotating the 2D facial image until multiple target keypoints from the identified multiple keypoints are aligned with the corresponding target keypoints of the standard face.

[0161] The process additionally includes step 1930: locating multiple parts in the rotated 2D facial image, each part being defined by a corresponding subset of the identified multiple key points.

[0162] The process further includes step 1940: extracting the color of each of the multiple parts based on the pixel values ​​of the 2D facial image, wherein each part is defined by a corresponding subset of key points.

[0163] The process additionally includes step 1950: using the extracted colors, generating a 3D model of a real-life person, the 3D model being matched with the corresponding facial feature colors of the 2D facial image.

[0164] Additional implementations may include one or more of the following features.

[0165] In some embodiments, the keypoint prediction model in step 1910 is formed based on machine learning from keypoints manually annotated by the user.

[0166] In some embodiments, the selected key points used for alignment in rotation step 1920 are located on the left and right symmetrical sides of the 2D facial image.

[0167] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include: selecting the median value of the R, G, and B values ​​of all pixels in a corresponding defined area within the corresponding portion as the predicted average color.

[0168] In some embodiments, step 1940, extracting the average color of each of the plurality of portions, may include: determining a region within the skin portion for skin color extraction; and selecting the median value of each of the R, G, and B values ​​of all pixels in the region for skin color extraction as the predicted average color of the skin portion. In some embodiments, the region within the skin portion for skin color extraction is determined to be a region on the face located below the eyes and above the lower edge of the nose.

[0169] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include eyebrow color extraction within an eyebrow portion, which includes: selecting an eyebrow as a target eyebrow corresponding to determining that an eyebrow is located on the side of a viewer that is closer to the 2D facial image; selecting two eyebrows as target eyebrows corresponding to determining that two eyebrows are equally close to a viewer in the 2D facial image; extracting one or more intermediate eyebrow regions within one or more target eyebrows; comparing each pixel value within one or more intermediate eyebrow regions with an average skin color; collecting pixels whose difference between the pixel values ​​of one or more intermediate eyebrow regions and the average skin color exceeds a threshold; and selecting the median value of each of the R, G, and B values ​​of the collected pixels used for eyebrow color extraction as the predicted average color of the eyebrow portion.

[0170] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include pupil color extraction within the eye portion, which includes: selecting an eye as a target eye corresponding to determining that an eye is located on the side of a viewer closer to the 2D facial image; selecting two eyes as target eyes corresponding to determining that two eyes are equally close to a viewer in the 2D facial image; extracting regions without eyelashes within one or more target eyes; comparing each pixel value in the extracted regions with a predetermined threshold; collecting pixels whose extracted region pixel values ​​exceed the predetermined threshold; and selecting the median value of each of the R, G, and B values ​​of the collected pixels for pupil color extraction as the predicted average color of the pupil.

[0171] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include lip color extraction within the lip portion, which includes: collecting all pixels in the region surrounded by keypoints of the lower lip; and selecting the median value of each of the R, G, and B values ​​of the collected pixels for lip color extraction as the predicted average color of the lip portion.

[0172] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include hair color extraction within a hair portion, the hair color extraction within a hair portion comprising: identifying a region including portions of the forehead extending from both sides into the hair portion; determining pixel color variations exceeding a predetermined threshold from the middle of the region to its left and right boundaries; dividing the region into a hair region and a skin region based on the pixel color variations exceeding the predetermined threshold; and selecting the median value of each of the R, G, and B values ​​of pixels in the hair region within the region as the predicted average color of the hair portion.

[0173] In some embodiments, a region including a portion of the forehead extending from both sides into the hairline is identified as a rectangular region, the lower boundary of which is at the two eyebrow corners, the left and right boundaries of which are at fixed distances outward from key points located on the left and right symmetrical sides of the 2D facial image, and the height of the rectangular region is equal to the distance from the upper edge of the eyebrow to the lower edge of the eye.

[0174] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include eyeshadow color extraction within an eyeshadow portion, which includes: selecting an eye as a target eye corresponding to determining that an eye is located on the side of a viewer closer to the 2D facial image; selecting two eyes as target eyes corresponding to determining that two eyes are equally close to a viewer of the 2D facial image; extracting one or more intermediate regions within one or more eyeshadow portions that are close to the target eyes; collecting pixels within the extracted intermediate regions that satisfy the following: brightness is higher than a predetermined brightness threshold to exclude eyelashes, and the difference between the pixel hue value and the average skin tone value exceeds a predetermined threshold; marking a pixel column as an eyeshadow column corresponding to determining that the number of pixels collected in a pixel column within one or more extracted intermediate regions is greater than a threshold; and selecting the median value of each of the R, G, and B values ​​of the collected pixels used for eyeshadow color extraction as the predicted eyeshadow color of the eyeshadow portion corresponding to determining that the ratio of the width of the eyeshadow column to the width of the extracted intermediate regions is greater than a specific threshold.

[0175] In some embodiments, the process of extracting color from a 2D facial image of a real person may additionally include: converting a texture map based on an average color while preserving the original brightness and color difference of the texture map, which includes: converting the average color from an RGB color space representation to an HSV (hue, saturation, brightness) color space representation; and adjusting the color of the texture map to reduce the difference between the median HSV value of the average color and the median HSV value of the texture map.

[0176] The methods and systems disclosed in this paper can be used in various scenarios, such as character modeling and game character generation. This lightweight approach can be flexibly applied to different devices, including mobile devices.

[0177] In some embodiments, the definition of facial key points in the current system and method is not limited to the current definition; other definitions are also possible as long as they adequately represent the contour of each part. Furthermore, in some embodiments, the colors directly returned in the scheme may not be used directly, but may be matched with a predetermined color list to achieve further color filtering and control.

[0178] Deformation methods that optimize the Laplacian operator require the mesh to be a differentiable manifold. However, in practice, meshes created by game artists often contain artifacts such as duplicate vertices and unsealed edges that can damage the properties of the manifold. Therefore, methods such as biharmonic deformation can only be used after careful mesh cleaning. The affine deformation method proposed in this paper does not use the Laplacian operator and therefore does not have such a strong constraint.

[0179] In some cases, a family of deformation methods represented by biharmonic deformation suffers from insufficient deformation capability. Solving the harmonic function of the first harmonic of the Laplacian operator often fails to achieve smooth results due to its low smoothness requirement. Solving the multiharmonic function of higher-order (>=3) Laplacian operators fails on many meshes due to its high requirement (at least 6th-order differentiability). In most cases, it is observed that solving only the biharmonic deformation of the second harmonic of the Laplacian operator yields acceptable results. Even so, biharmonic deformation can still be unsatisfactory due to its lack of adjustable degrees of freedom. The affine deformation proposed in this paper allows for fine deformation adjustment by changing the smoothness parameter, and the range of deformation results obtained by this affine deformation covers the range of deformation results obtained using biharmonic deformation.

[0180] Figure 20 This is a flowchart illustrating an exemplary head avatar deformation and generation process according to some implementations of this disclosure. Using the techniques proposed in this disclosure, the head mesh can be appropriately deformed without being bound to the skeleton. Therefore, the workload required by the artist is greatly reduced. This technique adapts to different styles of meshes for better versatility. In the production of game assets, artists can save head models in various formats using tools such as 3ds Max or Maya, but the internal representation of these formats is a polygonal mesh. Polygonal meshes can be easily converted into pure triangular meshes, which are called stencil models. For each stencil model, 3D key points are manually marked once on the stencil model. Then, based on the 3D key points detected and reconstructed from arbitrary face images, the stencil model can be used to deform into a unique head avatar.

[0181] Figure 21This is a diagram illustrating exemplary components of a head template model according to some implementations of this disclosure. The head template model 2102 typically includes portions such as face 2110, eyes 2104, eyelashes 2106, teeth 2108, and hair, as... Figure 21 As shown. Without rigging the skeleton, mesh deformation depends on the connection structure of the template mesh. Therefore, the template model needs to be decomposed into these semantic parts, and the facial mesh needs to be deformed first. All other parts can be automatically adjusted by setting and following certain key points on the facial mesh. In some embodiments, interactive tools are provided to detect all topologically connected parts, and users can use these interactive tools to easily recall those semantic parts for further deformation.

[0182] In some embodiments, facial keypoints can be obtained through detection algorithms or AI models. These keypoints need to be mapped to vertices on a template model for the purpose of driving mesh deformation. Due to the randomness of mesh connections and the lack of 3D human keypoint labeling data, no tool can automatically and accurately label 3D keypoints on arbitrary head models. Therefore, an interactive tool has been developed that allows for manual and rapid labeling of keypoints on 3D models. Figure 22 This is a diagram illustrating some exemplary key points marked on realistic-style 3D models (e.g., 2202, 2204) and cartoon-style 3D models (e.g., 2206, 2208) according to some implementations of this disclosure.

[0183] During the labeling process, the positions of the 3D keypoints labeled on the 3D model should match the keypoints in the image as closely as possible. Since keypoints are labeled on discrete vertices within the 3D model mesh, some degree of bias is unavoidable. To mitigate this bias, one approach is to define appropriate rules in the pose processing. Figure 23 This diagram illustrates an exemplary comparison between template model rendering, manually labeled keypoints, and AI-detected keypoints according to some implementations of this disclosure. In some embodiments, for those models formed relatively realistically, keypoint detection and reconstruction algorithms may be applied to the rendering of the template model (2302), and the results of 3D keypoints (2306) implemented, for example by artificial intelligence, may be further compared with the results of manually labeled keypoints (2304), and thus the deviation between the two sets of keypoints is calculated. When human images are detected, the calculated deviation is reduced from the keypoints detected in the real-life image, and the adverse effects of manual labeling are eliminated.

[0184] The affine deformation method disclosed in this paper is based on keypoint-driven mathematical modeling, which ultimately solves a system of linear equations. The method employs a single step that uses detected keypoints as boundary conditions to deform the template mesh, and incorporates different constraints during the optimization process. Figure 24 This is a diagram illustrating an exemplary affine transformation of a triangle according to some implementations of this disclosure.

[0185] In some embodiments, the deformation from the template mesh to the predicted mesh is considered as a combination of affine transformations of each triangle. The affine transformation of a triangle can be defined as... 3×3 Matrix T and translation vector d. For example... Figure 24 As shown, the position of the deformed vertex after the affine transformation is denoted as... v i ′=Tv i +d, i∈1...4 ,in v 1, v 2, v 3 represents each vertex of the triangle, and v4 is an additional point introduced along the normal direction of the triangle, satisfying the equation v 4 =v 1 +( v 2 v 1 )×(v 3 v 1 ) / sqrt(∣ ( v 2 v 1 )×(v 3 v 1 )∣) In the above equation, the result of the cross product is normalized so that it is proportional to the length of the triangle's edge. v4 is introduced because the coordinates of the three vertices are insufficient to determine a unique affine transformation. After introducing v4, we obtain the derived equation: T = [v' 2 - v' 1 v' 3 -v' 1 v' 4 - v' 1 ]×[v 2 - v 1 v 3 -v 1 v 4 - v 1 ] -1 And determine the non-translated part of matrix T. Because matrix V = [v 2 - v 1 v 3 -v 1 v 4 - v 1 ] -1The matrix depends only on the invariance of the template mesh and other deformation factors, so it can be pre-computed as a sparse coefficient matrix for later construction of a linear system.

[0186] To date, the non-translational part of the affine transformation T has been expressed mathematically. To construct an optimized linear system, assuming the number of mesh vertices is N and the number of triangles is F, consider the following four constraints: Constraints on key point locations: E k =Σ i=1 ||v' i - c' i || 2 ,c' i This represents the location of key points detected after mesh deformation.

[0187] Constraints on adjacency smoothness: E s =Σ i=1 Σ j∈adj(i) ||T i - T j || 2 This means that the affine transformations between adjacent triangles should be as similar as possible. Adjacency relationships can be queried and stored in advance to avoid redundant calculations and improve the performance of the system.

[0188] Constraints on characteristics: E i =Σ i=1 ||T i - I|| 2 , where I represents the identity matrix. This constraint means that the affine transformation should be as close to invariant as possible, which helps to preserve the properties of the template mesh.

[0189] Constraints of the original position: E l =Σ i=1 N||v' I - c i || 2 , where c i This indicates the position of each vertex on the template mesh before deformation.

[0190] The final constraint is the weighted sum of the above constraints: minE = w k E k + w s E s + w i E i + w l E l where the weight w k w s w i w lSort from strongest to weakest. Using the above constraints, a linear system of size (F + N) × (F + N) can be constructed, with the weights multiplied by their corresponding coefficients in the system. This is in addition to the extra points for each triangle. v' Beyond 4, the unknowns are the coordinates of each vertex after the deformation. Since the previous term is useful, it will be discarded. v' Results in section 4. During continuous deformation, all constraint matrices except those at keypoint locations can be reused. For meshes with thousands of vertices, affine transformations can achieve real-time performance of 30 fps on ordinary PCs and smartphones.

[0191] Figure 25 This is a diagram illustrating an exemplary comparison of deformation results of some head models with and without a hybrid deformation process according to some implementations of this disclosure.

[0192] In some embodiments, when deforming the head model of a game avatar, the region of interest is typically only the face. The top, back, and neck of the head should remain unchanged, otherwise mesh penetration between the head and hair or torso can occur. To avoid this problem, the stencil mesh and the results of affine deformation are linearly interpolated in a blended deformation manner. The weights used for blending can be drawn in 3D modeling software or calculated using biharmonic or affine deformation with minimal variation. For example, the weights on keypoints are set to 1s, while more markers are added to the head model ( Figure 25 (The dark spots in 2504), and the weights of these markers are set to 0. In some embodiments, inequality constraints are added during the solution process to force all weights to fall within the range of 0 to 1, but doing so will greatly increase the complexity of the solution. Experiments have shown that good results can be obtained by truncating weights that are less than 0 or greater than 1. Figure 25 As shown in 2504, the weight of the model portion with the darkest color is 1s, and the weight of the colorless model portion is 0s. In the curved weight rendering 2504, there is a natural transition between the bright keypoints and the dark markers. In the case of blending deformation, the back side of the model after deformation (such as...) Figure 25 (As shown in 2506) remains consistent with the original dorsal side (as shown in 2506) Figure 25 The same as shown in 2502). In the absence of mixed deformation, the back side of the model after deformation (as shown in 2502) Figure 25 (As shown in 2508) will not remain with the original dorsal side (as shown in 2508) Figure 25 (As shown in 2502).

[0193] In some embodiments, affine deformation can achieve different deformation effects by manipulating the weights of constraints, including the results of simulating double harmonic deformation. Figure 26This is a diagram illustrating an exemplary comparison of affine and biharmonic deformations with different weights according to some implementations of this disclosure. Figure 26 As shown, smoothness is the adjacent smoothness weight w s and feature weight w i The ratio is shown. Dark points are keypoints, and the darkness of the color represents the displacement between the deformed position and the original position of the vertex. In all deformation results, one keypoint remains unchanged, while another keypoint moves to the same position. This indicates that as the adjacent smoothness weight is gradually increased relative to the feature weight, the smoothness of the deformed sphere also increases accordingly. Furthermore, the results of biharmonic deformation match those of affine deformation, where the smoothness falls somewhere between 10 and 100. This indicates that affine deformation has a greater degree of freedom of deformation compared to biharmonic deformation.

[0194] Using the workflow described in this article, games can easily integrate intelligent head and avatar generation functionality. For example, Figure 27 This illustrates some implementations of the present disclosure using a realistic template model derived from a set of randomly selected images of women. Figure 27 (Not shown) Some exemplary results generated automatically. All personalized avatars reflect some characteristics of their corresponding images.

[0195] Figure 28 This is a flowchart 2800 illustrating an exemplary process for generating a 3D deformable head model from a 2D facial image of a real person according to some implementations of this disclosure.

[0196] The process of generating a 3D head deformable model from a 2D facial image includes step 2810: receiving a two-dimensional (2D) facial image, such as a convolutional neural network.

[0197] The process also includes step 2820: identifying the first set of key points in the 2D facial image based on an artificial intelligence (AI) model.

[0198] The process additionally includes step 2830: mapping a first set of key points to a second set of key points based on a set of user-provided key point annotations located on a 3D head template model, the second set of key points being located on multiple vertices of the mesh of the 3D head template model.

[0199] The process further includes step 2840: deforming the mesh of the 3D head template model by reducing the difference between the first set of keypoints and the second set of keypoints to obtain a deformed 3D head mesh model. In some embodiments, there is a correspondence between the keypoints in the first set and the keypoints in the second set. After projecting the second set of keypoints into the same space as the first set of keypoints, a function is generated to measure the positional difference between each keypoint in the first set of keypoints and the second set of keypoints. By deforming the mesh of the 3D head template model, the second set of keypoints in space is optimized when the function measuring the positional difference (e.g., position, adjacency smoothness, properties, etc.) between each keypoint in the first set of keypoints and the second set of keypoints is minimized.

[0200] The process additionally includes step 2850: applying a hybrid deformation method to the deformed 3D head mesh model to obtain a personalized head model based on the 2D facial image.

[0201] Additional implementations may include one or more of the following features.

[0202] In some embodiments, the mapping step 2830 may further include: associating a first set of key points on a 2D facial image with a plurality of vertices on a mesh of a 3D head template model; identifying a second set of key points based on a set of user-provided key point annotations located on a plurality of vertices of the mesh of the 3D head template model; and mapping the first set of key points and the second set of key points based on corresponding features identified by the corresponding key points on the face.

[0203] In some embodiments, a second set of keypoints is located by applying a previously calculated bias to this set of user-provided keypoint annotations. In some embodiments, the previously calculated bias is between a previous set of AI-identified keypoints in the 3D head template model and a previous set of user-provided keypoint annotations located at multiple vertices of the mesh of the 3D head template model.

[0204] In some embodiments, step 2840 of performing deformation may include: transforming the mesh of the 3D head template model into a deformed 3D head mesh model by using a mapping from a first set of key points to a second set of key points and by using boundary conditions of the deformation associated with the first set of key points.

[0205] In some embodiments, step 2840 of performing deformation may further include applying different constraints during deformation optimization, which includes one or more of key point locations, adjacency smoothness, characteristics, and original locations.

[0206] In some embodiments, step 2840 of performing deformation may further include applying constraints to the deformation process, the constraints being a weighted sum of one or more of key point locations, adjacency smoothness, properties, and original locations.

[0207] In some embodiments, step 2820 of identifying the first set of key points includes using a convolutional neural network (CNN).

[0208] In some embodiments, the deformation includes affine deformation without a Laplacian operator. In some embodiments, affine deformation is achieved by changing a smoothness parameter.

[0209] In some embodiments, the mesh of the 3D head template model can be deformed without being bound to a skeleton. In some embodiments, the facial deformation model includes a realistic style model or a cartoon style model.

[0210] In some embodiments, in step 2850, applying a hybrid deformation method to the deformed 3D head mesh model includes: specifying corresponding hybrid weights on the key points of the deformed 3D head mesh model according to the positions of the key points; and applying different degrees of deformation to key points with different hybrid weights.

[0211] In some embodiments, in step 2850, applying a hybrid deformation method to the deformed 3D head mesh model includes: keeping the back face of the deformed 3D head mesh model in the same shape as the original back face shape of the 3D head template model before deformation.

[0212] In some embodiments, the semantic elements on the template model are not limited to eyes, eyelashes, or teeth. Decorations such as glasses may be adaptively adjusted by adding and tracking new key points on the facial mesh.

[0213] In some embodiments, key points are manually added to the template model. In other embodiments, deep learning techniques can be used to automatically add key points to different template models.

[0214] In some embodiments, the solution process for affine deformation can utilize numerical techniques to further improve its computational performance.

[0215] In some embodiments, the systems and methods disclosed herein form a facial avatar generation system based on lightly weighted keypoints, which has many advantages, such as those listed below: The requirements for the input image are low. The system and method do not require the face to be directly facing the camera, and a certain degree of in-plane rotation, out-of-plane rotation, and occlusion will not significantly affect performance.

[0216] Suitable for both realistic and cartoon games. This system does not limit the game style to realistic games; it can also be applied to cartoon-style games.

[0217] Lightweight and customizable. Each module of this system is relatively lightweight and suitable for mobile devices. The modules within the system are decoupled, allowing users to use different combinations to build the final face generation system according to different game styles.

[0218] In some embodiments, for a given single photograph, the main face is first detected, and keypoint detection is performed. In real-world images, the face may not be facing the camera, and real-world faces are not always perfectly symmetrical. Therefore, the keypoints in the original image are preprocessed to achieve a consistent, symmetrical, and smooth set of keypoints. Then, the keypoints are adjusted according to the specific style of the game (e.g., enlarged eyes, slimmer face). After obtaining the stylized keypoints, the stylized keypoints are converted into control parameters for the facial model in the game, typically skeletal parameters or slider parameters.

[0219] In some embodiments, the viewpoint of the real face may not be directly facing the camera, which may lead to problems such as left-right asymmetry and key point detection errors. Figure 29 This is a diagram illustrating exemplary keypoint processing steps according to some implementations of this disclosure. Keypoints detected from the original image 2904 may not be used directly and require some processing. Here, the process is divided into three steps: normalization, symmetry, and smoothing, as follows... Figure 29 As shown.

[0220] In some embodiments, it is necessary to adjust the standard facial model in the game based on the predicted real facial keypoints. This process requires ensuring that the keypoints of the standard facial model in the game and the real face are aligned in size, position, and orientation. Therefore, the normalization 2906 of the predicted keypoints and keypoints on the game facial model includes the following components: size normalization, translation normalization, and angle normalization.

[0221] In some embodiments, all originally detected 3D facial key points are defined as p, where the i-th key point is p. i ={x i , y i , z i For example, the normalization origin is defined as the midpoint between the 1st and 17th keypoints (see [reference]). Figure 1 The definition of key points in the text is as follows: c = (p1 + p) 17 ) / 2. For the dimensions, the distance between the 1st and 17th keypoints and the origin is adjusted to 1, so that the 3D keypoints normalized by dimensions and translation are p'=(pc) / ||p1- c||.

[0222] In some embodiments, after normalizing the size and translation, the facial orientation is also normalized. For example... Figure 29 As shown in image 2902, the face in the actual photograph may not be directly facing the camera, and there will always be a specific deflection that can exist on the three coordinate axes. The predicted 3D keypoints of the face are rotated sequentially along the x, y, and z coordinate axes so that the face faces the camera. When rotating along the x-axis, the z-coordinates of keypoints 18 and 24 are (reference) Figure 1 (The definition of key points in the text) Alignment, that is, making the depth of the uppermost part of the bridge of the nose the same as the depth of the bottom of the nose, to obtain the rotation matrix R. X When rotating along the y-axis, the z-coordinates of keypoints 1 and 17 are aligned to obtain the rotation matrix R. Y When rotating along the z-axis, the y-coordinates of keypoints 1 and 17 are aligned to obtain the rotation matrix R. Z Therefore, the keypoints are oriented and the normalized keypoints are as follows: P 归一化 =R Z ×R Y ×R X ×P' In some embodiments, the size, position, and angle of normalized keypoints are adjusted to be uniform, but the resulting keypoints are often not perfect facial features. For example, the bridge of the nose is not straight at the center, and facial features may be asymmetrical. This is because: due to expressions or their inherent characteristics, real faces in photographs are not perfectly symmetrical, and this introduces additional errors when predicting keypoints. Although real faces may be asymmetrical, if the facial model in the game is asymmetrical, it will result in an unattractive appearance and significantly reduce the user experience. Therefore, keypoint symmetry, as shown in Figure 2908, is a necessary process.

[0223] Because the keypoints are already normalized, in some embodiments, a simple symmetry approach is to average the y and z coordinates of all left-right symmetrical keypoints instead of the original y and z coordinates. This method works well in most cases, but performance is sacrificed when the face is rotated at a large angle along the y-axis.

[0224] In some embodiments, use Figure 29Using a human face as an example, when the face is turned a large angle to the left, part of the eyebrows will be invisible. Simultaneously, due to perspective, the left eye will appear smaller than the right eye. While 3D keypoints can partially compensate for the effects of perspective, the 2D projections of the corresponding 3D keypoints still need to be preserved on the image. Therefore, excessive angular deflection will lead to a significant difference in the size of the eyes and eyebrows in the 3D keypoint detection results. To address the effects of angle, when the facial deflection angle along the y-axis is large, the eye and eyebrow closest to the camera are designated as the primary eye and eyebrow, and these are copied to the other side to reduce errors caused by angular deflection.

[0225] In some embodiments, due to unavoidable prediction errors of keypoints, symmetrical keypoints may still not match the real face in some individual cases. Because the shapes of the real face and facial features differ greatly, it is difficult to achieve a relatively accurate description using predetermined parametric curves. Therefore, when smoothing as shown in 2910, only some regions (e.g., facial contours, eyes, eyebrows, lower lip, etc.) are smoothed. These regions remain essentially monotonous and smooth, i.e., there are no uneven areas. In this case, the target curve should always be either a convex or concave curve.

[0226] In some embodiments, for the relevant boundaries, key points are checked one by one to see if they satisfy the definition of a convex curve (or a concave curve). Figure 30 This is a diagram illustrating an exemplary keypoint smoothing process 2910 according to some implementations of this disclosure. (See diagram for example.) Figure 30 As shown, without loss of generality, the target curve should be a convex curve. For each keypoint 3002, 3004, 3006, 3008, and 3010, check if its position is higher than the line connecting its left and right adjacent keypoints. If this condition is met, it means that the current keypoint satisfies the convex curve requirement. Otherwise, move the current keypoint upwards to the line connecting the left and right keypoints. For example, in Figure 30 In this process, if keypoint 3006 does not meet the constraints of a convex curve, keypoint 3006 will be moved to position 3012. If multiple keypoints are moved, the curve cannot be guaranteed to be convex or concave after the movement. Therefore, in some embodiments, multi-round smoothing is used to obtain a relatively smooth keypoint curve.

[0227] Different games have different facial styles. In some embodiments, key features of realistic faces need to be translated into the style required by the game. Realistic game faces are similar, but cartoon faces are quite different. Therefore, it is difficult to implement a uniform standard for the stylization of key features. In practice, the definition of stylization comes from the game designers, who adjust facial characteristics according to the specific game style.

[0228] In some embodiments, more general facial adjustment schemes that most games might require are implemented. These include adjustments to face length, width, and facial features. Custom corrections can be made based on different game art styles, adjustment levels, scaling ratios, etc. Users can also customize any special style adjustment method, such as changing the eye shape to a rectangle. The system supports any adjustment method.

[0229] In some embodiments, key points of a stylized face are used to deform a standard game face so that the key points of the deformed face reach the position of a target key point. Since most games use control parameters such as bones or sliders to adjust the face, a set of control parameters is needed to move the key points to the target position.

[0230] Since the definition of bones or sliders can vary across different games and is subject to modification at any time, directly defining a simple parameterized function from keypoints to bone parameters is no longer feasible. In some embodiments, machine learning methods are used to transform keypoints into parameters via a neural network, known as a K2P (keypoint-to-parameter) network. Because the number of common parameters and keypoints is relatively small (typically less than 100), K-layer fully connected networks are used in some embodiments.

[0231] Figure 31This is a block diagram illustrating an exemplary keypoint-to-control-parameter (K2P) conversion process according to some implementations of this disclosure. To utilize machine learning methods, in some embodiments, bone or slider parameters are first randomly sampled, the sampled bone or slider parameters are fed to a game client 3110, and keypoints are extracted from the generated game face. In this way, a large amount of training data (parameter 3112 and keypoint 3114 pairs) is obtained. Then, a self-supervised machine learning method is implemented, which consists of two steps: the first step is to train a P2K (parameter-to-keypoint) network 3116 to simulate the process of generating game parameter-to-keypoints. In the second step, a large number of unlabeled real face images 3102 are used to generate real facial keypoints 3104 according to the method described herein, and then a large number of stylized keypoints 3106 are generated. These unlabeled stylized keypoints 3106 are self-supervised learning training data. In some embodiments, a set of keypoints K is input into a K2P network 3108 to learn and obtain output parameters P. Since ground truth values ​​for the ideal parameters corresponding to these keypoints are unavailable, P is further fed into the P2K network 3116 trained in the first step to obtain keypoints K'. In some embodiments, a K2P network 3108 can be learned by calculating the mean squared error (MSE) loss between K and K'. In some embodiments, the P2K network 3116 is fixed during the second step and will not be further adjusted. With the aid of the P2K network 3116, a neural network is used to simulate the process of controlling the parameters of the game client 3110 to keypoints, thus laying the foundation for the learning of the K2P network 3108 in the second step. In this way, the final face generated by the parameters maintains close proximity to the keypoints of the generated target stylized face.

[0232] In some embodiments, when calculating the MSE loss between K and K', weights are simultaneously added to certain keypoints (e.g., keypoints in the eyes) by adjusting the corresponding weights. Since the definition of keypoints is predetermined and not affected by the game client's skeleton or slider, the weights are easier to adjust.

[0233] In some embodiments, to improve model accuracy in practical applications, neural networks can be trained separately for parts that can be decoupled. For example, if some skeletal parameters only affect keypoints in the eye region, while other parameters have no effect on that region, these parameters and these keypoints form a set of independent regions. For each such set of regions, a separate K2P model 3108 is trained, and each model can employ a lighter network design. This not only further improves model accuracy but also reduces computational complexity.

[0234] Figure 32Some exemplary results of automatic face generation in mobile games according to some implementations of this disclosure are shown. For example... Figure 32 The diagram illustrates the results of generating game facial avatar images (3204 and 3208) from the original facial images (3202 and 3206). In some embodiments, the open mouth is closed during stylization, and different levels of restrictions and cartoonization are applied to the nose, mouth, facial shape, eyes, and eyebrows. The final generated result still retains certain facial features and meets the aesthetic requirements of the game style.

[0235] Figure 33 This is a flowchart 3300 illustrating an exemplary process of customizing a standard face for an avatar in a game using a 2D facial image of a real-life person, according to some implementations of this disclosure.

[0236] The process of using 2D facial images of real-life people to customize the standard face of avatars in the game includes step 3310: identifying a set of real-life key points in the 2D facial image.

[0237] The process also includes step 3320: converting this set of real-life key points into a set of game-style key points associated with the avatar in the game.

[0238] The process further includes step 3330: by applying a keypoint-to-parameter (K2P) neural network model to this set of avatar keypoints, a set of facial control parameters for the standard face of the avatar in the game is generated, where each facial control parameter is associated with one of multiple facial features of the standard face. (As described above...) Figure 31 The K2P network 3108 described is a deep learning neural network model that predicts a set of facial control parameters based on a set of input avatar keypoints. Because different sets of avatar keypoints can correspond to different sets of facial control parameters, when a set of facial control parameters is applied to the standard face of an avatar, the keypoints of the adjusted standard face can have a set of keypoints similar to this set of input avatar keypoints.

[0239] The process additionally includes step 3340: adjusting multiple facial features of the standard face by applying this set of facial control parameters to the standard face.

[0240] Additional implementations may include one or more of the following features.

[0241] In some embodiments, in step 3330, the K2P neural network model is trained as follows: multiple training 2D facial images of real-life people are obtained; for each of the multiple training 2D facial images, a set of training game style or avatar keypoints is generated; each set of training game style keypoints is submitted to the K2P neural network model to obtain a set of control parameters; this set of control parameters is submitted to a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a set of predicted game style keypoints corresponding to this set of training game style keypoints; the K2P neural network model is updated by reducing the difference between this set of training game style keypoints and the corresponding set of predicted game style keypoints. (As described above...) Figure 31 As described, in contrast to the K2P network 3108, the P2K network 3116 is a deep learning neural network model that predicts a set of avatar keypoints based on a set of input facial control parameters. Since different sets of facial control parameters can cause different sets of avatar keypoints, when the two neural network models are considered to be performing the inverse process of each other, a set of output avatar keypoints associated with the P2K network 3116 should match a set of input avatar keypoints associated with the K2P network 3108.

[0242] In some embodiments, the pre-trained P2K neural network model is configured to: receive a set of control parameters, including skeletal or slider parameters associated with an avatar in the game; and predict a set of game style keypoints for the avatar in the game based on the set of control parameters.

[0243] In some embodiments, the difference between a set of trained game style keypoints and a corresponding set of predicted game style keypoints is the sum of the mean squared errors between the set of trained game style keypoints and the corresponding set of predicted game style keypoints.

[0244] In some embodiments, the trained K2P neural network model and the pre-trained P2K neural network model are game-specific.

[0245] In some embodiments, a set of real-life key points in a 2D facial image corresponds to the facial features of a real-life person in the 2D facial image.

[0246] In some embodiments, the standard face of the avatar in the game can be customized based on facial images of different real-life people to represent different characters in the game.

[0247] In some embodiments, the distorted face of the avatar is a cartoon-style face of a real person. In some embodiments, the distorted face of the avatar is a realistic-style face of a real person.

[0248] In some embodiments, in step 3320, converting this set of real-life key points into a set of game-style key points includes: normalizing this set of real-life key points into a regular space; making the normalized set of real-life key points symmetrical; and adjusting the symmetrical set of real-life key points according to a predetermined style associated with the avatar in the game.

[0249] In some embodiments, normalizing this set of real-life key points into a regular space includes: scaling the set of real-life key points into a regular space; and rotating the scaled set of real-life key points according to the orientation of the set of real-life key points in the 2D facial image.

[0250] In some embodiments, converting this set of real-life key points into a set of game-style key points further includes: smoothing a set of symmetrical key points to meet predetermined convex or concave curve requirements.

[0251] In some embodiments, adjusting a set of real-life key points for symmetry, based on a predetermined style associated with the avatar in the game, includes one or more of the following: face length adjustment, face width adjustment, facial feature adjustment, scaling adjustment, and eye shape adjustment.

[0252] The system and method disclosed in this paper can be applied to automatic face generation systems for various games (both realistic and cartoon-style games). The system features an easily integrated interface, improving the user experience.

[0253] In some embodiments, the systems and methods disclosed herein can be used in 3D facial avatar generation systems for various games, automating complex manual adjustment processes to improve user experience. Users can take selfies or upload existing photos. The system can extract facial features from the photos and then automatically generate control parameters for the game face (e.g., skeletons or sliders) using an AI facial generation system. The game client uses these parameters to generate a facial avatar, ensuring the created face reflects the user's facial features.

[0254] In some embodiments, the system can be easily customized for different games, including keypoint definitions, stylization methods, skeleton / slider definitions, and so on. Users can choose to adjust only certain parameters, automatically retrain the model, or add custom control algorithms. In this way, the invention can be easily deployed to different games.

[0255] Further embodiments also include the combination or other rearrangement of various subsets of the above embodiments in various other embodiments.

[0256] The image processing apparatus of the embodiments of this application is implemented with reference to the accompanying drawings. The image processing apparatus can be implemented in various forms, such as different types of computer devices, such as servers or terminals (e.g., desktop computers, laptop computers, or smartphones). The hardware structure of the image processing apparatus of the embodiments of this application is further described below. It can be understood that... Figure 34 Only an exemplary structure of the image processing apparatus is shown, not all structures, and it can be implemented as needed. Figure 34 The partial or complete structure shown.

[0257] refer to Figure 34 , Figure 34 This is a schematic diagram of an optional hardware structure of an image processing apparatus according to an embodiment of this application. In practical applications, it can be applied to servers or various terminals running applications. Figure 34 The image processing apparatus 3400 shown includes at least one processor 3401, a memory 3402, a user interface 3403, and at least one network interface 3404. The components in the image processing apparatus 3400 are coupled together via a bus system 3405. It will be understood that the bus 3405 is configured to enable connection and communication between the components. In addition to a data bus, the bus system 3405 may also include a power bus, a control bus, and a status signal bus. However, for the purpose of clarity, in... Figure 34 In this context, all buses are labeled as Bus System 3405.

[0258] User interface 3403 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, touch screen, etc.

[0259] It is understood that memory 3402 may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0260] In embodiments of this application, memory 3402 is configured to store different types of data to support the operation of image processing apparatus 3400. Examples of data include any computer program for performing operations on image processing apparatus 3400, such as executable program 34021 and operating system 34022. A program for performing the image processing methods of embodiments of this application may be included in executable program 34021.

[0261] The image processing methods disclosed in the embodiments of this application can be applied to, or executed by, processor 3401. Processor 3401 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image processing method can be accomplished using integrated logic circuits in the hardware of processor 3401 or instructions in software form. The processor 3401 can be a general-purpose processor, a digital signal processor (DSP), another programmable logic device, discrete gates, transistor logic devices, discrete hardware components, etc. Processor 3401 can implement or execute the methods, steps, and logic block diagrams provided in the embodiments of this application. A general-purpose processor can be a microprocessor, any conventional processor, etc. The steps in the methods provided in the embodiments of this application can be directly executed by a hardware decoding processor, or executed by combining hardware and software modules in the decoding processor. The software modules can be located in a storage medium. The storage medium is located in memory 3402. Processor 3401 reads information from memory 3402 and executes the steps of the image processing methods provided in the embodiments of this application by combining this information with its hardware.

[0262] In some embodiments, image processing and 3D face and head formation can be performed on a cloud via a set of servers or a network.

[0263] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted over a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the implementations described in this application. A computer program product may include a computer-readable medium.

[0264] The terminology used in the description of the implementations herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or,” as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. It should be further understood that when the terms “comprising” and / or “including” are used in this specification, these terms indicate the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0265] It should also be understood that while the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the implementation, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.

[0266] The description in this application is for illustrative purposes and is not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications, variations, and alternative implementations will be apparent to those skilled in the art from the teachings given in the foregoing description and the associated drawings. Embodiments were chosen and described to best explain the principles of the invention, its practical application, and to enable those skilled in the art to understand the invention from various implementations, and preferably to utilize the underlying principles, various implementations, and various modifications suitable for the intended particular use. Therefore, it should be understood that the scope of the claims is not limited to the specific examples of the disclosed implementations, and modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. A method for extracting color from a 2D facial image of an object, comprising: Based on the key point prediction model, multiple key points in the 2D facial image are identified. Rotate the 2D facial image until multiple target key points from the identified multiple key points are aligned with the corresponding target key points of the standard face; In the rotated 2D facial image, multiple parts are located, each part being defined by a corresponding subset of multiple identified key points; The color of each of the plurality of parts is extracted based on the pixel values ​​of the 2D facial image; as well as Using the extracted colors, a three-dimensional 3D model of the object is generated, which is matched with the corresponding facial feature colors of the 2D facial image; The process of converting the texture map of the 2D facial image based on colors extracted from the pixel values ​​of the 2D facial image while preserving the original brightness and color difference of the texture map includes: The extracted colors are converted from RGB color space representation to HSV color space representation; as well as Adjust the color of the texture map to reduce the difference between the median HSV value of the extracted color and the median HSV value of the texture map.

2. The method according to claim 1, wherein, The keypoint prediction model is formed based on machine learning from keypoints manually annotated by the user.

3. The method according to claim 1, wherein, The plurality of target key points used for alignment are located on the left and right symmetrical sides of the 2D facial image.

4. The method according to claim 1, wherein, Extracting the color of each of the plurality of parts includes: selecting the median value of the R, G, and B values ​​of all pixels in the corresponding defined region within the corresponding part as the predicted average color.

5. The method according to claim 1, wherein, Extracting the color of each of the plurality of parts includes: determining a region within the skin part for skin color extraction; and selecting the median value of the R, G, and B values ​​of all pixels in the region for skin color extraction as the predicted average color of the skin part.

6. The method according to claim 5, wherein, The area within the skin portion used for skin color extraction is defined as the area on the face located below the eyes and above the lower edge of the nose.

7. The method according to claim 1, wherein, Extracting the color of each of the plurality of parts includes extracting the eyebrow color within the eyebrow part, wherein the eyebrow color extraction within the eyebrow part includes: Corresponding to the determination that an eyebrow is located on the side of the viewer that is closer to the 2D facial image, the eyebrow is selected as the target eyebrow; For viewers who identify two eyebrows that are equally close to the 2D facial image, the two eyebrows are selected as target eyebrows; Extract one or more middle eyebrow regions within one or more of the target eyebrows; Compare the pixel value of each pixel within the one or more middle eyebrow regions with the average skin color; Collect pixels whose pixel values ​​within one or more intermediate eyebrow regions differ from the average skin color by a threshold; and The median value of each of the R, G, and B values ​​of the collected pixels used for eyebrow color extraction is selected as the predicted average color of the eyebrow portion.

8. The method according to claim 1, wherein, Extracting the color of each of the plurality of portions includes extracting the pupil color within the eye portion, wherein the pupil color extraction within the eye portion includes: Corresponding to the determination that an eye is located on the side of the viewer that is closer to the 2D facial image, that eye is selected as the target eye; For viewers whose two eyes are determined to be equally close to the 2D facial image, the two eyes are selected as target eyes; Extract one or more areas within the target eye that lack eyelashes; The value of each pixel within the extracted region is compared with a predetermined threshold. Collect pixels within the extracted region whose pixel values ​​exceed the predetermined threshold; and The median value of each of the R, G, and B values ​​of the collected pixels used for pupil color extraction is selected as the predicted average color of the pupil.

9. The method according to claim 1, wherein, Extracting the color of each of the plurality of portions includes lip color extraction within the lip portion, wherein the lip color extraction within the lip portion includes: Collect all pixels in the area surrounded by the keypoints of the lower lip; and The median value of each of the R, G, and B values ​​of the collected pixels used for lip color extraction is selected as the predicted average color of the lip portion.

10. The method according to claim 1, wherein, Extracting the color of each of the plurality of portions includes hair color extraction within the hair portion, wherein the hair color extraction within the hair portion includes: Identify the area including the portion of the forehead that extends from both sides into the hairline; From the center of the region to its left and right boundaries, determine pixel color changes that exceed a predetermined threshold; Based on the pixel color change exceeding a predetermined threshold, the region is divided into a hair region and a skin region; and The median value of the R, G, and B values ​​of the pixels in the hair region within the region is selected as the predicted average color of the hair portion.

11. The method according to claim 10, wherein, The region, including the portion of the forehead extending from both sides into the hairline, is identified as a rectangular region. The lower boundary of the rectangular region is at the two eyebrow corners, the left and right boundaries are at fixed distances outward from key points located on the left and right symmetrical sides of the 2D facial image, and the height of the rectangular region is equal to the distance from the upper edge of the eyebrow to the lower edge of the eye.

12. The method according to claim 1, wherein, Extracting the color of each of the plurality of portions includes extracting the eyeshadow color within the eyeshadow portion, wherein the eyeshadow color extraction within the eyeshadow portion includes: Corresponding to the determination that an eye is located on the side of the viewer that is closer to the 2D facial image, that eye is selected as the target eye; For viewers whose two eyes are determined to be equally close to the 2D facial image, the two eyes are selected as target eyes; Extract one or more intermediate regions within one or more of the eyeshadow portions that are close to the target eye; Collect pixels within one or more extracted intermediate regions that meet the following criteria: brightness is higher than a predetermined brightness threshold to exclude eyelashes, and the difference between the pixel hue value and the average skin color hue value exceeds a predetermined threshold. When the number of pixels collected in a pixel column within one or more extracted intermediate regions exceeds a threshold, the pixel column is marked as an eyeshadow column; and If the ratio of the width of the eyeshadow column to the width of one or more extracted intermediate regions is greater than a specific threshold, the median value of each of the R, G, and B values ​​of the collected pixels used for eyeshadow color extraction is selected as the predicted eyeshadow color of the eyeshadow portion.

13. An electronic device comprising one or more processing units, a memory coupled to the one or more processing units, and a plurality of programs stored in the memory, wherein, when executed by the one or more processing units, the plurality of programs cause the electronic device to perform a method for extracting color from a two-dimensional 2D facial image of an object according to any one of claims 1 to 12.

14. An apparatus for extracting color from a two-dimensional 2D facial image of an object, the apparatus comprising: Processing unit, the processing unit being configured to: Based on the key point prediction model, multiple key points in the 2D facial image are identified. Rotate the 2D facial image until multiple target key points from the identified multiple key points are aligned with the corresponding target key points of the standard face; In the rotated 2D facial image, multiple parts are located, each part being defined by a corresponding subset of multiple identified key points; The color of each of the plurality of parts is extracted based on the pixel values ​​of the 2D facial image; as well as Using the extracted colors, a three-dimensional 3D model of the object is generated, which is matched with the corresponding facial feature colors of the 2D facial image; The process of converting the texture map of the 2D facial image based on colors extracted from the pixel values ​​of the 2D facial image while preserving the original brightness and color difference of the texture map includes: The extracted colors are converted from RGB color space representation to HSV color space representation; as well as Adjust the color of the texture map to reduce the difference between the median HSV value of the extracted color and the median HSV value of the texture map.

15. A non-transitory computer-readable storage medium storing a plurality of programs, said plurality of programs being executed by an electronic device, said electronic device having one or more processing units, wherein, When the plurality of programs are executed by the one or more processing units, the plurality of programs cause the electronic device to perform a method for extracting color from a two-dimensional 2D facial image of an object according to any one of claims 1 to 12.

Images