Method of customizing standard face of virtual character, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a method, an electronic device, a storage medium, and a program product for customizing a standard face of a virtual character using a two-dimensional (2D) face image of a subject (e.g., a real person). The method includes: identifying a set of subject keypoints in the 2D face image; transforming the set of subject keypoints into a set of virtual character keypoints associated with the virtual character; generating a set of face control parameters for the standard face by applying a keypoints-to-parameters (K2P) neural network model to the set of virtual character keypoints, each parameter in the set of face control parameters being respectively associated with one facial feature of a plurality of facial features of the standard face; and adjusting the plurality of facial features of the standard face by applying the set of face control parameters to the standard face, wherein the adjusted standard face of the virtual character has facial features of the 2D face image of the subject.

Description

[0001] Cross-references to related applications

[0002] This application is a continuation-into-priority application of U.S. Patent Application No. 17 / 202,121, filed March 15, 2021, entitled “METHODS AND SYSTEMS FOR FORMING PERSONALIZED 3D HEAD AND FACIAL MODELS”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to image technology, and more specifically to a method, electronic device, storage medium, and program product for customizing the standard face of a virtual character. Background Technology

[0004] Commercial face capture systems with multiple sensors (e.g., multi-view cameras, depth sensors, etc.) are used to obtain accurate 3D facial models of people, with or without explicit markings. These tools capture geometric and textural information of the face from multiple sensors and fuse multimodal information into a general 3D facial model. Thanks to the multimodal information from various sensors, the obtained 3D facial model is accurate. However, these commercial systems are expensive and require additional software to process the raw data. Furthermore, these systems are typically deployed in face capture studios, requiring actors or volunteers to acquire data, making the data collection process time-consuming and costly. In short, acquiring 3D facial data using face capture systems is both expensive and time-consuming. Conversely, the widespread availability of smartphones and cameras today means that a large amount of RGB (red, green, blue) images are likely available. Using RGB images as input to generate 3D facial models can benefit from a large amount of image data.

[0005] Two-dimensional (2D) RGB images are merely projections of the 3D world onto a 2D plane. Recovering 3D geometry from a 2D image is an ill-posed problem, requiring optimization or learning algorithms to regulate the reconstruction process. For 3D face reconstruction, a 3D Morphable Model (3DMM) method based on parametric facial models has been developed and used. Specifically, facial models like the Basel Face Model (BFM) and the Surrey Face Model (SFM) are commonly used and require commercial licensing. The face model-based method uses a set of scanned 3D face models (showing various facial features and expressions) as its foundation, then generates parametric representations of facial features and expressions based on these 3D face models. The new 3D face can be represented as a linear combination of the parametric base 3D face models. Due to the nature of these methods, the 3D face models and parameter space used to form the foundation limit the expressive power of the face model-based method. Furthermore, the optimization process of fitting 3DMM parameters from the input face image or 2D landmarks further sacrifices detailed facial features in the face image. Therefore, face model-based methods cannot accurately recover 3D facial features and require commercial licenses to use face models such as BFM and SFM.

[0006] With the popularization of deep learning algorithms, semantic segmentation algorithms have received a lot of attention. Such algorithms can classify each pixel in a facial image into different categories, such as background, skin, hair, eyes, nose, and mouth.

[0007] While semantic segmentation methods can achieve relatively accurate results, performing semantic segmentation on all pixels is a highly complex problem, typically requiring intricate network structures and resulting in high computational complexity. Furthermore, training a semantic segmentation network requires labeling a large amount of training data, and semantic segmentation necessitates dividing the entire image into pixels, which is tedious, time-consuming, and expensive. Therefore, it is unsuitable for scenarios where high efficiency is more important than low average color accuracy.

[0008] Keypoint-driven deformation methods for optimizing the Laplacian operator and other derived operators have been well studied in academia. The mathematical expression for the biharmonic deformation can be represented as Δ 2 x′=0. The key point of the constraint, i.e., the boundary condition, can be expressed as x. b ′=x bc In the above equations, Δ is the Laplacian operator, x′ is the unknown position of the deformed mesh vertex, and x... bcThis refers to the location of a given keypoint after deformation. Each dimension requires a solution to the biLaplace equation. The biharmonic function is a solution to the biLaplace equation, but it is also a minimizer of the so-called "Laplace energy".

[0009] The essence of energy minimization is mesh smoothing. If the aforementioned minimizer is applied directly, all detailed features will be smoothed out. Furthermore, the deformed mesh should be identical to the original mesh when the positions of keypoints remain unchanged. For these considerations, the preferred application of biharmonic deformation is to solve for the displacements of vertices, rather than their positions. Thus, the deformed position can be written as x′=x+d, where d is the displacement of the unknown vertex in each dimension. Naturally, after d… b =x bc -x b The biharmonic transformation equation becomes Δ 2 d = 0, where d b It is the displacement of the key points after deformation.

[0010] With the rapid development of the gaming industry, the creation of custom-faced avatars is becoming increasingly popular. For ordinary players without artistic skills, adjusting control parameters to generate faces capable of depicting subtle changes is extremely difficult.

[0011] In some existing face generation systems and methods, such as the Justice face generation system, facial model prediction involves predicting 2D information in an image, such as the segmentation of eyebrows, mouth, nose, and other pixels in a photograph. These 2D segments are susceptible to out-of-face rotation and partial occlusion, and essentially require a frontal view. Furthermore, since the similarity between the final game's virtual face and the input is determined by a face recognition system, this method limits its application to games with only realistic styles. If the game's style is a cartoonish style very different from realistic faces, this method cannot be used.

[0012] In some other existing face generation systems and methods, such as the Moonlight Blade face generation system, realistic faces are reconstructed from input images. This method is limited to realistic-style games and cannot be applied to cartoon-style games. Secondly, the output parameter of this method is a reconstructed game-style facial mesh, and then template matching is performed on each part of the mesh. This method limits the combination of different surface parts. The overall diversity of game faces is closely related to the number of pre-generated templates. If a part (such as the mouth shape) has a small number of templates, it may produce few different variations, resulting in a lack of diversity in the generated faces. Summary of the Invention

[0013] Learning-based facial reconstruction and keypoint detection methods rely on 3D ground-truth data as a gold standard to train models that are as close as possible to the ground truth. Therefore, the 3D ground truth determines the upper limit of learning-based methods. To ensure accuracy in facial reconstruction and ideal keypoint detection, in some embodiments, 2D facial keypoint annotations are used to generate a ground-truth 3D facial model without using expensive facial capture systems. The method disclosed in this application generates a 3D ground-truth facial model that preserves detailed facial features of the input image, overcoming the shortcomings of existing facial models (such as 3DMM-based methods that lose facial features) and also avoiding the parametric facial models required by some existing facial model-based methods (such as BFM and SFM, both of which require commercial licensing).

[0014] In addition to facial landmark detection, some embodiments implement multi-task learning and transfer learning solutions for facial feature classification tasks, enabling the extraction of more information from the input facial image that complements the landmark information. The detected facial landmarks, together with the predicted facial features, are valuable for creating virtual facial avatars for players in computer or mobile games.

[0015] In some embodiments, this application discloses a lightweight method for extracting the average color of each part of a face from a single photograph, including the average colors of skin, eyebrows, pupils, lips, hair, and eyeshadow. Simultaneously, an algorithm is used to automatically transform a texture map based on the average color, such that the transformed texture retains the original brightness and color differences, but the primary color becomes the target color.

[0016] With the rapid development of computer vision and artificial intelligence (AI) technologies, the capture and reconstruction of 3D facial key points has reached a high level of accuracy. More and more games are utilizing AI detection to make game characters more lifelike. The method and system disclosed in this application customize 3D head virtual characters based on reconstructed 3D key points. The general key point-driven deformation method is applicable to any mesh. The head virtual character customization process and deformation method proposed in this application can be applied to scenarios such as automatic virtual character creation and expression reproduction.

[0017] This application discloses a method and system for automatically generating virtual facial characters in games based on a single photograph. By predicting facial key points, automatically processing these key points, and using deep learning methods to predict model parameters, the system disclosed in this application can automatically generate virtual facial characters in games that: 1) possess the characteristics of a realistic face from a photograph; and 2) conform to the style of the target game. This system can be applied to facial generation in both realistic and cartoon-style games and can be automatically adjusted according to different game models or skeletal definitions.

[0018] According to a first aspect of this application, a method for constructing a facial location map from a two-dimensional (2D) facial image of a subject includes: generating a coarse facial location map from the 2D facial image; predicting a first set of key points in the 2D facial image based on the coarse facial location map; identifying a second set of key points in the 2D facial image based on user-provided key point annotations; and updating the coarse facial location map to reduce the difference between the first set of key points and the second set of key points in the 2D facial image.

[0019] In some embodiments, the method for constructing a facial location map from a 2D facial image of a real person further includes: extracting a third set of key points as a final set of key points based on the updated facial location map, wherein the third set of key points has the same position as the first set of key points in the facial location map.

[0020] In some embodiments, the method of constructing a facial location map from a 2D facial image of a real person further includes: reconstructing a three-dimensional (3D) facial model of the real person based on the updated facial location map.

[0021] According to a second aspect of this application, a method for extracting color from a two-dimensional (2D) facial image of a subject includes: identifying multiple keypoints in the 2D facial image based on a keypoint prediction model; rotating the 2D facial image until multiple target keypoints from the identified multiple keypoints are aligned with corresponding target keypoints of a standard face; locating multiple portions in the rotated 2D facial image, wherein each portion is defined by a corresponding subset of the identified multiple keypoints; extracting the color of each portion from the pixel values ​​of the 2D facial image defined by the corresponding keypoint subsets; and using the colors extracted from the multiple portions of the 2D facial image to generate a three-dimensional (3D) model of the subject that matches the corresponding facial feature colors of the 2D facial image.

[0022] According to a third aspect of this application, a method for generating a three-dimensional (3D) deformable head model includes: receiving a two-dimensional (2D) facial image; identifying a first set of key points in the 2D facial image based on an artificial intelligence (AI) model; mapping the first set of key points to a second set of key points based on a user-provided set of key point annotations of multiple vertices of a mesh of a 3D head template model; deforming the mesh of the 3D head template model by reducing the difference between the first set of key points and the second set of key points to obtain a deformed 3D head mesh model; and applying a hybrid shaping method to the deformed 3D head mesh model to obtain a personalized head model based on the 2D facial image.

[0023] According to a fourth aspect of this application, a method is provided for customizing a standard face of a virtual character using a two-dimensional (2D) facial image of a subject. The method includes: identifying a set of subject keypoints in the 2D facial image; transforming the set of subject keypoints into a set of virtual character keypoints associated with the virtual character; generating a set of facial control parameters for the standard face by applying a keypoint-to-parameter (K2P) neural network model to the set of virtual character keypoints, each parameter in the set of facial control parameters being associated with one of a plurality of facial features of the standard face; and adjusting the plurality of facial features of the standard face by applying the set of facial control parameters to the standard face, wherein the adjusted standard face of the virtual character has facial features from the 2D facial image of the subject.

[0024] According to a fifth aspect of this application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When executed by the one or more processing units, the programs cause the electronic device to perform one or more methods as described above.

[0025] According to a sixth aspect of this application, a non-volatile computer-readable storage medium stores a plurality of programs executed by an electronic device having one or more processing units. When executed by the one or more processing units, the programs cause the electronic device to perform one or more methods as described above.

[0026] Note that the various embodiments described above can be combined with any other embodiments described in this application. The features and advantages described in the specification are not exhaustive, and in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, specification, and claims. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and instructional purposes and may not be intended to depict or limit the subject matter of the invention. Attached Figure Description

[0027] To gain a more detailed understanding of this disclosure, reference can be made to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and should not be considered limiting, as this specification may allow for other valid features.

[0028] Figure 1 This is a schematic diagram illustrating exemplary key point definitions according to some embodiments of the present disclosure.

[0029] Figure 2 This is a block diagram illustrating an exemplary keypoint generation process according to some embodiments of the present disclosure.

[0030] Figure 3 This is a schematic diagram illustrating an exemplary process of transforming an initial rough position diagram according to some embodiments of the present disclosure.

[0031] Figure 4 This is a schematic diagram illustrating an exemplary transformed positional view that does not cover the entire facial region according to some embodiments of the present disclosure.

[0032] Figure 5 This is a schematic diagram illustrating an exemplary process of modifying and transforming a positional view according to some embodiments of the present disclosure to cover the entire facial area.

[0033] Figure 6 This is a schematic diagram illustrating some exemplary results of a position map correction algorithm according to some embodiments of the present disclosure.

[0034] Figure 7A and Figure 7B The illustrations show some exemplary comparisons between final location diagrams and initial rough location diagrams according to some embodiments of the present disclosure.

[0035] Figure 8A This is a schematic diagram illustrating an exemplary eyeglasses classification network structure according to some embodiments of the present disclosure.

[0036] Figure 8B This is a schematic diagram illustrating an exemplary female hair prediction network structure according to some embodiments of the present disclosure.

[0037] Figure 8C This is a schematic diagram illustrating an exemplary male hair prediction network structure according to some embodiments of the present disclosure.

[0038] Figure 9A The illustration shows some exemplary eyeglass classification prediction results according to some embodiments of the present disclosure.

[0039] Figure 9B The illustration shows some exemplary female hair prediction results according to some embodiments of the present disclosure.

[0040] Figure 9C The illustration shows some exemplary male hair prediction results according to some embodiments of the present disclosure.

[0041] Figure 10 This is a flowchart illustrating an exemplary process for constructing a facial position map from a 2D facial image of a real person according to some embodiments of the present disclosure.

[0042] Figure 11 This is a flowchart illustrating an exemplary color extraction and adjustment process according to some embodiments of the present disclosure.

[0043] Figure 12 An exemplary skin color extraction method according to some embodiments of the present disclosure is illustrated.

[0044] Figure 13 An exemplary eyebrow color extraction method according to some embodiments of the present disclosure is illustrated.

[0045] Figure 14 An exemplary pupil color extraction method according to some embodiments of the present disclosure is illustrated.

[0046] Figure 15 An exemplary hair color extraction area is illustrated in a hair color extraction method according to some embodiments of the present disclosure.

[0047] Figure 16 The illustration shows an exemplary separation between hair pixels and skin pixels within a hair color extraction region according to some embodiments of the present disclosure.

[0048] Figure 17 An exemplary eyeshadow color extraction method according to some embodiments of the present disclosure is illustrated.

[0049] Figure 18 The illustration shows some exemplary color adjustment results according to some embodiments of the present disclosure.

[0050] Figure 19 This is a flowchart illustrating an exemplary process for extracting colors from a 2D facial image of a real person according to some embodiments of the present disclosure.

[0051] Figure 20 This is a flowchart illustrating an exemplary head-based virtual character deformation and generation process according to some embodiments of the present disclosure.

[0052] Figure 21 This is a schematic diagram illustrating the synthesis of an exemplary header template model according to some embodiments of the present disclosure.

[0053] Figure 22This is a schematic diagram illustrating some exemplary key point markings on realistic-style 3D models and cartoon-style 3D models according to some embodiments of the present disclosure.

[0054] Figure 23 This is a schematic diagram illustrating an exemplary comparison between template model rendering, manually marked key points, and AI-detected key points according to some embodiments of the present disclosure.

[0055] Figure 24 This is a schematic diagram illustrating an exemplary affine transformation of a triangle according to some embodiments of the present disclosure.

[0056] Figure 25 This is a schematic diagram illustrating an exemplary comparison of deformation results of some head models with and without a hybrid molding process according to some embodiments of the present disclosure.

[0057] Figure 26 This is a schematic diagram illustrating an exemplary comparison of affine deformations and biharmonic deformations with different weights according to some embodiments of the present disclosure.

[0058] Figure 27 The illustrations show some exemplary results automatically generated from randomly selected images of women using a real template model according to some embodiments of this disclosure.

[0059] Figure 28 This is a flowchart illustrating an exemplary process for generating a 3D deformable head model from a 2D facial image of a real person according to some embodiments of the present disclosure.

[0060] Figure 29 This is a schematic diagram illustrating exemplary key point processing steps according to some embodiments of the present disclosure.

[0061] Figure 30 This is a schematic diagram illustrating an exemplary keypoint smoothing process according to some embodiments of the present disclosure.

[0062] Figure 31 This is a block diagram illustrating exemplary key points of the control parameter (K2P) conversion process according to some embodiments of the present disclosure.

[0063] Figure 32 The illustrations show some exemplary results of automatically generating faces for mobile games according to some embodiments of the present disclosure.

[0064] Figure 33 This is a flowchart illustrating an exemplary process of customizing the standard face of a virtual character in a game using a 2D facial image of a real person, according to some embodiments of the present disclosure.

[0065] Figure 34This is a schematic diagram of an exemplary hardware structure of an image processing apparatus according to some embodiments of the present disclosure.

[0066] By convention, the various features illustrated in the accompanying drawings may not be drawn to scale. Accordingly, for clarity, the dimensions of various features may be arbitrarily enlarged or reduced. Additionally, some drawings may not depict all parts of a given system, method, or apparatus. Finally, the same reference numerals may be used to denote the same features throughout the specification and drawings. Detailed Implementation

[0067] Reference will now be made in detail to the specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices.

[0068] Before describing the embodiments of this application in further detail, the names and terms involved in the embodiments of this application are described, and the names and terms involved in the embodiments of this application have the following interpretations.

[0069] Facial key points: Predefined landmarks that define the shape of certain facial features, such as the corners of the eyes, chin, tip of the nose, and corners of the mouth.

[0070] Facial features: facial borders, eyes, eyebrows, nose, mouth, and other parts.

[0071] Facial reconstruction: Reconstructing the 3D geometry of a human face. Common representation methods include mesh models, point clouds, or depth maps.

[0072] RGB image: a three-channel image format consisting of red, green, and blue channels.

[0073] Location map: The red, green, and blue channels of a standard image format are used to store the x, y, and z coordinates of the facial region, which is a representation of the 3D face.

[0074] Facial feature classification: including hairstyle classification, and whether or not glasses are worn.

[0075] Convolutional Neural Networks (CNNs): A type of deep neural network most commonly used for analyzing visual images.

[0076] Base network: A CNN-like network that is used as a feature extractor by one or more downstream tasks.

[0077] The Laplace operator: a differential operator given by the divergence of the gradient of a function on Euclidean space.

[0078] Differentiable manifold: A type of topological space that locally resembles a linear space and can be used for calculus operations.

[0079] Biharmonic function: A quartic differentiable function defined on a differentiable manifold, whose squared Laplace operator equals 0.

[0080] Keypoint-driven deformation: a class of methods that deform a mesh by changing the position of certain vertices.

[0081] Biharmonic transformation: a transformation that utilizes the optimization of biharmonic functions and certain boundary conditions.

[0082] Affine deformation: The keypoint-driven deformation method proposed in this disclosure optimizes the affine transformation of triangles to achieve mesh deformation.

[0083] Facial Model: A mesh of a standard face from a predefined target game.

[0084] Bone / Slider: Control parameters that deform the facial model.

[0085] As mentioned earlier, even when both the input 2D image and 2D keypoints are fed into the optimization process to fit the 3DMM parameters, the optimization must strike a balance between the 3D facial model fitting based on the foundation (i.e., a set of 3D facial models) and the fidelity of the 2D keypoints. This optimization results in the obtained 3D facial model conflicting with the input 2D keypoints, thus sacrificing the facial detail information provided by the input 2D keypoints. In existing 3D facial reconstruction methods, facial capture schemes, while producing accurate reconstructions, are costly, time-consuming, and the data obtained shows limited variations in facial features (due to a limited number of actors). On the other hand, facial model-based methods can use 2D images or 2D landmark annotations as input, but the resulting 3D model is not accurate. To meet the demands of the rapid development of computer / mobile games, it is necessary to achieve ideal 3D model accuracy while reducing the required cost and time. To meet these requirements, this application discloses a novel 3D benchmark ground truth face model generation algorithm that takes a 2D image, 2D key point annotations, and a coarse 3D face model (location map format) as input, transforms the coarse 3D model based on the 2D key points, and finally generates a 3D face model with well preserved facial detail features.

[0086] In addition to addressing key issues in facial reconstruction and keypoint prediction, this application discloses a facial feature classification method based on multi-task learning and transfer learning, which is partially built upon a facial reconstruction and keypoint prediction framework. Specifically, the base network for facial reconstruction and keypoint prediction is reused to perform glasses classification (wearing glasses or not wearing glasses) through multi-task learning. A linear classifier is trained on top of the existing facial reconstruction and keypoint prediction framework, which greatly reuses existing models and avoids introducing another larger network for image feature extraction. Furthermore, another shared base network is used for male and female hairstyle classification. Hairstyle is an important facial feature that complements facial keypoints or 3D facial models. Adding hairstyle and glasses prediction during the creation of 3D virtual avatars for users can better reflect the user's facial features and provide a better personalized experience.

[0087] Facial landmark prediction has been a research topic in computer vision for decades. With the development of artificial intelligence and deep learning in recent years, convolutional neural networks (CNNs) have facilitated progress in facial landmark prediction. 3D face reconstruction and facial landmark detection are two intertwined problems; solving one can simplify the other. The traditional approach is to first solve 2D facial landmark detection and then infer a 3D facial model based on the estimated 2D facial landmarks. However, when the face in an image is tilted (nodding or shaking), some facial landmarks are occluded, leading to incorrect 2D facial landmark estimations. Therefore, the 3D facial model built on these incorrect 2D facial landmarks becomes inaccurate.

[0088] Since ground-value benchmarks determine the upper limit of deep learning-based methods, existing 3D face model datasets are not only limited in number but also primarily intended for academic research. On the other hand, face model-based methods require the use of Basel Face Model (BFM) or Surrey Face Model (SFM), both of which require commercial licensing. Therefore, high accuracy and a large pool of ground-value 3D benchmarks are critical issues when training any face reconstruction or keypoint estimation model.

[0089] Besides facial landmark prediction, facial feature classification is also a crucial aspect of creating user-defined 3D virtual avatars. With predicted facial landmarks, style transfer can be applied only to specific facial features (i.e., eyes, eyebrows, nose, mouth, and facial contours). However, to better reflect the user's facial features, matching the user's hairstyle and adding glasses (if the user is wearing glasses in the input image) is highly useful. Based on these requirements, a facial feature classification method based on multi-task learning and transfer learning was developed to predict male / female hairstyles and glasses (whether worn or not), making the created virtual avatars more personalized and improving the user experience.

[0090] In some embodiments, to represent the three-dimensional shape of the main facial features, methods such as... Figure 1 The key point representation shown is shown. Figure 1 This is a schematic diagram illustrating exemplary keypoint definitions according to some embodiments of the present disclosure. In other words, there is a mapping relationship between the sequence number of the keypoint and a specific location on the face. For example, sequence number 9 corresponds to the bottom of the chin, sequence number 21 corresponds to the tip of the nose, and so on. Keypoints are numbered in the order that they define specific facial features. Keypoints are concentrated on the boundaries of major facial parts, such as facial contours, eye contours, and eyebrow contours. The more keypoints there are, the more difficult the prediction becomes, but the higher the accuracy of the shape representation. In some embodiments, in Figure 1 The code uses 96 key points. In some embodiments, users can modify the specific definitions and number of key points according to their needs.

[0091] Many algorithms can predict the 3D coordinates of facial landmarks. Higher-performing methods use deep learning algorithms based on large amounts of offline 3D training data. However, in some embodiments, any 3D landmark prediction algorithm can be used. In some embodiments, the definition of landmarks is not fixed, and users can customize the definition of landmarks according to their needs.

[0092] To address the problem of generating ground truth 3D facial models, the following automated algorithm was developed, which takes 2D RGB images, 2D keypoint annotations, and a coarse location map as input. Figure 2 This is a block diagram illustrating an exemplary keypoint generation process according to some embodiments of the present disclosure. For example, a 2D RGB image of a face is used as input image 202, the 2D RGB image having a corresponding initial coarse location map 204, where each pixel in the initial coarse location map represents the spatial coordinates of a corresponding facial point in the 2D RGB image. 2D keypoint annotation 208 represents a set of keypoints provided by the user for correcting a set of keypoints 206 detected from the initial coarse location map 204.

[0093] Figure 3 This is a schematic diagram illustrating an exemplary process of transforming an initial rough position diagram according to some embodiments of the present disclosure.

[0094] In some embodiments, the 3D reconstruction method is used to convert an input facial image into a location map containing 3D depth information of facial features. For example, the location map can be a 2D three-color (RGB) channel map with a 256×256 matrix array, where each array element has coordinates (x, y, z) representing a 3D location on the facial model. The 3D location coordinates (x, y, z) are represented by the RGB pixel values ​​on the location map for each array element. Specific facial features are located at fixed 2D locations within the 2D location map. For example, the tip of the nose can be identified by the 2D array element locations at X=128 and Y=128 in the location map. Similarly, specific keypoints identified for specific facial features on the face can also be located at the same array element locations on the 2D location map. However, specific keypoints can have different 3D location coordinates (x, y, z) depending on the different input facial images for the location map.

[0095] In some embodiments, such as Figure 2 and Figure 3 As shown, an initial coarse location map (204, 304) is obtained from the input image (202, 302) using a 3D reconstruction method. Then, the (x, y) coordinates of the corresponding keypoints (206, 306) in the initial location map are adjusted using the input 2D keypoint annotations (208, 308) to ensure that the adjusted (x, y) coordinates of the keypoints in the adjusted location map are the same as those in the annotated 2D keypoints. Specifically, first, a set of 96 keypoints is obtained from the initial location map P. Based on the keypoint index, this set of 96 keypoints is called K = k_i, where each k_i is the 2D coordinate (x, y) of the keypoint, i = 0, ..., 95. A second set of 96 keypoints A = a_i is obtained from the 2D keypoint annotations (208, 308), where a_i is the 2D (x, y) coordinate, i = 0, ..., 95. Secondly, estimate the spatial transformation mapping (210, 310) from K to A, defined as T: Ω->Ω, where, Then, the obtained transformation T is applied to the initial position map P to obtain the transformed position map P'(212, 312). In this way, the transformed position map P'(212, 312) retains the detailed facial features of the person in the input image (202, 302), while also possessing reasonable 3D depth information. Therefore, the solution disclosed in this application provides an accurate and practical alternative for generating 3D baseline ground truth information, thereby avoiding the use of expensive and time-consuming face capture systems.

[0096] In some embodiments, since the 96 facial key points only cover a portion of the entire facial area (i.e., below the eyebrows and inside the facial contour), for example, in Figure 3In the image, keypoints from the ear to the chin follow the jawline but are not on the visible facial contour. When the face in the input image is tilted, the connected keypoint contours do not cover the entire facial area. Furthermore, when manually annotating keypoints, regardless of whether the face in the image is tilted, keypoints can only be annotated along the visible facial contour (i.e., occluded keypoints cannot be accurately annotated). Therefore, in the transformed position map P'(212, 312), a portion of the facial region has no valid values ​​because the transform map T(210, 310) does not estimate this region. Additionally, the forehead region is located above the eyebrows, so T also has no estimate in this region. All these issues result in the transformed position map P'(212, 312) having no valid values ​​in certain areas. Figure 4 This is a schematic diagram illustrating an exemplary transformed positional view that does not cover the entire facial area according to some embodiments of the present disclosure.

[0097] exist Figure 4 In the center, the top circles (402, 406) highlight the forehead area, while the right circles (404, 408) indicate areas where the key point outlines are smaller than the visible facial outlines.

[0098] In some embodiments, to address the aforementioned issues and make the algorithm robust to tilted faces commonly found in facial images, methods such as... Figure 2 The correction process 214 is shown. Based on the head pose and a rough 3D facial model, keypoints in the transformed location map are shifted along the facial contour to match the visible facial contour. Missing values ​​in the facial contour regions can then be filled in the obtained location map. However, values ​​in the forehead region remain missing. To cover the forehead region, control points are expanded by adding eight landmark points from the four corners of the image to the keypoint sets K and A.

[0099] Figure 5 This is a schematic diagram illustrating an exemplary process of modifying and transforming a position map according to some embodiments of the present disclosure to cover the entire facial region. The position map modification process is... Figure 5 As shown in the image.

[0100] In some embodiments, the head pose is first determined based on a coarse position map P to determine whether the head is tilted to the left or right, where left or right is defined in the 3D facial model space (e.g., as shown in the image). Figure 5 As shown, the face is tilted to the left. Based on determining whether the face is tilted to the left or right, the key points on the corresponding side of the facial contour are adjusted. Key points on the right side of the facial contour have indices from 1 to 8, and key points on the left side have indices from 10 to 17. Taking a face tilted to the left as an example, the 2D projection of the initial position map P is calculated to obtain the depth map, as shown... Figure 5Image 502 is shown. The facial contour keypoints k_i (i = 10, ..., 17) on the left side are shifted to the right until they reach the boundary of the depth map. Then, the original keypoint positions are replaced with the new coordinates. Similarly, when the face is tilted to the right, the processed keypoints are indexed by k_i (i = 1, ..., 8) and searched to the left. After adjusting the facial contour keypoints, the updated keypoints are visualized as follows. Figure 5 Image 504 in the image, the updated location map coverage is shown as Figure 5 Image 506 in the image. The updated location map better covers the facial contour area, but the forehead area still has missing values.

[0101] In some embodiments, to cover the forehead region, two anchor points are added at each corner of the image domain Ω as additional keypoints k_i, i = 96, ..., 103, to obtain an updated set of keypoints K' (e.g., ...). Figure 5 (As shown in image 508). The same operation is performed on the manually annotated keypoint set a_i, i = 96, ..., 103, to obtain the updated A'. Using the updated keypoint sets K' and A', the transformation mapping T' is re-estimated and then applied to the initial location map P to obtain the final location map P(…). Figure 2 216 in the middle), to cover the entire facial area (such as Figure 5 (As shown in image 510). The final key point 218 is derived from the final position diagram 216.

[0102] Figure 6 These are schematic diagrams illustrating some exemplary results of a position map correction algorithm according to some embodiments of the present disclosure. 602 is an illustration of the position map after the initial transformation. 604 is an illustration of the position map after the update following correction of the facial contour. 606 is an illustration of the final position map.

[0103] Figure 7A and Figure 7B The illustrations show some exemplary comparisons between final location diagrams and initial rough location diagrams according to some embodiments of the present disclosure. Figure 7A In one example, the nose and its associated 3D model and keypoint 702 in the initial location image are incorrect and do not reflect the facial features of the person at all (highlighted by the arrow). However, after applying the method described in this application, the nose is perfectly aligned with the image and its associated 3D model and keypoint 704 (highlighted by the arrow) in the final location image. Figure 7BIn the second example, several inaccuracies exist in the initial location map and its associated 3D model and keypoints 706, such as facial contours, an open mouth, and a mismatched nose shape (indicated by arrows). In the final location map and its associated 3D model and keypoints 708, all these errors are corrected (indicated by arrows).

[0104] Hairstyle and glasses classification are crucial for the creation of facial virtual characters in mobile game applications. In some embodiments, this application implements solutions based on multi-task learning and transfer learning to address these issues.

[0105] In some embodiments, four different classification tasks (head) were performed for female hair prediction. The classification categories and parameters are shown below:

[0106] Category Head 1: Curved

[0107] Straight hair (0); Curly hair (1)

[0108] Category 2: Long Hair

[0109] Short hair (0); Long hair (1)

[0110] Category Header 3: Bangs

[0111] No bangs or middle part (0); left side part (1); right side part (2); M-shaped (3); straight bangs (4); natural bangs (5); wispy bangs (6)

[0112] Category 4: Braids

[0113] A single braid (0); two or more braids (1); a single bun (2); two or more buns (3); other (4).

[0114] In some embodiments, three different classification tasks (head) were performed for male hair prediction. The classification categories and parameters are as follows:

[0115] Classification of head 1: Very short (0), curly (1), other (2)

[0116] Category 2: No bangs (0), Middle-parted bangs (1), Natural bangs (2)

[0117] Category 3: Left-side parted bangs (0) and right-side parted bangs (1)

[0118] In some embodiments, glasses classification is a binary classification task. The classification parameters are as follows:

[0119] No glasses (0); Glasses (1).

[0120] Among various deep learning image classification models, those that achieve state-of-the-art accuracy on ImageNet typically have large model sizes and complex architectures, such as EfficientNet, Noisy Student, and FixRes. When deciding which architecture to use as the base network for feature extractors, both prediction accuracy and model size must be considered. In practice, a 1% improvement in classification accuracy may not bring noticeable changes to the end user, but the model size can increase exponentially. Given that trained models may need to be deployed on clients, a smaller base network allows for flexible deployment on both server and client sides. Therefore, for example, MobileNetV2 can be used as the base network for transfer learning on different classification heads. The MobileNetV2 architecture is based on an inverse residual structure, where the input and output of the residual block are both thin bottleneck layers, unlike traditional residual models that use expanded representations in the input. MobileNetV2 uses lightweight deep convolutions in the intermediate expansion layers to filter features.

[0121] For glasses classification, a multi-task learning approach was used. A keypoint prediction network was repeatedly used as the base network with its parameters frozen. A binary classifier was trained using feature vectors with cross-entropy loss in the bottleneck layer of a U-shaped network. Figure 8A This is a schematic diagram illustrating an exemplary eyeglasses classification network structure according to some embodiments of the present disclosure. Figure 8B This is a schematic diagram illustrating an exemplary female hair prediction network structure according to some embodiments of the present disclosure. Figure 8C This is a schematic diagram illustrating an exemplary male hair prediction network structure according to some embodiments of the present disclosure.

[0122] Figure 9A The illustration shows some exemplary eyeglass classification prediction results according to some embodiments of the present disclosure. Figure 9B The illustration shows some exemplary female hair prediction results according to some embodiments of the present disclosure. Figure 9C The illustration shows some exemplary male hair prediction results according to some embodiments of the present disclosure.

[0123] Figure 10This is a flowchart 1000 illustrating an exemplary process for constructing a facial location map from a 2D facial image of a real person according to some embodiments of this disclosure. In real life, different people have different facial features, so the same key points corresponding to the same facial features (e.g., the position of eyebrows on a person's face) may have very different spatial coordinates. The face detection problem becomes more challenging because the 2D facial images used to generate 3D facial models are taken at different angles and under different lighting conditions, and research in this area has been a very active topic in the field of computer vision. In this application, various methods are proposed to improve the efficiency and accuracy of detecting facial key points in any 2D facial image of a subject, from real people to cartoon characters. In some embodiments, a set of facial key points from the same facial image provided by a user is used as a reference to correct or improve a set of facial key points initially detected by a computer-implemented method. For example, because there is a one-to-one mapping between user-provided facial key points and computer-generated facial key points based on their respective sequence numbers, the correction of computer-generated facial key points is defined as an optimization problem, namely, reducing the difference between the two sets of facial key points (e.g., the difference measured by their corresponding spatial coordinates in the location map).

[0124] The process of constructing a facial location map includes step 1010: generating a rough facial location map from a 2D facial image.

[0125] The process also includes step 1020: predicting the first set of key points in the 2D facial image based on a rough facial location map.

[0126] The process also includes step 1030: identifying a second set of key points in the 2D facial image based on user-provided key point annotations.

[0127] The process further includes step 1040: updating the coarse facial location map to reduce the difference between the first set of keypoints and the second set of keypoints in the 2D facial image. For example, by reducing the difference between the first set of keypoints and the second set of keypoints in the 2D facial image in their corresponding spatial coordinates, the first set of keypoints in the 2D facial image based on the coarse facial location map is modified to be more similar to the second set of keypoints in the 2D facial image based on user-provided keypoint annotations, which is generally considered more accurate. The modification of the first set of facial keypoints automatically triggers an update of the initial coarse facial location map, which was generated based on the initial coarse facial location map. The updated coarse facial location map can then be used to predict a more accurate set of keypoints from the 2D facial image. It should be noted that generating the second set of keypoints in the 2D facial image based on user-provided keypoint annotations does not mean it is done manually. Instead, the user can employ another computer-implemented method to perform the annotation. In some embodiments, although the number of key points in the second set (e.g., 10 to 20) is only a fraction of the number of key points in the first set (e.g., 96 or even higher), the fact that the second set of key points is more accurate contributes to the overall improvement of the first set of key points.

[0128] In one implementation, the process further includes step 1050: extracting a third set of key points as the final set of key points based on the updated facial location map / final location map, the third set of key points having the same position as the first set of key points in the facial location map. In some embodiments, the position of the key points in the facial location map is represented by the 2D coordinates of the array elements in the location map. As described above, the updated facial location map benefits from the second set of key points in the 2D facial image based on user-provided key point annotations, therefore the third set of key points is more accurate and can be used in fields such as computer vision for more accurate face detection, or in the field of computer graphics for more accurate 3D facial modeling.

[0129] In one implementation, as an alternative to or supplement to step 1050, the process further includes step 1060: reconstructing a 3D facial model of the real person based on the updated facial location map. In one example, the 3D facial model is a 3D depth model.

[0130] Additional implementations may include one or more of the following features.

[0131] In some embodiments, the updating step 1040 may include: transforming the coarse facial location map into a transformed facial location map, and correcting the transformed facial location map. As described above, compared to the initial coarse facial location map, the transformed facial location map can retain more detailed facial features of the person in the input image, thus the 3D facial model based on the transformed facial location map is more accurate.

[0132] In some embodiments, the transformation includes: estimating a transformation mapping from a rough facial location map to a transformed facial location map by learning the difference between the first set of key points and the second set of key points; and applying the transformation mapping to the rough facial location map.

[0133] In some embodiments, the correction includes: based on the determination of tilt in the 2D facial image, adjusting key points corresponding to the transformed facial position map on the occluded side of the facial contour to cover the entire facial area. As mentioned above, different 2D facial images may have been captured from different angles. This correction step can correct for deviations or errors caused by different image capturing conditions and preserve a more accurate 3D facial model from the 2D facial image. Furthermore, compared to the initial coarse facial position map, the transformed facial position map can preserve more detailed facial features of the person in the input image, thus the 3D facial model based on the transformed facial position map is more accurate.

[0134] In some embodiments, the first set of key points may include 96 key points.

[0135] In some embodiments, the process of constructing a facial location map may include facial feature classification.

[0136] In some embodiments, facial feature classification is performed via deep learning methods.

[0137] In some embodiments, facial feature classification is performed via multi-task learning or transfer learning methods.

[0138] In some embodiments, facial feature classification includes hair prediction classification.

[0139] In some embodiments, hair prediction classification includes female hair prediction with multiple classification tasks, which may include: curl, length, bangs, and braids.

[0140] In some embodiments, hair prediction classification includes male hair prediction with multiple classification tasks, which may include: curl / length, bangs, and parting.

[0141] In some embodiments, facial feature classification includes glasses prediction classification. Glasses prediction classification includes classification tasks that may include: wearing glasses and not wearing glasses.

[0142] The method and system disclosed in this application can generate accurate 3D facial models (i.e., location maps) based on 2D keypoint annotations used to generate 3D baseline ground truth. This method not only avoids the use of BFM and SFM facial models, but also better preserves detailed facial features, preventing the loss of these important features caused by facial model-based methods.

[0143] In addition to providing key points, using deep learning-based solutions to provide supplementary facial features such as hairstyle and glasses is crucial for personalizing virtual facial characters based on user-input facial images.

[0144] While hairstyle and glasses prediction for facial feature classification are disclosed as examples in this application, the framework is not limited to these example tasks. The framework and solution are based on multi-task learning and transfer learning, meaning it can be easily extended to other facial features, such as female makeup type classification, male beard type classification, and mask-wearing classification. The framework is well-suited for expansion to more tasks based on the needs of various computer or mobile games.

[0145] In some embodiments, this application describes a lightweight color extraction method based on key points. The lightweight image processing algorithm quickly estimates local pixels without segmenting all pixels, thus improving efficiency.

[0146] During training, users do not need to have pixel-level markings, but only mark a few key points, such as the corners of the eyes, the borders of the mouth, and the eyebrows.

[0147] The lightweight color extraction method disclosed in this application can be used in personalized facial generation systems for various games. To provide more freedom in generating personalized characters, many games have begun to adopt adjustable methods. In addition to adjusting the face shape, users can also choose different color combinations. For aesthetic purposes, game faces often use predefined textures instead of real facial textures. The method and system disclosed in this application only require the user to upload a photo to automatically extract the average color of each part of the face. Simultaneously, the system can automatically modify the texture based on the extracted colors, making the generated personalized facial features closer to the real colors in the user's photo, thereby improving the user experience. For example, if the user's skin tone is darker than the average skin tone of most people, the skin tone of the character in the game will be correspondingly darker. Figure 11 This is a flowchart illustrating an exemplary color extraction and adjustment process according to some embodiments of the present disclosure.

[0148] To locate the various parts of the face, as shown above Figure 1 As shown, keypoints are defined for the main features of the face. The algorithm described above is used for keypoint prediction. Unlike semantic segmentation methods, keypoints are predicted only within the image, without classifying each pixel, thus significantly reducing prediction costs and the cost of labeling training data. With these keypoints, the various parts of the face can be roughly located.

[0149] Figure 12An exemplary skin color extraction method according to some embodiments of the present disclosure is illustrated. In order to extract features from the image, the facial region in the original image 1202 needs to be rotated so that key points 1 and 17 on the left and right sides of the face are aligned with corresponding key points on the left and right sides of a standard face, as shown in the rotated and aligned image 1204.

[0150] Next, the skin tone pixel detection area is determined. The bottom coordinates of the eye keypoint are selected as the upper boundary of the detection area, and the bottom coordinates of the nose keypoint are selected as the lower boundary of the detection area. The left and right boundaries are determined by the facial boundary keypoints. In this way, the skin color detection area is obtained as shown in region 1208 of image 1206.

[0151] Not all pixels in region 1208 are skin pixels; these pixels may also include eyelashes, nostrils, nasolabial folds, hair, etc. Therefore, the median R, G, and B values ​​of all pixels in this region are selected as the final predicted average skin color.

[0152] Figure 13 The illustration depicts an exemplary eyebrow color extraction method according to some embodiments of the present disclosure. For the average eyebrow color, the primary eyebrow, i.e., the eyebrow closer to the camera side, is first selected as the target. In some embodiments, if both eyebrows are primary eyebrows, the pixels of both eyebrows are extracted. Assuming the left eyebrow is the primary eyebrow, as... Figure 13 As shown, the quadrilateral region formed by keypoints 77, 78, 81, and 82 is selected as the eyebrow pixel search area. This is because the eyebrow hairs near the outer edges are too thin, and the impact of small keypoint errors will be amplified. Since the eyebrow hairs near the inner edges are usually sparse and blend with the skin color, the middle eyebrow region 1302 is selected to collect pixels. Each pixel must first be compared with the average skin color, and only pixels with a difference greater than a certain threshold are collected. Finally, similar to the skin color, the median of the R, G, and B values ​​of the collected pixels is selected as the final average eyebrow color.

[0153] Figure 14 The illustration shows an exemplary pupil color extraction method according to some embodiments of the present disclosure. Similar to eyebrow color extraction, when extracting pupil color, the side of the dominant eye closer to the lens is first selected. In some embodiments, if both eyes are dominant eyes, the pixels from both sides are collected together. In addition to the pupil itself, the enclosed area contained within the eye keypoint may also include eyelashes, sclera (white of the eye), and reflections. During pixel collection, these pixels should be removed as much as possible to ensure that the majority of the final pixels come from the pupil itself.

[0154] To remove eyelash pixels, the key points of the eye are along the y-axis ( Figure 14 (Vertical direction) contracting inward by a certain distance to form Figure 14 The area shown is 1402. This is to remove sclera and reflections (such as...). Figure 14 (As shown in circle 1404), such pixels are further excluded in this area 1402. For example, if the R, G, and B values ​​of a pixel are all greater than a predefined threshold, that pixel is excluded. Pixels collected in this way ensure that the majority come from the pupil itself. Similarly, the color median is used as the average pupil color.

[0155] In some embodiments, for lip color extraction, only the pixels in the lower lip region are detected. The upper lip is typically thinner, relatively sensitive to keypoint errors, and because its color is lighter, it does not represent lip color well. Therefore, after rotating and correcting the image, all pixels in the region enclosed by the lower lip keypoints are collected, and the median color is used to represent the average lip color.

[0156] Figure 15 The illustration shows an exemplary hair color extraction region used in a hair color extraction method according to some embodiments of the present disclosure. Hair color extraction is more difficult than the previous sections. The main reason is that everyone's hairstyle is unique, and the background of the photo is also complex and diverse. Therefore, it is difficult to locate the hair pixels. One way to accurately locate hair pixels is to use a neural network to segment the hair pixels in the image. Since the annotation cost of image segmentation is very high, and game applications do not require very high-precision color extraction, a keypoint-based approximate prediction method is adopted.

[0157] To obtain hair pixels, the detection area is first determined. For example... Figure 15 As shown, the detection area 1502 is rectangular. Its lower boundary is the two eyebrow corners, and its height (vertical line 1504) is the distance 1506 from the upper edge of the eyebrow to the lower edge of the eye. Keypoints 1 and 17 on the left and right sides extend to the left and right by fixed distances, respectively. The resulting hair pixel detection area 1502... Figure 15 As shown in the image.

[0158] Figure 16 The illustration shows an exemplary separation between hair pixels and skin pixels within a hair color extraction region according to some embodiments of this disclosure. Typically, the detection region contains three types of pixels: skin, hair, and background. In some more complex cases, headwear is also included. Because the left and right extents of our detection region are relatively conservative, it is assumed that in most cases, the number of hair pixels will far exceed the number of background pixels. Therefore, the main process is to divide the pixels of the detection region into hair or skin.

[0159] For each row of pixels in the detection region, skin color changes are typically continuous, such as from light to dark, with a noticeable change at the boundary between skin color and hair color. Therefore, the middle pixel of each row is chosen as the starting point (1608), and skin pixels are detected to the left and right. First, a relatively conservative threshold is used to find a reliable skin color pixel, and then the detection expands to the left and right. If adjacent pixels have similar colors, they are also marked as skin color. This method considers the gradation of skin color and can obtain relatively accurate results. Figure 16 As shown, within the hair color extraction area 1602, darker areas such as 1604 represent skin color pixels, and brighter areas such as 1606 represent hair color pixels. Within the hair color area, the median of the collected R, G, and B values ​​of the hair color pixels is selected as the final average hair color.

[0160] Figure 17 The illustration shows an exemplary eyeshadow color extraction method according to some embodiments of the present disclosure. The extraction of eyeshadow color differs slightly from the previous sections. This is because eyeshadow is makeup that may or may not be present. Therefore, when extracting eyeshadow color, it is first necessary to determine whether eyeshadow is present; if present, its average color is extracted. Eyebrow and pupil color extraction are similar; eyeshadow color extraction is performed only on the portion of the main eye closest to the lens.

[0161] First, it's necessary to determine which pixels belong to the eyeshadow. For the detection area of ​​eyeshadow pixels, use region 1702 within lines 1704 and 1706, such as... Figure 17 As shown, the left and right sides of region 1702 are defined as the inner and outer corners of the eye, while the upper and lower sides of the region are the lower edge of the eyebrow and the upper edge of the eye. In addition to the possible eyeshadow pixels in region 1702, there may also be eyelashes, eyebrows, and skin, which need to be excluded when extracting eyeshadow.

[0162] In some embodiments, to eliminate the influence of eyebrows, the upper edge of the detection area is further lowered. To reduce the influence of eyelashes, pixels with brightness below a certain threshold are excluded. To distinguish eyeshadow from skin color, the difference between the hue of each pixel and the average skin color hue is checked. Only pixels with a difference greater than a certain threshold are collected as potential eyeshadow pixels. Hue is used instead of RGB values ​​because the average skin color primarily collects the skin color below the eyes, while the skin color above the eyes may have greater brightness variations. Since color is not sensitive to brightness, it is relatively stable. Therefore, hue is more suitable for determining whether a pixel is skin.

[0163] The above process determines whether a pixel in each detection area belongs to eyeshadow. In some embodiments, if eyeshadow is not present, some pixels may still be incorrectly identified as eyeshadow.

[0164] To reduce the aforementioned errors, each column of the detection region is examined. If the number of eyeshadow pixels in the current column exceeds a certain threshold, the current column is marked as an eyeshadow column. If the ratio of the eyeshadow column to the width of the detection region exceeds a certain threshold, eyeshadow is considered to exist in the current image, and the intermediate color of the collected eyeshadow pixels is used as the final color. In this way, a few pixels misclassified as eyeshadow will not cause incorrect judgments about the overall eyeshadow.

[0165] Considering art style, most games typically don't allow free adjustment of the colors in all of the above-mentioned aspects. For the parts where color adjustment is allowed, it usually only allows matching a predefined set of colors. Taking hair as an example, if a hairstyle allows five hair colors to be selected, the hairstyle in the resource pack will contain texture images corresponding to each hair color. During detection, the desired hair rendering effect can be obtained by simply selecting the texture image with the closest color based on the hair color prediction result.

[0166] In some embodiments, when only a single color texture image is provided, the color of the texture image can be reasonably altered based on any detected color. For ease of color conversion, the commonly used RGB color space representation is converted to the HSV color model. The HSV color model consists of three dimensions: hue (H), saturation (S), and lightness (V). Hue (H) in this model is represented as a 360-degree color range, where red is 0 degrees, green is 120 degrees, and blue is 240 degrees. Saturation (S) represents the mixture of spectral colors and white. Higher saturation results in a brighter color. When saturation approaches 0, the color is close to white. Lightness (V) represents the brightness of the color, ranging from black to white. After color adjustment, the HSV median of the texture image is expected to match the predicted color. Therefore, the hue value of each pixel can be calculated as follows: H i ′=(H i +H′-H)%1, where H i ′ and H i H represents the hue of pixel i before and after adjustment, and H and H′ represent the median hue of the texture image before and after adjustment.

[0167] Unlike hue, which is a continuous space with consecutive beginnings and endings, saturation and brightness have boundary singularities such as 0 and 1. If a linear processing method similar to hue adjustment is used, many pixel values ​​will exhibit excessively high or low saturation or brightness when the median of the initial or adjusted image is close to 0 or 1. This phenomenon results in unnatural colors. To address this issue, the following non-linear curve is used to fit the saturation and brightness before and after pixel adjustment:

[0168] y = 1 / (1+(1-α)(1- x ) / (α x )), α∈(0,1)

[0169] In the above equation, x and y represent the saturation or brightness values ​​before and after adjustment, respectively. The only uncertain parameter is α, whose derivation formula is:

[0170] α = 1 / (1+x / (1-x)×(1-y) / y)

[0171] This equation ensures that α falls within the interval of 0 to 1. Taking saturation as an example, the initial median saturation... S The target saturation value S can be easily calculated based on the input image. t This can be obtained through hair color extraction and color space conversion. Therefore, α = 1 / (1+ S / (1- S )×(1-S t ) / S t For each pixel S in the default texture image. i The adjusted value can then be calculated using the following equation: S i =1 / (1+(1-α)(1-S)) i ) / (αS i The same calculation method also applies to lightness.

[0172] To make the adjusted texture image display more closely resemble a realistic image, special processing is applied to different parts. For example, to maintain low saturation in the hair, [the following settings are used]. S′ = S′ × V′ ^0.3. Figure 18 The illustrations show some exemplary color adjustment results according to some embodiments of this disclosure. Column 1802 illustrates some default texture images provided by a specific game, column 1804 illustrates some texture images adjusted from the corresponding default texture images in the same row based on the actual image displayed at the top of column 1804, and column 1806 illustrates some texture images adjusted from the corresponding default texture images in the same row based on the actual image displayed at the top of column 1806.

[0173] Figure 19 This is a flowchart 1900 illustrating an exemplary process for extracting color from a 2D facial image of a real person according to some embodiments of the present disclosure.

[0174] The process of extracting color from a 2D facial image of a real person includes step 1910: identifying multiple key points in the 2D facial image based on a key point prediction model.

[0175] The process also includes step 1920: rotating the 2D facial image until multiple target keypoints from the identified multiple keypoints are aligned with the corresponding target keypoints of the standard face;

[0176] The process further includes step 1930, which involves locating multiple parts in the rotated 2D facial image, each part being defined by a corresponding subset of the identified multiple key points.

[0177] The process also includes step 1940: extracting the color of each of the multiple parts defined by the corresponding subset of key points from the pixel values ​​of the 2D facial image.

[0178] The process also includes step 1950: using the colors of multiple parts extracted from the 2D facial image to generate a personalized 3D model of a real person that matches the colors of the corresponding facial features in the 2D facial image.

[0179] Additional implementations may also include one or more of the following features.

[0180] In some embodiments, the keypoint prediction model in step 1910 is formed by machine learning based on keypoints manually annotated by the user.

[0181] In some embodiments, the selected key points used for alignment in rotation step 1920 are located on the left and right symmetrical sides of the 2D facial image.

[0182] In some embodiments, in step 1940, extracting the average color of each of the multiple portions may include: selecting the median of the R, G, B values ​​of all pixels in the corresponding defined region within the corresponding portion as the predicted average color.

[0183] In some embodiments, step 1940, extracting the average color of each of the plurality of portions, may include: determining a skin color extraction region within a skin portion, and selecting the median of the R, G, and B values ​​of all pixels in the skin color extraction region as the predicted average color of the skin portion. In some embodiments, the skin color extraction region within a skin portion is determined to be the area below the eyes and above the lower edge of the nose.

[0184] In some embodiments, step 1940, extracting the average color of each of the plurality of portions, may include eyebrow color extraction within the eyebrow portion, the eyebrow color extraction comprising: selecting an eyebrow as a target eyebrow based on determining that the eyebrow is located on the side closer to the observer of the 2D facial image; selecting two eyebrows as target eyebrows based on determining that both eyebrows are equally close to the observer of the 2D facial image; extracting one or more intermediate eyebrow regions within one or more target eyebrows; comparing each pixel value within the one or more intermediate eyebrow regions with the average skin color; collecting pixels within the one or more intermediate eyebrow regions whose pixel value difference with the average skin color exceeds a threshold; and selecting the median of each of the R, G, and B values ​​of the pixels collected for eyebrow color extraction as the predicted average color of the eyebrow portion.

[0185] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include pupil color extraction within the eye portion, the pupil color extraction comprising: selecting an eye as a target eye based on determining that the eye is located on the side of the observer closer to the 2D facial image; selecting two eyes as target eyes based on determining that both eyes are equally close to the observer in the 2D facial image; extracting one or more regions without eyelashes within one or more target eyes; comparing each pixel value within one or more extracted regions with a predetermined threshold; collecting pixels within one or more extracted regions whose pixel values ​​exceed the predetermined threshold; and selecting the median of each of the R, G, and B values ​​of the pixels collected for pupil color extraction as the predicted average color of the pupil.

[0186] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include lip color extraction within the lip portion, the lip color extraction comprising: collecting all pixels in the region surrounded by keypoints of the lower lip, and selecting the median of each of the R, G, B values ​​of the pixels collected for lip color extraction as the predicted average color of the lip portion.

[0187] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include hair color extraction within a hair portion, the hair color extraction comprising: identifying a region including a forehead portion extending to both sides of the hair portion; determining pixel color variations exceeding a predetermined threshold from the middle of the region to its left and right boundaries; dividing the region into a hair region and a skin region based on the pixel color variations exceeding the predetermined threshold; and selecting the median of each of the R, G, and B values ​​of the pixels in the hair region within the region as the predicted average color of the hair portion.

[0188] In some embodiments, the region including the portion of the forehead extending to the hairline on both sides is identified as a rectangular region, with its lower boundary at the two eyebrow corners, its left and right boundaries at fixed distances outward from key points located on the symmetrical left and right sides of the 2D facial image, and its height at a certain distance from the upper edge of the eyebrow to the lower edge of the eye.

[0189] In some embodiments, in step 1940, extracting the average color of each of the plurality of portions may include eyeshadow color extraction within an eyeshadow portion, the eyeshadow color extraction comprising: selecting an eye as a target eye based on determining that the eye is located on the side of the observer closer to the 2D facial image; selecting two eyes as target eyes based on determining that both eyes are equally close to the observer in the 2D facial image; extracting one or more intermediate regions within the eyeshadow portion near one or more target eyes, collecting pixels in the one or more extracted intermediate regions whose brightness is higher than a predetermined brightness threshold to exclude eyelashes, and pixels whose pixel hue value differs from the average skin hue value by a predetermined threshold; marking a pixel column as an eyeshadow column based on determining that the number of pixels collected in a pixel column within one or more extracted intermediate regions is greater than a threshold; and selecting the median of each of the R, G, and B values ​​of the pixels collected for eyeshadow color extraction as the predicted eyeshadow color of the eyeshadow portion based on determining that the ratio of the width of the eyeshadow column to the width of the extracted intermediate regions is greater than a specific threshold.

[0190] In some embodiments, the process of extracting color from a 2D facial image of a real person may further include converting a texture map based on an average color while preserving the original brightness and color differences of the texture map, which includes: converting the average color from an RGB color space representation to an HSV (hue, saturation, lightness) color space representation, and adjusting the color of the texture map to reduce the difference between the median HSV value of the average color and the median HSV value of the pixels in the texture map.

[0191] The methods and systems disclosed in this application can be used in various scenarios, such as character modeling and game character generation. This lightweight method can be flexibly applied to different devices, including mobile devices.

[0192] In some embodiments, the definition of facial key points in the current system and method is not limited to the current definition; other definitions can be used as long as they can adequately express the contour of each part. Additionally, in some embodiments, the colors directly returned in the solution may not be used directly, but can be matched with a predefined color list to achieve further color filtering and control.

[0193] Deformation methods for optimizing the Laplacian operator require the mesh to be a separable micromanifold. However, in practice, meshes created by game artists often contain artifacts such as repeating vertices and unclosed edges that can disrupt manifold properties. Therefore, methods such as biharmonic deformation can only be used after careful mesh cleaning. The affine deformation method proposed in this application does not use the Laplacian operator and therefore does not have such a strong constraint.

[0194] Deformation methods, such as biharmonic deformation, suffer from insufficient deformation capability in some cases. Solving the harmonic function of the Laplacian operator in a single step often fails to achieve smooth results due to its low smoothness requirement. Solving the multiharmonic function of higher-order (≥3) Laplacian operators is impossible on many meshes due to its requirement of at least 6th-order differentiability. In most cases, only biharmonic deformation solving of the Laplacian operator twice can yield acceptable results. Even so, the deformation results are still unsatisfactory due to the lack of tuning freedom. The affine deformation proposed in this application allows for fine deformation tuning by changing the smoothness parameter, and its deformation results cover the range of applications using biharmonic deformation.

[0195] Figure 20 This is a flowchart illustrating an exemplary head-shaped virtual character deformation and generation process according to some embodiments of the present disclosure. Using the techniques proposed in this disclosure, the head mesh can be appropriately deformed without being combined with the skeleton. Therefore, the workload required by artists is greatly reduced. This technique can adapt to different styles of meshes, thus achieving better versatility. During the creation of game assets, artists can use tools such as 3ds Max or Maya to save head models in various formats, but the internal representation of these formats is a polygonal mesh. Polygonal meshes can be easily converted into pure triangular meshes, called template models. For each template model, 3D key points are manually marked once on the template model. After this, based on the 3D key points detected and reconstructed from arbitrary face images, the template model is deformed into a virtual character head.

[0196] Figure 21 This is a schematic diagram illustrating the synthesis of an exemplary header template model according to some embodiments of this disclosure. For example... Figure 21 As shown, the head template model 2102 typically consists of parts such as the face 2110, eyes 2104, eyelashes 2106, teeth 2108, and hair. Without being integrated with the skeleton, mesh deformation depends on the connection structure of the template mesh. Therefore, the template model needs to be decomposed into those semantic parts, and the facial mesh needs to be deformed first. By setting and tracking certain key points on the facial mesh, all other parts can be automatically adjusted. In some embodiments, an interactive tool is provided to detect all topologically connected parts, which the user can use to easily deduce those semantic parts for further deformation.

[0197] In some embodiments, facial keypoints can be obtained using detection algorithms or AI models. To drive mesh deformation, these keypoints need to be mapped to vertices on a template model. Due to the randomness of mesh connections and the lack of 3D human keypoint annotation data, no tool has yet been developed to automatically and accurately annotate 3D keypoints on arbitrary head models. Therefore, an interactive tool has been developed to quickly and manually annotate keypoints on 3D models. Figure 22 This is a schematic diagram illustrating exemplary key point markings on realistic 3D models (such as 2202, 2204) and cartoon-style 3D models (such as 2206, 2208) according to some embodiments of the present disclosure.

[0198] In the tagging process, the positions of 3D keypoints tagged on the 3D model should match the keypoints in the image as closely as possible. Since keypoints are tagged on discrete vertices of the 3D model mesh, introducing bias is unavoidable. To counteract this bias, one approach is to define appropriate rules in the pose processing. Figure 23 This is a schematic diagram illustrating an exemplary comparison between template model rendering, manually labeled keypoints, and AI-detected keypoints according to some embodiments of the present disclosure. In some embodiments, for those producing relatively realistic models, keypoint detection and reconstruction algorithms can be applied to the rendering results of the template model (2302). For example, artificial intelligence can be used to further compare the 3D keypoint results (2306) with the manually labeled results (2304), thereby calculating the deviation between the two sets of keypoints. When detecting images of people, the deviation calculated based on keypoints detected in real-life images is reduced, and the adverse effects of manual labeling are eliminated.

[0199] The affine deformation method disclosed in this application is a keypoint-driven mathematical modeling approach that ultimately solves a system of linear equations. The method disclosed here employs a step to deform the template mesh using detected keypoints as boundary conditions, and utilizes different constraints during the optimization process. Figure 24 This is a diagram illustrating an exemplary affine transformation of a triangle according to some embodiments of the present disclosure.

[0200] In some embodiments, the deformation from the template mesh to the predicted mesh is considered as a combination of affine transformations for each triangle. A affine transformation of a triangle can be defined as a 3×3 matrix T and a translation vector d. Figure 24 As shown, the position of the deformed vertex after the affine transformation is represented by v. i ′=Tv i+d, i∈1...4, where v1, v2, v3 represent each vertex of the triangle, and v4 is an additional point introduced along the triangle's normal direction, satisfying the equation v4=v1+(v2-v1)×(v3-v1) / sqrt( / (v2-v1)×(v3-v1) / ). In this equation, the cross product is normalized and proportional to the side length of the triangle. v4 is introduced because the coordinates of the three vertices are insufficient to determine a unique affine transformation. After introducing v4, the derived equation is: T=[v′2-v′1v′3-v′1v′4-v′1]×[v2-v1 v3-v1 v4-v1] -1 Furthermore, the non-translated portion of matrix T was determined. Since matrix V = [v2-v1 v3-v1 v4-v1] -1 It depends only on the template mesh and does not change due to other deformation factors, so it can be pre-computed as a sparse coefficient matrix for later use in constructing linear systems.

[0201] Thus, the non-translational part of the affine transformation T in the mathematical formula has been expressed. When constructing the optimized linear system, assuming the number of mesh vertices is N and the number of triangles is F, consider the following four constraints:

[0202] Constraints on key point locations: E k =∑ i＝1 ||v′ i -c′ i || 2 , c′ i This represents the location of key points detected after mesh deformation.

[0203] Constraints on adjacency smoothness: E s =∑ i＝1 ∑ j∈adj(i) ||T i -T j || 2 This means that the affine transformations between adjacent triangles should be as similar as possible. Adjacency relationships can be pre-queried and stored to avoid redundant calculations and improve the performance of system construction.

[0204] Constraints on characteristics: E i =∑ i＝1 ||T i -I|| 2 , where I represents the identity matrix. This constraint implies that the affine transformation should be as close to invariant as possible, which helps to preserve the properties of the template mesh.

[0205] Constraints of the original position: E l =∑ i＝1 N||v′ I -c i ||2 , where c i This indicates the position of each vertex on the template mesh before deformation.

[0206] The final constraint is the weighted sum of the above constraints: minE = w k E k +w s E s +w i E i +w l E l , where the weight w k w s w i w l The constraints are arranged from strongest to weakest. Using these constraints, a linear system of size (F+N)×(F+N) can be constructed, where the weights are multiplied by their corresponding coefficients. Besides the additional point v′4 for each triangle, the unknowns are the coordinates of each vertex after deformation. Since the first few terms are useful, the result for v′4 is discarded. During continuous deformation, all constraint matrices, except for those for keypoint locations, can be reused. On ordinary personal computers and smartphones, affine transformations can achieve real-time performance of 30fps for meshes with thousands of vertices.

[0207] Figure 25 This is a schematic diagram illustrating an exemplary comparison of deformation results of some head models with and without a hybrid molding process according to some embodiments of the present disclosure.

[0208] In some embodiments, when deforming the head model of a game character, the region of interest is typically only the face. The top of the head, the back of the head, and the neck should remain unchanged, otherwise mesh penetration between the head and hair or torso will occur. To avoid this problem, a blendshape technique can be used to linearly interpolate the result of the affine deformation and the stencil mesh. The weights used for blending can be drawn in 3D modeling software or calculated using biharmonic or affine deformation with slight modifications. For example, setting the weights on keypoints to 1 while adding more markers ( Figure 25 The dark points (from model 2504) are added to the head model, and their weights are set to 0. In some embodiments, inequality constraints are added during the solution process to force all weights to be in the range of 0 to 1, but this significantly increases the complexity of the solution. Experiments show that good results can be obtained by pruning weights that are less than 0 or greater than 1. Figure 25As shown in 2504, the darkest part of the model has a weight of 1, and the colorless part has a weight of 0. A natural transition is achieved between light and dark keypoints in the blended weight rendering of 2504. When using Blendshape, the back face of the model (as shown in the image)... Figure 25 As shown in 2506, it remains identical to the original model after deformation (e.g., Figure 25 As shown in 2502). If Blendshape is not used, the back side of the deformed model (as shown in 2502). Figure 25 As shown in 2508, it will not remain consistent with the original model (such as...). Figure 25 (As shown in 2502) is the same.

[0209] In some embodiments, affine deformation can achieve different deformation effects by manipulating the weights of constraints, including simulating the results of biharmonic deformation. Figure 26 This is a schematic diagram illustrating an exemplary comparison of affine deformations and biharmonic deformations with different weights according to some embodiments of this disclosure. Figure 26 As shown, smoothness is the adjacent smoothness weight w s With feature weight w i The ratio of the two values ​​is used. Darker points are keypoints, and the shade of color represents the displacement between the deformed vertex position and its original position. In all deformation results, one keypoint remains unchanged, while another keypoint moves to the same position. This indicates that the smoothness of the deformed sphere increases accordingly as the adjacency smoothness weight and the property weight are gradually increased. Furthermore, the results of biharmonic deformation can be matched with those of affine deformation, with smoothness ranging from 10 to 100. This indicates that affine deformation has more degrees of freedom compared to biharmonic deformation.

[0210] Using the workflow described in this application, games can easily integrate the functionality of intelligently generating virtual head characters. For example, Figure 27 The illustrations show some exemplary results, which are based on some embodiments of this disclosure, using realistic template models from a set of randomly selected images of women. Figure 27 (Not shown in the image) Automatically generated. All personalized head-shaped avatars reflect certain characteristics of their corresponding images.

[0211] Figure 28 This is a flowchart 2800 illustrating an exemplary process for generating a 3D deformable head model from a 2D facial image of a real person according to some embodiments of the present disclosure.

[0212] The process of generating a 3D deformable head model from a 2D facial image includes step 2810: receiving a two-dimensional (2D) facial image.

[0213] The process also includes step 2820: identifying the first set of key points in the 2D facial image based on an artificial intelligence (AI) model.

[0214] The process further includes step 2830: mapping the first set of key points to a second set of key points on multiple vertices of the mesh of the 3D head template model, based on a set of key point annotations provided by the user on the 3D head template model.

[0215] The process further includes step 2840: deforming the mesh of the 3D head template model to obtain a deformed 3D head mesh model by reducing the difference between the first set of keypoints and the second set of keypoints. In some embodiments, there is a correspondence between keypoints in the first set and keypoints in the second set. After projecting the second set of keypoints into the same space as the first set of keypoints, a function is generated that measures the positional difference between each keypoint in the first set and the keypoints in the second set. By deforming the mesh of the 3D head template model, the second set of keypoints in space is optimized when the function that measures the positional difference (e.g., position, adjacency smoothness, properties, etc.) between each keypoint in the first set and the keypoints in the second set is minimized.

[0216] The process also includes step 2850: applying a hybrid shaping method to the deformed 3D head mesh model to obtain a personalized head model based on the 2D facial image.

[0217] Additional implementations may include one or more of the following features.

[0218] In some embodiments, the mapping step 2830 may further include: associating a first set of key points on a 2D facial image with multiple vertices on a mesh of a 3D head template model; identifying a second set of key points based on a set of key point annotations provided by the user on multiple vertices of the mesh of the 3D head template model; and mapping the first set of key points and the second set of key points based on the corresponding recognition features of the corresponding key points on the face.

[0219] In some embodiments, a second set of keypoints is located by applying a previously calculated bias to a user-provided set of keypoint annotations. In some embodiments, the previously calculated bias is between a previously AI-identified set of keypoints of the 3D head template model and a previously user-provided set of keypoint annotations on multiple vertices of the 3D head template model's mesh.

[0220] In some embodiments, step 2840 of performing deformation may include: transforming the mesh of the 3D head template model into a deformed 3D head mesh model by using a mapping from a first set of key points to a second set of key points and by using boundary conditions of the deformation associated with the first set of key points.

[0221] In some embodiments, step 2840 of performing deformation may further include: applying different constraints during deformation optimization, including one or more of key point location, adjacency smoothness, characteristics, and original location.

[0222] In some embodiments, step 2840 of performing deformation may further include: applying constraints to the deformation process, the constraints being a weighted sum of one or more of key point locations, adjacency smoothness, properties, and original locations.

[0223] In some embodiments, step 2820 of identifying the first set of key points includes using a convolutional neural network (CNN).

[0224] In some embodiments, the deformation includes affine deformation without a Laplacian operator. In some embodiments, affine deformation is optimized by changing the smoothness parameter.

[0225] In some embodiments, the mesh of the 3D head template model can deform without being integrated with the skeleton. In some embodiments, the facial deformation model includes a realistic style model or a cartoon style model.

[0226] In some embodiments, in step 2850, applying the hybrid shaping method to the deformed 3D head mesh model includes: assigning corresponding hybrid weights to the key points of the deformed 3D head mesh model according to the location of the key points; and applying different levels of deformation to key points with different hybrid weights.

[0227] In some embodiments, in step 2850, applying a hybrid shaping method to the deformed 3D head mesh model includes maintaining the back shape of the deformed 3D head mesh model the same as the original back shape of the 3D head template model before deformation.

[0228] In some embodiments, the semantic elements on the template model are not limited to eyes, eyelashes, or teeth. By adding and tracking new key points on the facial mesh, it is potentially possible to adaptively adjust decorations such as glasses.

[0229] In some embodiments, keypoints on the template model are added manually. In other embodiments, deep learning techniques can also be used to automatically add keypoints for different template models.

[0230] In some embodiments, the solution procedure for affine deformation can utilize numerical techniques to further improve its computational performance.

[0231] In some embodiments, the systems and methods disclosed in this application form a facial virtual character generation system based on lightweight keypoints, which has many advantages, such as those listed below:

[0232] The requirements for the input image are low. This system and method do not require the face to be directly facing the camera, and a certain degree of in-plane rotation, out-of-plane rotation, and occlusion will not have a significant impact on performance.

[0233] Suitable for both real-world and cartoon-style games. This system is not limited to real-world game styles; it can also be applied to cartoon-style games.

[0234] Lightweight and customizable. Each module of this system is relatively lightweight and suitable for mobile devices. The modules are decoupled, allowing users to create the final face generation system using different combinations to suit different game styles.

[0235] In some embodiments, for a given single photograph, the main face is detected first, followed by keypoint detection. In real-world images, faces may not be facing the camera, and real faces are not always perfectly symmetrical. Therefore, keypoints in the original image are preprocessed to obtain a uniform, symmetrical, and smooth set of keypoints. These keypoints are then adjusted according to the game's specific style (such as enlarged eyes and a thinner face). After obtaining the stylized keypoints, they are converted into control parameters for the facial model in the game, typically skeletal or slider parameters.

[0236] In some embodiments, the viewpoint of the real face may not be directly facing the camera, and there may be problems such as left-right asymmetry and key point detection errors. Figure 29 This is a schematic diagram illustrating exemplary keypoint processing steps according to some embodiments of the present disclosure. Keypoints detected from the original image 2904 cannot be used directly and require certain processing. Here, the process is divided into three steps: normalization, symmetry, and smoothing, as follows: Figure 29 As shown.

[0237] In some embodiments, it is necessary to adjust the standard facial model in the game based on predictions of real facial key points. This process requires ensuring that the key points of the standard facial model in the game are aligned with the real face in terms of scale, position, and orientation. Therefore, normalizing the predicted key points and the key points on the game facial model 2906 includes the following components: scale normalization, translation normalization, and angle normalization.

[0238] In some embodiments, all the originally detected 3D facial key points are defined as p, where the i-th key point is p. i ={x i y i , z i For example, the origin for normalization is defined as the midpoint between keypoint 1 and keypoint 17 (see reference). Figure 1(Definition of key points), that is, c = (p1 + p) 17 ) / 2. In terms of scale, the distance between the 1st and 17th keypoints and the origin is adjusted to 1, so the 3D keypoints normalized by scale and translation are p'=(pc) / ||p1-c||.

[0239] In some embodiments, after normalizing the scale and translation, the facial orientation is further normalized. For example... Figure 29 As shown in image 2902, the face in the actual photograph may not be directly facing the camera; there is always some degree of deflection, which may exist along the three coordinate axes. The predicted 3D facial keypoints are rotated sequentially along the x, y, and z coordinate axes so that the face faces the camera. When rotating along the x-axis, the z-coordinates of keypoints 18 and 24 are (see reference). Figure 1 The definition of the midpoint is aligned, that is, the depth of the uppermost part of the bridge of the nose is the same as the depth of the bottom of the nose, to obtain the rotation matrix R. X When rotating along the y-axis, the z-coordinates of keypoints 1 and 17 are aligned, resulting in the rotation matrix R. Y When rotating along the z-axis, the y-coordinates of keypoints 1 and 17 are aligned, resulting in the rotation matrix R. Z In this way, the orientation of the key points is aligned, and the normalized key points are shown below:

[0240] P norm =R Z ×R Y ×R X ×P'

[0241] In some embodiments, although the proportions, positions, and angles of the normalized keypoints have been adjusted to be uniform, the resulting keypoints are often not perfect faces. For example, the bridge of the nose is not a straight line centered, and facial features may not be symmetrical. This is because real faces in photographs are not perfectly symmetrical due to expressions or their inherent characteristics, thus introducing additional errors when predicting keypoints. While real faces may not be symmetrical, asymmetrical facial models in games can result in unattractive appearances and significantly degrade the user experience. Therefore, keypoint symmetry, as shown in 2908, is a necessary process.

[0242] Because the keypoints have been normalized, in some embodiments, a simple symmetry method is to average the y and z coordinates of all left and right symmetrical keypoints to replace the original y and z coordinates. This method works well in most cases, but performance is affected when the face is rotated at a large angle along the y-axis.

[0243] In some embodiments, use Figure 29Using a human face as an example, when the face is turned a large angle to the left, part of the eyebrows will be invisible. Simultaneously, due to perspective, the left eye will appear smaller than the right eye. Although 3D keypoints can partially compensate for the effects of perspective, the 2D projections of the corresponding 3D keypoints still need to be preserved in the image. Therefore, excessive angular deflection can lead to significant differences in the size of the eyes and eyebrows in the 3D keypoint detection results. To address the effects of angle, when the face is turned a large angle along the y-axis, the eye and eyebrow closer to the camera are designated as the primary eye and eyebrow, and copied to the other side to reduce errors caused by angular deflection.

[0244] In some embodiments, since prediction errors of keypoints are unavoidable, symmetrical keypoints may still not match the real face in some individual cases. Because the shape and features of real faces vary greatly, it is difficult to achieve a relatively accurate description using predefined parametric curves. Therefore, when smoothing is performed as shown in 2910, only some regions are smoothed, such as the contours of the face, eyes, eyebrows, lower lip, etc. These regions remain essentially monotonous and smooth, i.e., without jagged edges. In this case, the target curve should always be either a convex or concave curve.

[0245] In some embodiments, the key points of the relevant boundaries are checked one by one to see if they satisfy the definition of a convex curve (or a concave curve). Figure 30 This is a schematic diagram illustrating an exemplary keypoint smoothing process 2910 according to some embodiments of the present disclosure. For example... Figure 30 As shown, without loss of generality, the target curve should be convex. For each keypoint 3002, 3004, 3006, 3008, and 3010, check if its position is above the line connecting its adjacent left and right keypoints. If this condition is met, it means the current keypoint satisfies the convex curve requirement. Otherwise, move the current keypoint upwards to the line connecting the left and right keypoints. For example, in Figure 30 If keypoint 3006 does not meet the constraints of a convex curve, it is moved to position 3012. Moving multiple keypoints may not guarantee that the resulting curve will be either convex or concave. Therefore, in some embodiments, multi-round smoothing is used to obtain a relatively smooth keypoint curve.

[0246] Different games have different facial styles. In some embodiments, it is necessary to transform the key features of a realistic face into the style required by the game. Realistic game faces are largely similar, but cartoon faces vary greatly. Therefore, it is difficult to have a unified standard for stylizing key features. In practice, the definition of stylization comes from the game designers, who adjust facial characteristics according to the specific game style.

[0247] In some embodiments, a more general facial adjustment scheme is implemented that most games might require. For example, facial length adjustment, width adjustment, and facial feature adjustment. Adjustments such as adjustment level and scaling ratio can be customized according to different game art styles. Users can also customize adjustment methods for any special style, such as changing the eye shape to a rectangle. The system supports any adjustment method.

[0248] In some embodiments, a standard game face is deformed using stylized facial key points so that the key points of the deformed face reach the position of the target key points. Since most games use control parameters such as bones or sliders to adjust the face, a set of control parameters is needed to move the key points to the target position.

[0249] Since the definitions of bones or sliders may differ across games and are subject to change, directly defining a simple parameterized function from keypoints to bone parameters is not feasible. In some embodiments, machine learning methods are used to convert keypoints into parameters via a neural network called a K2P (Keypoint to Parameter) network. Because the number of parameters and keypoints is generally small (typically less than 100), K-layer fully connected networks are used in some embodiments.

[0250] Figure 31This is a block diagram illustrating exemplary keypoints for a control parameter (K2P) conversion process according to some embodiments of the present disclosure. To utilize machine learning methods, in some embodiments, skeletal or slider parameters are first randomly sampled, fed to a game client 3110, and keypoints are extracted from the generated game face. This yields a large amount of training data (parameter 3112 and keypoint 3114 pairs). A self-supervised machine learning method is then implemented, comprising two steps: the first step is to train a P2K (parameter-to-keypoint) network 3116, simulating the game parameter-to-keypoint generation process. In the second step, according to the method described in this application, real facial keypoints 3104 are generated using a large number of unlabeled real facial images 3102, and then a large number of stylized keypoints 3106 are generated. These unlabeled stylized keypoints 3106 are self-supervised learning training data. In some embodiments, a set of keypoints K is input into a K2P network 3108 for learning to obtain output parameters P. Since ground truth values ​​for the ideal parameters corresponding to these keypoints are unavailable, P is further input into the P2K network 3116 trained in the first step to obtain keypoints K'. In some embodiments, a K2P network 3108 can be learned by calculating the mean squared error (MSE) loss between K and K'. In some embodiments, the P2K network 3116 is fixed during the second step and is not further adjusted. With the aid of the P2K network 3116, the process of controlling the parameters of the game client 3110 to the keypoints is simulated using a neural network, thus laying the foundation for the learning of the K2P network 3108 in the second step. In this way, the final face generated by the parameters will closely approximate the keypoints of the generated target stylized face.

[0251] In some embodiments, when calculating the MSE loss between K and K', weights are added to certain keypoints, such as eye keypoints, by adjusting the corresponding weights. Since the keypoints are predefined and unaffected by the game client's skeleton or slider, the weights are easier to adjust.

[0252] In some embodiments, to improve model accuracy in practical applications, decoupled parts can be trained separately using neural networks. For example, if some skeletal parameters only affect keypoints in the eye region, while other parameters have no effect on that region, then these parameters and those keypoints constitute a separate set of regions. A separate K2P model 3108 is trained for each such set of regions, and each model can employ a more lightweight network design. This not only further improves model accuracy but also reduces computational complexity.

[0253] Figure 32 The illustrations show some exemplary results of automatically generating mobile game faces according to some embodiments of this disclosure. For example... Figure 32 The illustration shows the results of generating a virtual character face image for the game from the original facial images (3202 and 3206) (3204 and 3208). In some embodiments, during stylization, an open mouth is closed, and the nose, mouth, face shape, eyes, and eyebrows are subject to varying degrees of restriction and cartoonization. The final generated result still retains certain facial characteristics, conforming to the aesthetic requirements of the game style.

[0254] Figure 33 This is a flowchart 3300 illustrating an exemplary process of customizing the standard face of a virtual character in a game using a 2D facial image of a real person, according to some embodiments of the present disclosure.

[0255] The process of customizing the standard face of a virtual character in a game using a 2D facial image of a real person includes step 3310: identifying a set of key points of the subject in the 2D facial image. As mentioned above, the subject can be a real person or a virtual character from the virtual world.

[0256] The process also includes step 3320: transforming the set of main key points into a set of virtual character key points associated with the virtual character in the game.

[0257] The process further includes step 3330: generating a set of facial control parameters for the standard face of the virtual character by applying a keypoint-to-parameter (K2P) neural network model to the set of virtual character keypoints, whereby each facial control parameter is associated with one of a plurality of facial features of the standard face. (As described above...) Figure 31 The K2P network 3108 is a deep learning neural network model that predicts a set of facial control parameters based on a set of input virtual character key points. Because different sets of virtual character key points may correspond to different sets of facial control parameters, when the set of facial control parameters is applied to the standard face of a virtual character, the key points of the adjusted standard face can have a set of key points similar to the set of input virtual character key points.

[0258] The process further includes step 3340: adjusting multiple facial features of the standard face by applying the set of facial control parameters to the standard face.

[0259] Additional implementations may include one or more of the following features.

[0260] In some embodiments, in step 3330, the K2P neural network model is trained by: obtaining multiple training 2D facial images of real people; generating a set of training game style keypoints or training virtual character keypoints for each of the multiple training 2D facial images; submitting the set of training game style keypoints or training virtual character keypoints to the K2P neural network model to obtain a set of facial control parameters; submitting the set of facial control parameters to a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a set of predicted game style keypoints or predicted virtual character keypoints corresponding to the training game style keypoints or training virtual character keypoints; and updating the K2P neural network model by reducing the difference between the set of training game style keypoints or training virtual character keypoints and the corresponding set of predicted game style keypoints or predicted virtual character keypoints. As described above... Figure 31 In contrast to the K2P network 3108, the P2K network 3116 is a deep learning neural network model that predicts a set of virtual character keypoints based on an input set of facial control parameters. Since different sets of facial control parameters may result in different sets of virtual character keypoints, when the two neural network models are considered to perform the inverse process of each other, a set of output virtual character keypoints associated with the P2K network 3116 should match a set of input virtual character keypoints associated with the K2P network 3108.

[0261] In some embodiments, a pre-trained P2K neural network model is configured to: receive a set of control parameters including skeletal parameters or slider parameters associated with a virtual character in the game; and predict a set of game style keypoints for the virtual character in the game based on the set of control parameters.

[0262] In some embodiments, the difference between the set of trained game style keypoints and the corresponding set of predicted game style keypoints is the sum of the mean squared errors between the set of trained game style keypoints and the corresponding set of predicted game style keypoints.

[0263] In some embodiments, the trained K2P and pre-trained P2K neural network models are game-specific.

[0264] In some embodiments, the set of real key points in the 2D facial image corresponds to real facial features in the 2D facial image.

[0265] In some embodiments, the standard face of a virtual character in a game can be customized for different characters in the game based on facial images of different real people.

[0266] In some embodiments, the distorted face of the virtual character is a cartoon-style face of a real person. In some embodiments, the distorted face of the virtual character is a realistic-style face of a real person.

[0267] In some embodiments, in step 3320, transforming the set of real key points into the set of game style key points includes: normalizing the set of real key points to a normalized space; symmetricizing the set of normalized real key points; and adjusting the set of symmetricized real key points according to a predefined style associated with a virtual character in the game.

[0268] In some embodiments, normalizing the set of real key points into a normalized space includes: scaling the set of real key points into the normalized space; and rotating the scaled set of real key points according to their orientation in the 2D facial image.

[0269] In some embodiments, transforming the set of real key points into the set of game-style key points further includes: smoothing the set of symmetrical key points to meet predefined convex or concave curve requirements.

[0270] In some embodiments, the set of symmetrical real key points are adjusted according to a predefined style associated with a virtual character in the game, including one or more of the following: facial length adjustment, facial width adjustment, facial feature adjustment, scaling adjustment, and eye shape adjustment.

[0271] The system and method disclosed in this application can be applied to automatic face generation systems for various games, including realistic and cartoon-style games. The system features an easy-to-integrate interface, enhancing the user experience.

[0272] In some embodiments, the systems and methods disclosed in this application can be used in 3D facial virtual character generation systems for various games, automating the complex manual optimization process and improving the user experience. Users can take selfies or upload existing photos. The system can extract facial features from the photos and then automatically generate control parameters (such as bones or sliders) for the game's face through an AI facial generation system. The game client uses these parameters to generate a facial virtual character, ensuring that the created face possesses the user's facial features.

[0273] In some embodiments, the system can be easily customized for different games, including keypoint definitions, stylization methods, skeleton / slider definitions, and so on. Users can choose to adjust only certain parameters, automatically retrain the model, or add custom control algorithms. In this way, the invention can be easily deployed to different games.

[0274] Further embodiments also include various subsets of the above embodiments that are combined or otherwise rearranged in various other embodiments.

[0275] In this application, an image processing apparatus according to embodiments of the present application is implemented with reference to the accompanying drawings. The image processing apparatus can be implemented in various forms, such as different types of computer devices, such as servers or terminals (e.g., desktop computers, laptop computers, or smartphones). The hardware structure of the image processing apparatus according to embodiments of the present application is further described below. It will be understood that... Figure 34 Only an exemplary structure of the image processing device is shown, not all structures, and it can be implemented as needed. Figure 34 The part or the whole structure shown.

[0276] refer to Figure 34 , Figure 34 This is a schematic diagram of an optional hardware structure of an image processing apparatus according to an embodiment of this application, and in practical applications, it can be applied to servers or various terminals running applications. Figure 34 The image processing apparatus 3400 shown includes at least one processor 3401, a memory 3402, a user interface 3403, and at least one network interface 3404. The components in the image processing apparatus 3400 are coupled together via a bus system 3405. It is understood that the bus 3405 is configured to enable connection and communication between the components. In addition to including a data bus, the bus system 3405 may further include a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 34 All buses are labeled as Bus System 3405.

[0277] The user interface 3403 may include a monitor, keyboard, mouse, trackball, click wheel, keys, buttons, touchpad, or touch screen, etc.

[0278] It is understood that memory 3402 can be volatile memory or non-volatile memory, or it can include both volatile memory and non-volatile memory.

[0279] In embodiments of this application, memory 3402 is configured to store different types of data to support the operation of image processing apparatus 3400. Examples of data include any computer program for performing operations on image processing apparatus 3400, such as executable program 34021 and operating system 34022, and a program for performing the image processing method of embodiments of this application may be included in executable program 34021.

[0280] The image processing method disclosed in the embodiments of this application can be applied to, or executed by, processor 3401. Processor 3401 can be an integrated circuit chip and has signal processing capabilities. In implementation, each step of the image processing method can be accomplished using integrated logic circuits in the hardware of processor 3401 or instructions in software form. The aforementioned processor 3401 can be a general-purpose processor, a digital signal processor (DSP), another programmable logic device, a discrete gate, a transistor logic device, a discrete hardware component, etc. Processor 3401 can implement or execute the methods, steps, and logic block diagrams provided in the embodiments of this application. A general-purpose processor can be a microprocessor, any conventional processor, etc. The steps in the methods provided in the embodiments of this application can be directly executed by a hardware decoding processor, or can be executed by combining hardware modules and software modules in the decoding processor. The software modules can be located in a storage medium. The storage medium is located in memory 3402. Processor 3401 reads information from memory 3402 and executes the steps of the image processing method provided in the embodiments of this application by combining this information with its hardware.

[0281] In some embodiments, image processing and 3D face and head formation can be performed on a set of servers on a network or in the cloud.

[0282] In one or more examples, the described functions can be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, these functions can be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium can include a computer-readable storage medium (as opposed to a tangible medium such as a data storage medium) or a communication medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, a computer-readable medium can generally correspond to (1) a non-volatile tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the implementations described in this application. A computer program product can include a computer-readable medium.

[0283] The terminology used in the description of implementations in this application is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. As used in the description of particular implementations and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the described features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0284] It will also be understood that although the terms "first," "second," etc., may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the implementation, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.

[0285] The description in this application is given for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications, variations, and alternative implementations will be apparent to those skilled in the art from the teachings presented in the foregoing description and the associated drawings. The embodiments were chosen and described in order to best explain the principles of the invention, its practical application, and to enable others skilled in the art to understand various implementations of the invention, and preferably to utilize the basic principles and various implementations with various modifications to suit the particular intended use. Therefore, it should be understood that the scope of the claims is not limited to the specific examples of the disclosed implementations, and that modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. A method for customizing the standard face of a virtual character using a two-dimensional (2D) facial image of a subject, characterized in that, include: Identify a set of key points of a subject in a 2D facial image; Transform the aforementioned set of main key points into a set of virtual character key points associated with the virtual character; By applying a keypoint-to-parameter (K2P) neural network model to the keypoints of the set of virtual characters, a set of facial control parameters for a standard face is generated, each of the facial control parameters being associated with one of a plurality of facial features of the standard face. as well as By applying the set of facial control parameters to the standard face, multiple facial features of the standard face are adjusted; The keypoint-to-parameter (K2P) neural network model is trained in the following manner: Obtain multiple training 2D facial images of the subject; Generate a set of training virtual character key points associated with the virtual character for each of the plurality of training 2D facial images; Each set of training virtual character key points is submitted to the Key Point to Parameter (K2P) neural network model to obtain a set of facial control parameters; The set of facial control parameters is submitted to a pre-trained parameter-to-keypoint (P2K) neural network model to obtain a set of predicted virtual character keypoints corresponding to the set of trained virtual character keypoints; and The keypoint-to-parameter (K2P) neural network model is updated by reducing the difference between the set of trained virtual character keypoints and the corresponding set of predicted virtual character keypoints.

2. The method according to claim 1, characterized in that, The pre-trained parameter-to-keypoint (P2K) neural network model is configured as follows: Receives a set of facial control parameters, including skeletal or slider parameters associated with the virtual character; and Predict a set of key points for the virtual character based on the set of facial control parameters.

3. The method according to claim 2, characterized in that, The difference between the set of trained virtual character key points and the corresponding set of predicted virtual character key points is the sum of the mean square errors between the set of trained virtual character key points and the corresponding set of predicted virtual character key points.

4. The method according to claim 2, characterized in that, The trained keypoint-to-parameter (K2P) neural network model and the pre-trained parameter-to-keypoint (P2K) neural network model are associated with the game.

5. The method according to claim 1, characterized in that, The set of key points of the subject in the 2D facial image corresponds to the facial features of the subject in the 2D facial image.

6. The method according to claim 1, characterized in that, Based on the facial images of different subjects, the standard faces of the virtual characters are customized for different characters in the game.

7. The method according to claim 1, characterized in that, The standard face of the adjusted virtual character is the cartoon-style face of the main character.

8. The method according to claim 1, characterized in that, The adjusted virtual character's standard face is the real-style face of the subject.

9. The method according to claim 1, characterized in that, Transforming the aforementioned set of main key points into a set of virtual character key points includes: Normalize the aforementioned set of key points into a standard space; To symmetricize the normalized set of key points; and The set of symmetrical key points of the main body is adjusted according to a predefined style associated with the virtual character to obtain the set of key points of the virtual character.

10. The method according to claim 9, characterized in that, The process of normalizing the set of key points to a standardized space includes: Scaling the set of key points of the main body into the normative space; and Based on the orientation of the set of key points in the 2D facial image, the set of key points is rotated and scaled.

11. The method according to claim 9, characterized in that, The step of transforming the set of main body key points into the set of virtual character key points further includes: smoothing the symmetrical set of main body key points to meet the predefined requirements of convex curves or concave curves.

12. The method according to claim 9, characterized in that, Based on a predefined style associated with the virtual character, the symmetrical set of key body points is adjusted, including one or more of the following: facial length adjustment, facial width adjustment, facial feature adjustment, scaling adjustment, and eye shape adjustment.

13. An electronic device, characterized in that, The device includes one or more processing units, a memory coupled to the one or more processing units, and a plurality of programs stored in the memory, which, when executed by the one or more processing units, cause the electronic device to perform the method as described in any one of claims 1 to 12.

14. A non-volatile computer-readable storage medium for storing multiple programs, characterized in that, The plurality of programs are executed by an electronic device having one or more processing units, wherein, when executed by the one or more processing units, the plurality of programs cause the electronic device to perform the method as described in any one of claims 1 to 12.

15. A computer program product comprising instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 12.

Images