Face image cartoon processing method and device, computer device and storage medium

Abstract

The application relates to a face image cartoon processing method and device, computer equipment and a storage medium. The method comprises the following steps: extracting face feature points and image features from a real face image; constructing a three-dimensional simulation face model corresponding to the real face image based on the face feature points; obtaining difference features between the simulation face model and a three-dimensional simulation face template; migrating the difference features to a three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulation face template and the three-dimensional cartoon face template, to obtain a three-dimensional cartoon face model; and performing image rendering on the cartoon face model according to the image features, to generate a three-dimensional cartoon image. The method can effectively construct a three-dimensional cartoon image with high precision and real face features and image features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and the technical field of image processing, in particular to a face image cartoonization processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the rapid development of computer technology and artificial intelligence, various image processing technologies based on computer vision are emerging, such as three-dimensional reconstruction of information in a two-dimensional image, such as generating a three-dimensional cartoon face. In related technologies, the features of each facial feature part are usually extracted, and then cartoon face materials similar to the facial feature parts are matched in a three-dimensional cartoon face database, and then the cartoon face materials are scaled and subjected to three-dimensional cartoon face reconstruction and interpolation operations to generate a three-dimensional cartoon face.

[0003] However, the current method needs to disassemble the facial features of the three-dimensional cartoon face one by one, which is low in processing efficiency, and the generated three-dimensional cartoon face is low in precision and has a large difference in similarity with a real face. SUMMARY

[0004] Therefore, it is necessary to provide a face image cartoonization processing method, device, computer equipment and storage medium capable of effectively generating a three-dimensional cartoon face similar to a real user face.

[0005] A face image cartoonization processing method, the method comprising:

[0006] extracting facial feature points and image features from a real face image;

[0007] constructing a three-dimensional simulation face model corresponding to the real face image based on the facial feature points;

[0008] obtaining difference features between the simulation face model and a three-dimensional simulation face template;

[0009] transferring the difference features to a three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulation face template and the three-dimensional cartoon face template to obtain a three-dimensional cartoon face model;

[0010] rendering the cartoon face model according to the image features to generate a three-dimensional cartoon image.

[0011] A face image cartoonization processing device, the device comprising:

[0012] a feature extraction module configured to extract facial feature points and image features from a real face image;

[0013] a three-dimensional face reconstruction module configured to construct a three-dimensional simulation face model corresponding to the real face image based on the face feature points;

[0014] a difference feature extraction module configured to obtain difference features between the simulation face model and a three-dimensional simulation face template;

[0015] a difference feature migration module configured to migrate the difference features to a three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulation face template and the three-dimensional cartoon face template, to obtain a three-dimensional cartoon face model;

[0016] a three-dimensional cartoon image generation module configured to perform image rendering on the cartoon face model according to the image features, to generate a three-dimensional cartoon image.

[0017] In an embodiment, the face feature points are two-dimensional face feature points; the three-dimensional face reconstruction module is further configured to obtain a feature point mapping matrix of the two-dimensional face feature points in a three-dimensional simulation face template; perform parameter estimation based on the face feature points and the feature point mapping matrix, to obtain three-dimensional face parameters; and construct a three-dimensional simulation face model corresponding to the real face image based on the three-dimensional face parameters.

[0018] In an embodiment, the three-dimensional face reconstruction module is further configured to perform camera parameter iterative estimation based on the face feature points and the feature point mapping matrix, to obtain camera parameters after a first iteration condition is met; perform face parameter iterative estimation based on the face feature points, the feature point mapping matrix and the camera parameters, to obtain face shape basis parameters and face expression basis parameters after a second iteration condition is met; and construct a three-dimensional simulation face model corresponding to the real face image according to the camera parameters, the face shape basis parameters and the face expression basis parameters.

[0019] In an embodiment, the simulation face model includes a plurality of triangular faces in a topological structure; the difference feature extraction module is further configured to obtain a first deformation gradient on a triangular face corresponding to each triangular face in the simulation face model; and obtain an affine transformation mapping matrix between each triangular face in the simulation face model and the corresponding triangular face in the three-dimensional simulation face template according to the first deformation gradient; the affine transformation mapping matrix is used to represent the difference features between the simulation face model and the three-dimensional simulation face template.

[0020] In an embodiment, the topology of the three-dimensional cartoon face template includes a plurality of triangular faces; the difference feature migration module is further configured to align the simulated face model with the three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulated face template and the three-dimensional cartoon face template; find, in each triangular face of the aligned three-dimensional cartoon face template, a triangular face that matches each triangular face of the simulated face model, to obtain a triangular face mapping relationship between each triangular face of the simulated face model and the triangular face of the three-dimensional cartoon face template; and migrate the difference feature to the three-dimensional cartoon face template according to the triangular face mapping relationship, to obtain a three-dimensional cartoon face model.

[0021] In an embodiment, the difference feature migration module is further configured to deform, according to the triangular face mapping relationship and the affine transformation mapping matrix, an original triangular face in the three-dimensional cartoon face template to determine a second deformation gradient between the original triangular face and a deformed triangular face in the three-dimensional cartoon face template; iteratively deform, according to the triangular face mapping relationship and the affine transformation mapping matrix, each triangular face in the aligned three-dimensional cartoon face template in a direction that minimizes a difference between the second deformation gradient and a first deformation gradient; and obtain a three-dimensional cartoon face model after an iteration stop condition is met.

[0022] In an embodiment, the feature extraction module is further configured to extract, from a real face image, facial feature points, contour key points, and respective semantic information; obtain face feature points according to the facial feature points and the contour key points and respective semantic information; and extract an image feature of the real face image based on the face feature points.

[0023] In an embodiment, the image feature includes a native image feature and an additional image feature; the feature extraction module is further configured to perform face alignment processing on the real face image to obtain an aligned real face image; identify the native image feature from the aligned real face image based on the face feature points; and identify the additional image feature from the aligned real face image based on the face feature points.

[0024] In an embodiment, the native image feature includes a hairstyle feature; the feature extraction module is further configured to extract a hair feature from the aligned real face image by using a trained hairstyle recognition network, and generate a hair mask map according to the hair feature; divide the hair mask map into at least two mask map sub-regions based on the face feature points; and obtain a hairstyle feature according to a distribution of the hair feature in the at least two mask map sub-regions.

[0025] In an embodiment, the feature extraction module is further configured to extract an additional image region from the aligned real face image according to the distribution of the face feature points; identify an additional image category of the additional image region by using a trained target classification network; and obtain an additional image feature in the real face image according to the additional image category.

[0026] In an embodiment, the image features include a skin surface color feature, a native image feature, and an additional image feature; the three-dimensional cartoon image generation module is further configured to extract pixels of a skin surface color region based on the distribution of the face feature points, obtain a skin surface color feature according to the pixels of the skin surface color region, perform skin surface color rendering on the three-dimensional cartoon face model according to the skin surface color feature, obtain a matching image material according to the native image feature and the additional image feature, and render the image material to the three-dimensional cartoon face model after the skin surface color rendering to obtain a three-dimensional cartoon image with face features and image features.

[0027] In an embodiment, the device further includes a display module configured to display a cartoon image selection interface including a three-dimensional image option, capture a real face image of a user in response to a selection operation on the three-dimensional image option, and display an image preview interface and display the real face image and the three-dimensional cartoon image in the image preview interface.

[0028] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0029] extracting face feature points and image features from a real face image;

[0030] constructing a three-dimensional simulation face model corresponding to the real face image based on the face feature points;

[0031] obtaining a difference feature between the simulation face model and a three-dimensional simulation face template;

[0032] migrating the difference feature to a three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulation face template and the three-dimensional cartoon face template to obtain a three-dimensional cartoon face model;

[0033] performing image rendering on the cartoon face model according to the image features to generate a three-dimensional cartoon image.

[0034] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0035] extracting facial feature points and image features from a real face image;

[0036] constructing a three-dimensional simulation face model corresponding to the real face image based on the facial feature points;

[0037] obtaining difference features between the simulation face model and a three-dimensional simulation face template;

[0038] migrating the difference features to the three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulation face template and the three-dimensional cartoon face template, to obtain a three-dimensional cartoon face model;

[0039] image rendering on the cartoon face model according to the image features, to generate a three-dimensional cartoon image.

[0040] A computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium; the processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor implements the following steps when executing the computer instructions:

[0041] extracting facial feature points and image features from a real face image;

[0042] constructing a three-dimensional simulation face model corresponding to the real face image based on the facial feature points;

[0043] obtaining difference features between the simulation face model and a three-dimensional simulation face template;

[0044] migrating the difference features to the three-dimensional cartoon face template based on a semantic mapping relationship between the three-dimensional simulation face template and the three-dimensional cartoon face template, to obtain a three-dimensional cartoon face model;

[0045] image rendering on the cartoon face model according to the image features, to generate a three-dimensional cartoon image.

[0046] The aforementioned method, apparatus, computer equipment, and storage medium for cartoonizing facial images extract facial feature points and image features from real facial images, and then construct a 3D simulated facial model corresponding to the real facial image based on the facial feature points. Next, by acquiring the difference features between the simulated facial model and the 3D simulated facial template, and considering the pre-defined semantic mapping relationship between the 3D simulated facial template and the 3D cartoon facial template, the 3D cartoon facial template is deformed and transferred based on the difference features according to the semantic mapping relationship. This allows for accurate and effective transfer of the difference features between the simulated facial model and the 3D simulated facial template to the 3D cartoon facial template, resulting in a 3D cartoon facial model with both real facial features from the real facial image and cartoonized features. Further image rendering of the cartoon facial model based on image features renders the image features from the real facial image into the cartoon facial model, thereby generating a 3D cartoon image that is even more similar to the real facial image. Attached Figure Description

[0047] Figure 1 This is an application environment diagram of a face image cartoonization processing method in one embodiment;

[0048] Figure 2 This is a flowchart illustrating a method for cartoonizing facial images in one embodiment;

[0049] Figure 3 This is a schematic diagram of a real human face image and the corresponding constructed 3D simulated human face model in one embodiment;

[0050] Figure 4 This is a schematic diagram of the topological structure of a 3D simulated human face template in one embodiment;

[0051] Figure 5 This is a schematic diagram of a 3D simulated face template and a corresponding 3D simulated face model in one embodiment;

[0052] Figure 6 This is a schematic diagram illustrating the alignment process between a simulated human face model and a 3D cartoon face template in one embodiment.

[0053] Figure 7 This is a schematic diagram illustrating the transfer of differential features to a 3D cartoon face template in one embodiment;

[0054] Figure 8 This is a schematic diagram illustrating the extraction of facial feature points from a real face image in one embodiment;

[0055] Figure 9 This is a schematic diagram illustrating face alignment processing of a real face image in one embodiment;

[0056] Figure 10A schematic diagram of an image after face feature point extraction and face alignment processing and face region division on a real face image in an embodiment;

[0057] Figure 11 A schematic diagram of a hair mask image in an embodiment;

[0058] Figure 12 A schematic diagram of a process of classifying glasses features in an embodiment;

[0059] Figure 13 A schematic diagram of a skin surface color chart in an embodiment;

[0060] Figure 14 A schematic diagram of a three-dimensional cartoon image after three-dimensional cartoon reconstruction on a real face image in an embodiment;

[0061] Figure 15 A schematic diagram of three groups of real face images and corresponding three-dimensional cartoon images in an embodiment;

[0062] Figure 16 A schematic diagram of a cartoon image selection interface in an embodiment;

[0063] Figure 17 A schematic diagram of an image shooting interface in an embodiment;

[0064] Figure 18 A schematic diagram of a preview interface of a display image in an embodiment;

[0065] Figure 19 A schematic diagram of a three-dimensional cartoon image adjustment interface in an embodiment;

[0066] Figure 20 A structural block diagram of a face image cartoon processing device in an embodiment;

[0067] Figure 21 An internal structure diagram of a computer device in an embodiment;

[0068] Figure 22 An internal structure diagram of a computer device in another embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0070] The face image cartoonization method provided in this application can be applied to computer devices. The computer device can be a terminal or a server. It is understood that the face image cartoonization method provided in this application can be applied to a terminal, a server, or a system including both a terminal and a server, and is implemented through interaction between the terminal and the server.

[0071] The face image cartoonization method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions.

[0072] Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established, deploying various types of virtual resources within the resource pool for external customers to choose from.

[0073] Specifically, terminal 102 acquires or collects real face images and uploads them to server 104. Server 104 extracts facial feature points and image features from the real face images, constructs a 3D simulated face model corresponding to the real face image based on the facial feature points, and obtains the difference features between the simulated face model and the 3D simulated face template. Server 104 further transfers the difference features to the 3D cartoon face template based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, thereby effectively obtaining a 3D cartoon face model; then, based on the image features, it performs image rendering on the cartoon face model, thereby generating a 3D cartoon image similar to a real face, and outputting the 3D cartoon image.

[0074] It is understood that the face image cartoonization processing method in the various embodiments of this application employs computer vision technology and machine learning technology from artificial intelligence, which can effectively achieve the automatic generation of three-dimensional cartoon images similar to real human faces. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making.

[0075] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision researches related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. This application uses computer vision technology to cartoonize real human face images to generate 3D cartoon characters that resemble real faces.

[0076] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formulaic learning. It can be understood that the face feature point detection network, object classification network, and hairstyle recognition network used in some embodiments of this application are trained using machine learning techniques. Based on these machine learning techniques, they can more accurately extract feature points, classify objects, and recognize hairstyles from real face images.

[0077] In one embodiment, such as Figure 2 As shown, a method for cartoonizing facial images is provided. The method is illustrated using a computer device as an example. Specifically, the computer device can be...Figure 1 The method can also be applied to systems that include both terminals and servers, and is implemented through the interaction between the terminals and servers. This embodiment includes the following steps:

[0078] S202, extract facial feature points and image features from real face images.

[0079] Among them, real face images are images of faces captured in real-world scenarios. They are two-dimensional face images that include the user's face. Real face images can be frontal photos taken in real time by a camera, or face images that include a face obtained from local storage or the internet.

[0080] The real face image in this embodiment can be a two-dimensional face image with any pose and expression.

[0081] Facial feature points are understood to be features extracted from images containing faces, representing facial information through multiple key points. "Multiple" refers to two or more key points. Real-world facial images include at least one facial feature point among eyebrows, eyes, nose, lips, and chin. In one embodiment, facial feature points may be primarily distributed at at least one facial region, such as the brow bone, bridge of the nose, eyes, lips, and jawline.

[0082] In one embodiment, facial feature points may include facial key points and contour key points. These key points also carry corresponding semantic information. For example, the semantic information may include the location of the key points, the geometric features between them, and other information. The geometric features between the key points include at least one of the following: distance, area, and angle between the key points. The facial key points, contour key points, and corresponding semantic information can reflect the contour and expression of a face.

[0083] Among them, image features refer to the image-related features included in a real human face image.

[0084] In one embodiment, image features may include native image features and additional image features. Native image features refer to the inherent features of a real face in a real face image. For example, native image features include at least one of the following: hairstyle features, skin color features, birthmark features, mole features, etc. Additional image features refer to additional image-related features in a real face image that are not inherent to the user. For example, additional features worn by the user in a real face image, specifically additional features included in the face region and the surrounding area. For example, additional image features include accessory features, such as features of at least one accessory, such as glasses, earrings, etc.

[0085] After acquiring a real face image, the computer device extracts facial feature points from the image. Specifically, a pre-trained face detection network or a preset facial feature point detection algorithm can be used to extract facial feature points, and then the image features in the real face image can be extracted based on these feature points.

[0086] In one embodiment, since the acquired original real face image may contain various noises and random interferences, image preprocessing such as grayscale correction and noise filtering can be performed on the acquired real face image. Specifically, the computer device first detects the face region in the real face image, and preprocesses the real face image based on the face region detection results. For the real face image, the preprocessing process may include at least one of the following: lighting compensation, grayscale transformation, histogram equalization, normalization, geometric correction, filtering, and sharpening. After preprocessing the real face image, facial feature points and image features are further extracted from the preprocessed real face image.

[0087] In one embodiment, a pre-trained deep neural network can be used to extract facial features from real-world facial images. Specifically, the pre-trained deep neural network may include sub-networks for different facial features to extract different features separately. In another embodiment, different deep neural networks may be used to extract different facial features separately.

[0088] The method in the various embodiments of this application is to cartoonize a real human face image to reconstruct it in three dimensions, thereby generating a three-dimensional cartoon image, so that the generated three-dimensional cartoon image has facial features and image features similar to a real human face.

[0089] S204, constructs a three-dimensional simulated face model corresponding to a real face image based on facial feature points.

[0090] In this context, a real human face image is a two-dimensional image, meaning a planar image that does not contain depth information. Three-dimensional (3D) refers to a spatial system created by adding a direction vector to a two-dimensional planar system. Therefore, a three-dimensional simulated human face model can be understood as a three-dimensional facial geometry structure. Specifically, it can be a three-dimensional simulated human face model constructed by modeling facial feature points from a two-dimensional real human face image, thus possessing the real facial features found in the real image.

[0091] After extracting facial feature points and image features from a real face image, the computer device first establishes a mapping relationship between the extracted 2D and 3D feature points of the face. Specifically, the computer device can match and map the 2D coordinate information of the facial feature points with key feature points in a preset 3D simulated face template to obtain the corresponding 3D coordinate information. Then, it corrects the facial posture based on the facial orientation to obtain facial features with normalized expression. Finally, based on the 3D deformed and corrected facial features, it generates a 3D simulated face model corresponding to the real face image.

[0092] Among them, the 3D simulated face template is a 3D average face model obtained by averaging the facial features corresponding to a large amount of face data. For example, a 3D Morphable Face Model (3DMM) can be used for face reconstruction. Face reconstruction here refers to reconstructing a 3D face model corresponding to a 2D real face image. 3DMM is a general-purpose 3D face model that uses a fixed number of pixels to represent a face and can be used to construct a 3D face shape from a 2D face image.

[0093] S206, Obtain the difference features between the simulated human face model and the 3D simulated human face template.

[0094] It is understandable that difference features refer to the differences between facial features, that is, the changes in a real face image relative to the average face template.

[0095] Specifically, after the computer device constructs a three-dimensional simulated face model corresponding to the real face image based on two-dimensional facial feature points, it compares the simulated face model with the average three-dimensional simulated face template, extracts the variation features of the simulated face model relative to the average three-dimensional simulated face template, and thus extracts the difference features between the simulated face model and the three-dimensional simulated face template.

[0096] Among them, the difference features can be extracted by proportional extraction, that is, by extracting the proportional mapping relationship between the simulated face model and the 3D simulated face template. This proportional mapping relationship reflects the changes in the shape of the simulated face model. Therefore, by extracting the proportional mapping relationship, the difference features between the simulated face model and the 3D simulated face template can be effectively extracted.

[0097] S208, based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, transfers the differential features to the 3D cartoon face template to obtain a 3D cartoon face model.

[0098] Among them, the 3D cartoon face template refers to a preset general cartoon face model, which can be an average 3D cartoon face model obtained by averaging the cartoon faces corresponding to a large number of simulated face models.

[0099] It is understandable that 3D simulated face models and templates, as well as 3D cartoon face models and templates, are all 3D face models with topological structures. These topological structures include information such as the number of vertices, their order, and the connections between them. While 3D simulated face models and templates share a consistent topological structure, 3D cartoon face models and templates possess different topological structures.

[0100] The semantic mapping relationship refers to the mapping relationship between a preset 3D simulated face template and a 3D cartoon face template. In one embodiment, the semantic mapping relationship can be the mapping relationship between the triangular faces in the topological structures corresponding to the 3D simulated face template and the 3D cartoon face template, respectively. In another embodiment, the semantic mapping relationship can also be the mapping relationship between facial key points in the 3D simulated face template and facial key points in the 3D cartoon face template.

[0101] After extracting the difference features between the simulated human face model and the 3D simulated human face template, the computer equipment transfers these difference features to the 3D cartoon human face template based on the semantic mapping relationship between the two templates. Specifically, the computer equipment first aligns the 3D simulated human face template and the 3D cartoon human face template according to the semantic mapping relationship between their topological structures. This means that the computer equipment transforms the two templates to be semantically aligned, making them identical.

[0102] Since the 3D simulated face model and the 3D simulated face template have the same topological structure, the computer device then transfers the differences between the 3D simulated face model and the 3D cartoon face template based on the topological structure of the aligned 3D simulated face template to the 3D cartoon face template, thereby effectively generating a 3D cartoon face model that has both cartoon image and real facial features.

[0103] S210: Render the cartoon face model based on its features to generate a 3D cartoon image.

[0104] The generated cartoon face model possesses the features of a real human face image.

[0105] It is understandable that a 3D cartoon character is a non-realistic cartoon image that has both cartoonish features and realistic human facial features.

[0106] Since the generated cartoon face model has the features of a real face image, the computer equipment further renders the cartoon face model according to the image features, so as to render the image features in the real face image into the cartoon face model, thereby generating a three-dimensional cartoon image that is more similar to the real face in the real face image.

[0107] In the aforementioned method for cartoonizing facial images, the computer device extracts facial feature points and image features from a real facial image, and then constructs a 3D simulated facial model corresponding to the real facial image based on the facial feature points. Next, by acquiring the difference features between the simulated facial model and the 3D simulated facial template, and considering the pre-defined semantic mapping relationship between the two, the 3D cartoon facial template is deformed and transferred based on the difference features according to this semantic mapping relationship. This allows for accurate and effective transfer of the difference features between the simulated facial model and the 3D simulated facial template to the 3D cartoon facial template, resulting in a 3D cartoon facial model with both real facial features from the real facial image and cartoonized features. Further image rendering of the cartoon facial model based on image features renders the image features from the real facial image into the cartoon facial model, thereby generating a 3D cartoon image that is even more similar to the real face in the real facial image.

[0108] In one embodiment, the facial feature points are two-dimensional facial feature points; constructing a three-dimensional simulated face model corresponding to a real face image based on the facial feature points includes: obtaining the feature point mapping matrix of the two-dimensional facial feature points in the three-dimensional simulated face template; performing parameter estimation based on the facial feature points and the feature point mapping matrix to obtain three-dimensional face parameters; and constructing a three-dimensional simulated face model corresponding to a real face image based on the three-dimensional face parameters.

[0109] Among them, 3D face parameters refer to the face parameters that need to be solved in the process of constructing a 3D simulated face model corresponding to a real face image. For example, 3D face parameters may include basic parameters of face shape and basic parameters of face expression.

[0110] A 3D simulated face template is a three-dimensional facial geometric structure, specifically a parametric model that uses various parameters to represent the geometric structure of a face. The feature point mapping matrix refers to the mapping relationship between two-dimensional and three-dimensional facial feature points, that is, the 2D-3D feature point mapping matrix.

[0111] After extracting two-dimensional facial feature points from a real face image, the computer device uses these two-dimensional facial feature points to perform three-dimensional face reconstruction. Specifically, the computer device first obtains the feature point mapping matrix of the two-dimensional facial feature points in a three-dimensional simulated face template to establish the mapping relationship between the two-dimensional and three-dimensional facial feature points. The feature point mapping matrix is ​​used to represent the mapping relationship between the two-dimensional and three-dimensional facial feature points.

[0112] Understandably, because a face contains a lot of prior information, it can be constructed by linearly combining a series of parameters. Therefore, a 3D simulated face model can be built through parameter estimation. These parameters include camera parameters and 3D face parameters, among others.

[0113] The computer then performs parameter estimation based on facial feature points and the feature point mapping matrix. Specifically, through parameter estimation, the final 3D facial parameters are obtained. Based on these estimated 3D facial parameters, the computer then constructs a 3D simulated facial model corresponding to the real face in the real facial image.

[0114] In one embodiment, the computer device may first estimate camera parameters based on facial feature points and a feature point mapping matrix, and then perform parameter estimation based on the facial feature points, the feature point mapping matrix, and the camera parameters to obtain the final 3D facial parameters. Then, based on the estimated camera parameters and 3D facial parameters, a 3D simulated facial model corresponding to the real face in a real facial image is constructed.

[0115] In this implementation, parameter estimation is performed based on two-dimensional facial feature points and the feature point mapping matrix of the two-dimensional facial feature points in the three-dimensional simulated facial template. Then, the obtained three-dimensional facial parameters are used to reconstruct the three-dimensional face, thereby accurately constructing a three-dimensional simulated facial model with real facial features.

[0116] In one embodiment, the step of estimating parameters based on facial feature points and a feature point mapping matrix to obtain three-dimensional facial parameters includes: performing iterative estimation of camera parameters based on facial feature points and a feature point mapping matrix, and obtaining camera parameters after satisfying a first iteration condition; performing iterative estimation of facial parameters based on facial feature points, a feature point mapping matrix, and camera parameters, and obtaining facial shape basis parameters and facial expression basis parameters after satisfying a second iteration condition.

[0117] The steps of constructing a three-dimensional simulated face model corresponding to a real face image based on three-dimensional face parameters include: constructing a three-dimensional simulated face model corresponding to a real face image based on camera parameters, face shape basis parameters, and face expression basis parameters.

[0118] It is understandable that, similar to a 3D simulated face template, a 3D simulated face model is a 3D face model with a topological structure, and both share the same topological structure. The 3D face parameters include face shape basis parameters and face expression basis parameters. The face shape basis parameters control the appearance of the generated face, while the face expression basis parameters control the expression of the generated face.

[0119] Iteration is an activity involving repeated feedback processes, typically aimed at approximating a desired goal or result. In computer technology, it can refer to repeatedly executing a program or instruction using computer equipment—that is, repeatedly executing the loop steps in a program until a certain condition is met. This process is called iteration. Each repetition of the process is called an "iteration," and the result of each iteration serves as the initial value for the next iteration.

[0120] Iterative estimation is a process that uses a preset algorithm or formula, such as solving a system of equations or finding the eigenvalues ​​of a matrix, to repeatedly execute a series of computational steps, sequentially deriving subsequent quantities from preceding quantities. Each result in this process is obtained by performing the same computational steps on the previous result. In this embodiment, iterative estimation is used to solve for the camera parameters, face shape basis parameters, and face expression basis parameters in the 3D face parameters.

[0121] When performing parameter estimation, computer equipment can employ a step-by-step iterative estimation method to estimate camera parameters and 3D face parameters separately. Step-by-step iterative estimation can be understood as dividing the iterative estimation process into different stages of iterative estimation steps, each solving for the required parameter values ​​separately. For example, step-by-step iterative estimation may include a first-stage iterative estimation and a second-stage iterative estimation.

[0122] It is understandable that the computer device can first estimate the camera parameters through a first-stage iterative estimation. Then, through a second-stage iterative estimation, it can estimate the basic parameters of the face shape and the basic parameters of the face expression.

[0123] Specifically, the computer device first performs a first-stage iterative estimation based on facial feature points and the feature point mapping matrix to iteratively estimate camera parameters. After satisfying the first iteration condition, the camera parameters are obtained. The first iteration condition refers to the condition for stopping the iterative estimation of camera parameters. For example, the first iteration condition could be reaching a preset number of iterations, or the convergence value of the camera parameters reaching a preset convergence threshold, etc.

[0124] After obtaining the camera parameters, the computer device further iterates the facial parameters using facial feature points, the feature point mapping matrix, and the camera parameters in a second stage. Upon satisfying the second iteration condition, the facial shape basis parameters and facial expression basis parameters are obtained. The second iteration condition refers to the condition for stopping the iterative estimation of facial parameters. For example, the second iteration condition could be reaching a preset number of iterations, or the convergence values ​​of the facial shape basis parameters and facial expression basis parameters reaching a preset convergence threshold.

[0125] The computer then constructs a three-dimensional simulated face model corresponding to the real face image based on the estimated camera parameters, face shape base parameters, and face expression base parameters.

[0126] In one embodiment, the objective function corresponding to the 3D face model can be expressed as follows:

[0127]

[0128]

[0129] Where M represents the 3D simulated human face template, i.e., the average face. S, R, T represent camera parameters, where S represents the scaling factor, R represents the rotation matrix, and T represents the translation factor, i.e., the displacement vector. id represents the shape basis, exp represents the expression basis, K represents the number of facial feature points, k represents the k-th facial feature point, and A id Represents the shape base, α id A represents the shape basis coefficient. exp Represents the base of expression, α exp L represents the expression basis coefficient. k denoted as 2D facial feature points, p represents the coefficients of 3DMM, and λ represents the regularization factor used to avoid overfitting.

[0130] The quantities that need to be solved are camera parameters, facial shape basis parameters, and facial expression basis parameters. The objective function is a nonlinear equation, which can be solved jointly using nonlinear optimization. For example, the parameters can be iterated using methods such as calculating the Jacobian matrix of the cost function or the Gauss-Newton method.

[0131] To ensure the real-time performance of the algorithm, a step-by-step linear solution approach is adopted. The joint solution process is decomposed into a step-by-step solution process for three sets of parameters: camera parameters, face shape basis parameters, and face expression basis parameters. When solving for one set of parameters, the other sets of parameters are set as constants. This way, solving each set of parameters becomes a linear problem, allowing for direct analytical results. The specific parameter estimation steps are as follows:

[0132] (1) Solving for camera parameters: Set the shape and expression basis parameters to constants. For example, set the initial values ​​to zero. Here, the camera model is simplified to a weak perspective projection model, which is suitable for situations where the object's depth is very small relative to its distance from the camera. The camera model only has scale, translation, and rotation components. The golden ratio method is used for calculation, constructing a 2x8 equation from each 2D-3D pair: Here, X... i This represents the homogeneous coordinates of a 3D point, x i P represents the non-homogeneous coordinates of a 2D point. 1 and P 2 This represents the first two rows of the camera projection matrix (3x4):

[0133]

[0134] n facial feature points can construct a system of 2n x 8 equations. By solving this system of equations, the projection matrix P can be obtained, which is the projection matrix of the 3D facial feature points onto the 2D facial feature points. Then, the S, R, and T parameters of the camera parameters can be decomposed from it.

[0135] (2) Solving for the face shape basis coefficients: Fix the camera parameters and expression basis parameters, for example, by setting them to constants. The camera parameters have already been obtained in the previous step, and the expression basis parameters are initialized to zero. The objective function is then transformed into a capability function, the expression of which is as follows:

[0136]

[0137] Here let α id =x, then p=α id =x, and S,R,T,A id, A exp ,α exp ,L k Since both are known quantities, we have:

[0138] ,

[0139] Therefore, the energy function can be expressed as:

[0140]

[0141] By adding a diagonal matrix Ω to Ax+b for weighting, the following energy function can be obtained:

[0142]

[0143] Here, x represents the expression coefficient to be solved, A represents the result after relevant transformations based on the shape basis, average face, and camera matrix, b represents the assignment of 2D human feature points, Ω represents the weighting factor, and λ represents the regularization factor. The regularization term is used to avoid overfitting. Specifically, the analytical result can be obtained using linear least squares.

[0144] (3) Solving for facial expression basis parameters: Set the camera parameters and facial shape basis parameters to constants, and solve them in a similar way to solving for the shape basis. The expression can be as follows:

[0145]

[0146] The facial expression basis parameters can be obtained by solving the energy function obtained in step (3).

[0147] By iteratively executing steps (1)-(3), after satisfying the iteration conditions, the camera parameters, face shape base parameters, and face expression base parameters can be obtained.

[0148] In the process of iteratively executing steps (1)-(3), the camera parameters can be estimated iteratively in the first iteration stage, and the camera parameters can be obtained after the first iteration condition is met.

[0149] Then, in the second iteration stage, based on the facial feature points, the feature point mapping matrix, and the obtained camera parameters, the facial parameters are iteratively estimated. After satisfying the second iteration condition, the facial shape basis parameters and facial expression basis parameters are obtained.

[0150] In another embodiment, the facial shape basis parameters and facial expression basis parameters can also be estimated in different iterative stages. Specifically, in the first iterative stage, camera parameters are iteratively estimated, and the camera parameters are obtained after satisfying the first iterative condition. Then, in the second iterative stage, facial parameters are iteratively estimated based on facial feature points, the feature point mapping matrix, and the obtained camera parameters, and the facial shape basis parameters are obtained after satisfying the second iterative condition. Further, in the third iterative stage, facial parameters are iteratively estimated based on facial feature points, the feature point mapping matrix, the obtained camera parameters, and the facial shape basis parameters, and the facial expression basis parameters are obtained after satisfying the third iterative condition.

[0151] In this embodiment, since camera parameters have the greatest impact on the overall parameter estimation result, face shape basis parameters affect the global face, and face expression basis parameters affect the face sub-regions, parameter estimation is performed in the step-by-step linear solution process according to the order of camera parameters, face shape basis parameters, and face expression basis parameters. That is, the parameters are updated step by step according to their impact on the final result from largest to smallest, thereby obtaining a more accurate and realistic 3D simulated face model.

[0152] like Figure 3 The image shown is a schematic diagram of a real human face image and the corresponding constructed 3D simulated human face model in one embodiment. Figure 3 (a) is the image after extracting facial feature points from a real face image. Figure 3 (b) shows a constructed 3D simulated face model with real facial features from a real face image.

[0153] In another embodiment, artificial intelligence-based methods can be used to determine the basic parameters of facial shape and facial expression corresponding to a facial image. For example, by inputting a facial image into a trained deep neural network model, the model can extract deep features from the facial image and obtain the basic parameters of facial shape and facial expression based on the extracted features. Then, based on the basic parameters of facial shape and facial expression output by the neural network model, and a 3D simulated facial template based on a 3DMM model, a 3D simulated facial model corresponding to the real facial image can be constructed.

[0154] In one embodiment, the topological structure of the simulated face model includes multiple triangular faces; obtaining the difference features between the simulated face model and the three-dimensional simulated face template includes: obtaining the first deformation gradient of each triangular face in the simulated face model on the corresponding triangular face in the three-dimensional simulated face template; obtaining the affine transformation mapping matrix between each triangular face in the simulated face model and the corresponding triangular face in the three-dimensional simulated face template based on the first deformation gradient; the affine transformation mapping matrix is ​​used to characterize the difference features between the simulated face model and the three-dimensional simulated face template.

[0155] The deformation gradient refers to the degree of change in the geometric topology of the face. The first deformation gradient reflects the change characteristics of the 3D simulated face model corresponding to the real face relative to the 3D simulated face template.

[0156] As can be understood, a 3D simulated face model is similar to a 3D simulated face template; it is also a type of 3D geometric model. The topological structure of a geometric model includes a large number of vertices, forming multiple triangular faces. Generally, the more vertices, the higher the accuracy of the 3D cartoon face model. By deforming these triangular faces, various shapes and expressions of 3D simulated faces can be simulated.

[0157] Typically, the topological structure of a geometric model consists of many small triangular facets. The topological structure of a 3D simulated face template includes multiple triangular facets, and the topological structure of a 3D simulated face model also includes multiple triangular facets. The topological structure of the 3D simulated face model and the 3D simulated face template is consistent; that is, each triangular facet in the 3D simulated face model has a corresponding triangular facet in the 3D simulated face template. In one embodiment, the basic unit for deformation and transfer processing is the triangular facet of the geometric model.

[0158] For example, such as Figure 4 The diagram shown is a schematic representation of the topological structure of a 3D simulated human face template in one embodiment. Wherein, Figure 4 The diagram includes a topological structure diagram (a) of a 3D simulated face template, a triangular face (a1) within the topological structure of the 3D simulated face template, and a triangular face (b1) obtained after deformation of the triangular face (a1). From the topological structure diagram (a), it can be seen that the topological structure of the 3D simulated face template includes multiple vertices; any three vertices connected form a triangular face. The topological structure of the 3D simulated face template can be considered as being composed of multiple triangular faces. Since the 3D simulated face template is a deformable geometric model, in the process of reconstructing a simulated face model corresponding to a real face based on the 3D simulated face template, it can be considered that the triangular faces in the topological structure of the 3D simulated face template undergo deformation corresponding to the features of a real face, thus obtaining a simulated face model corresponding to a real face. Figure 4 The center is a vertex of one of the triangles (a1) in the topological structure of the 3D simulated human face template. i1、 i2, i3, after deforming triangle (a1), we obtain triangle (b1) and its corresponding vertices vi1, vi2, and vi3. It can be seen that the shapes of triangle (a1) and triangle (b1) and the positions of their corresponding vertices have changed. By deforming each triangle in the 3D simulation face template, we can simulate and reconstruct 3D simulation face models with various shapes and expressions.

[0159] like Figure 5 As shown, Figure 5The system includes a 3D simulated face template (A) and a 3D simulated face model (B) constructed based on the 3D simulated face template (A) and corresponding to a real face image. The constructed simulated face model (B) has both the features of the 3D simulated face template (A) and the real face features of the real face image.

[0160] Since the 3D simulated face model corresponding to a real face is generated based on the 3D simulated face template, the 3D simulated face model has both the real face features in the real face image and the average face features in the 3D simulated face template, including important information such as facial features and contour features.

[0161] Therefore, by extracting the difference features between the 3D simulated face model and the 3D simulated face template, that is, extracting the difference features between the user's 3D face and the average 3D face, and then transferring the extracted difference features to the preset 3D cartoon face template, that is, the average cartoon face, a 3D cartoon face with the user's real facial features can be obtained.

[0162] Specifically, after the computer device constructs a three-dimensional simulated face model corresponding to a real face image, it obtains the first deformation gradient of each triangular face in the simulated face model on the corresponding triangular face in the three-dimensional simulated face template based on the correspondence between the topological structure of the three-dimensional simulated face model and the three-dimensional simulated face template.

[0163] Furthermore, the proportions of the triangular faces in the topological structures of the two models can be extracted. This means extracting the proportional mapping relationship between the corresponding triangular faces in the user's simulated face model and the 3D simulated face template, rather than the difference relationship. A triangular mapping relationship can be obtained between the corresponding triangular faces of the two 3D models, which is the first deformation gradient. The first deformation gradient reflects the shape changes of the triangular faces in space, such as spatial angles and side lengths, and thus reflects the shape changes of the constructed simulated face model.

[0164] The computer device obtains the affine transformation mapping matrix between each triangle facet in the simulated face model and the corresponding triangle facet in the 3D simulated face template, based on the first deformation gradient between the triangle faces corresponding to the triangle faces in the two 3D models. This affine transformation mapping matrix includes the mapping relationship between all triangle faces in the simulated face model and the 3D simulated face template, as well as the first deformation gradient. The obtained affine transformation mapping matrix is ​​then used to characterize the differences between the simulated face model and the 3D simulated face template.

[0165] In this embodiment, the affine transformation mapping matrix is ​​obtained by directly calculating the deformation gradient of the corresponding triangular faces between the user's simulated face model and the three-dimensional simulated face template. This effectively extracts the variation features of the simulated face model relative to the three-dimensional simulated face template, thereby accurately extracting the difference features between the simulated face model and the three-dimensional simulated face template.

[0166] In one embodiment, the topological structure of the 3D cartoon face template includes multiple triangular faces. Based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, the differential features are transferred to the 3D cartoon face template to obtain a 3D cartoon face model. This includes: aligning the simulated face model and the 3D cartoon face template based on the semantic mapping relationship between the simulated face template and the 3D cartoon face template; finding triangular faces in each triangular face of the aligned 3D cartoon face template that match each triangular face of the simulated face model to obtain the triangular face mapping relationship between each triangular face of the simulated face model and the triangular face of the 3D cartoon face template; and transferring the differential features to the 3D cartoon face template according to the triangular face mapping relationship to obtain a 3D cartoon face model.

[0167] It's understandable that, similar to 3D simulated face templates, 3D cartoon face templates are also a type of 3D geometric topological structure model. However, the topological structures of 3D simulated face templates and 3D cartoon face templates differ. The topological structure of a 3D cartoon face template also includes a large number of vertices, forming multiple triangular faces. However, 3D simulated face templates and 3D cartoon face templates do not need to have the same number of vertices, triangular faces, or the same connection methods between points. By deforming the individual triangular faces in a 3D cartoon face template, various shapes and expressions of 3D cartoon faces can be simulated.

[0168] Since a semantic mapping relationship has been established in advance between the topological structures of the 3D simulated face template and the 3D cartoon face template, and the reconstructed simulated face model has the same topological structure as the 3D simulated face template, the correspondence between the topological structures of the simulated face model and the 3D cartoon face template can be obtained based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template.

[0169] After the computer device extracts the difference features between the user's 3D simulated face model and 3D simulated face template, it can then use the extracted difference features to deform the preset 3D cartoon face template based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, so as to transfer the difference features to the 3D cartoon face template, thereby obtaining a 3D cartoon face with the user's real facial features.

[0170] Specifically, the computer device first determines the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template. Then, according to this semantic mapping relationship, it obtains the semantic correspondence between the triangular faces in the simulated face model and the triangular faces in the 3D cartoon face template. Finally, according to the principle of semantic alignment, it deforms the simulated face model and the 3D cartoon face template to be consistent.

[0171] For example, such as Figure 6 The image shown is a schematic diagram illustrating the alignment process between a simulated human face model and a 3D cartoon face template in one embodiment. Wherein, Figure 6 The 3D simulated face template (A) and the simulated face model (B) have the same topological structure, while the simulated face model (B) and the 3D cartoon face template (C) have different topological structures. As can be seen from the figure, one has a back of the head and the other does not. The triangular correspondence between the simulated face model (B) and the 3D cartoon face template (C) is unknown, so it needs to be obtained through model alignment. Therefore, based on the preset semantic mapping relationship between the 3D simulated face template (A) and the 3D cartoon face template (C), the simulated face model (B) and the 3D cartoon face template (C) are semantically deformed and aligned to obtain the aligned 3D cartoon face template (C1).

[0172] The computer then searches for matching triangles in the aligned 3D cartoon face template to those in the simulated face model. Specifically, a matching triangle can be the two semantically nearest or closest triangles, meaning triangles that conform to the principle of semantic consistency. This effectively obtains the triangle mapping relationship between the triangles in the simulated face model and those in the 3D cartoon face template. This triangle mapping relationship represents the variation characteristics between the simulated face model and the 3D cartoon face template.

[0173] The computer equipment can then perform deformation and transfer processing on the 3D cartoon face template according to the triangular face mapping relationship and the difference features, thereby realizing the transfer of difference features to the 3D cartoon face template, thus effectively obtaining a 3D cartoon face model with real facial features.

[0174] For example, the reconstructed 3D simulated face model corresponding to the real face image, relative to the differences in the nose of the 3D simulated face template, such as a large nose or a high nose bridge, can be applied to the nose of the 3D cartoon face template by applying the differences in the nose to the nose of the simulated face model, thereby achieving equivalent changes in the nose of the simulated face model, and thus enabling semantically accurate transfer of the difference features to the 3D cartoon face template.

[0175] In this embodiment, since there is a preset semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, the semantic correspondence between the triangular faces in the simulated face model and the triangular faces in the 3D cartoon face template can be obtained by following the semantic mapping relationship. Thus, the simulated face model and the 3D cartoon face template can be accurately deformed to be consistent according to the semantic mapping relationship and semantic alignment principle.

[0176] In one embodiment, the differential features are transferred to a 3D cartoon face template according to the triangular face mapping relationship to obtain a 3D cartoon face model. This includes: deforming the original triangular faces in the 3D cartoon face template according to the triangular face mapping relationship and the affine transformation mapping matrix to determine a second deformation gradient between the original and deformed triangular faces in the 3D cartoon face template; iteratively deforming each triangular face in the aligned 3D cartoon face template according to the triangular face mapping relationship and the affine transformation mapping matrix in a direction that minimizes the difference between the second deformation gradient and the first deformation gradient; and obtaining a 3D cartoon face model after satisfying the iteration stopping condition.

[0177] Among them, iterative deformation processing refers to performing multiple deformation processes on the 3D cartoon face template to continuously optimize the cartoon face model, so that the final 3D cartoon face model is obtained after meeting the corresponding iteration stopping conditions.

[0178] Understandably, the second deformation gradient is the deformation gradient between the original triangular faces and the deformed triangular faces in the 3D cartoon face template during the deformation process based on the obtained difference features.

[0179] The computer equipment aligns the simulated human face model with the 3D cartoon face template, and obtains the triangular face mapping relationship between each triangle in the simulated human face model and the triangle in the 3D cartoon face template. Then, based on the triangular face mapping relationship and the affine transformation mapping matrix, it deforms the original triangles in the 3D cartoon face template. Specifically, according to the triangular face mapping relationship and the first deformation gradient corresponding to the affine transformation mapping matrix, the corresponding original triangles in the 3D cartoon face template are deformed. During deformation, the original triangles in the 3D cartoon face template change, and the computer equipment determines the second deformation gradient between the original and deformed triangles in the 3D cartoon face template.

[0180] It's understandable that reconstructing a 3D cartoon face is a continuous optimization process. Therefore, the computer equipment needs to perform iterative deformation processing to optimize the 3D cartoon face model. Specifically, after determining the second deformation gradient, the computer equipment iteratively deforms each triangle in the aligned 3D cartoon face template according to the triangular face mapping relationship and affine transformation mapping matrix, in the direction that minimizes the difference between the second deformation gradient and the first deformation gradient. After satisfying the iteration stopping condition, the final 3D cartoon face model can be obtained.

[0181] The iteration stopping condition can be a preset number of iterations, the difference between the second deformation gradient and the first deformation gradient can reach a preset difference threshold, or the difference between the second deformation gradient and the first deformation gradient can no longer decrease.

[0182] like Figure 7 The image shown is a schematic diagram illustrating the transfer of differential features to a 3D cartoon face template in one embodiment. Wherein, Figure 7 The difference features between the 3D simulated face template (A) and the 3D simulated face model (B) can be obtained. Since there is a pre-defined semantic mapping relationship between the 3D simulated face template (A) and the 3D cartoon face template (C), and the affine transformation mapping matrices of each triangular face in the 3D simulated face template (A) and the simulated face model (B) have been obtained, i.e., the difference features, the 3D cartoon face template (C) is then deformed according to the difference features to obtain the 3D cartoon face model (D). Thus, by deforming the first deformation gradient between the 3D simulated face template (A) and the simulated face model (B), and the second deformation gradient between the 3D cartoon face template (C) and the 3D cartoon face model (D) until they are consistent, it can be considered that the difference between the 3D simulated face template (A) and the simulated face model (B) has been transferred to the 3D cartoon face template (C), thereby obtaining the 3D cartoon face model (D) corresponding to the real face image.

[0183] In this embodiment, based on the preset semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, and the affine transformation mapping matrix of each triangular face in the 3D simulated face template and the simulated face model, the triangular faces in the 3D cartoon face template are iteratively deformed. This minimizes the difference between the second deformation gradient of the triangular faces in the 3D cartoon face template and the first deformation gradient corresponding to the affine transformation mapping matrix. As a result, the difference features can be accurately transferred to the 3D cartoon face template, thereby effectively obtaining a 3D cartoon face model with realistic facial features.

[0184] In one embodiment, extracting facial feature points and image features from a real face image includes: extracting facial feature key points, contour key points, and their corresponding semantic information from the real face image; obtaining facial feature points based on the facial feature key points, contour key points, and their corresponding semantic information; and extracting image features from the real face image based on the facial feature points.

[0185] Among them, facial key points refer to the key points corresponding to the facial features, such as the key points corresponding to at least one of the facial features, including eyebrows, eyes, nose, lips, and chin. Contour key points refer to the key points corresponding to the overall contour of the face, such as the key points corresponding to the jawline contour. The semantic information corresponding to facial key points and contour key points respectively refers to the geometric feature information between each key point, such as distance, area, and angle.

[0186] After acquiring a real face image, the computer device extracts key facial features, contour key points, and their corresponding semantic information from the real face image to obtain facial feature points in the real face image. Specifically, the computer device can use a pre-trained face detection network or a preset facial feature point detection algorithm to extract facial feature points from the real face image.

[0187] In one embodiment, a computer device can pre-train a face feature extraction network using sample images including human faces. This face feature extraction network can employ a machine learning model based on deep networks such as CNN (Convolutional Neural Networks), ResNet (Deep Residual Convolutional Network), DenseNet (Dense Convolutional Network), and DPN (Dual Path Network). In this embodiment, the face feature extraction network improves its robustness, real-time performance, and accuracy of feature extraction by optimizing the deep network structure, cropping the network size, and improving the quality of the sample images.

[0188] Specifically, computer equipment can extract facial features from real face images using a trained facial feature extraction network. This extraction can yield 256 facial feature points, each with specific semantic information. For example, point x can represent the nose, and point y can represent the eye. By using a pre-trained, highly accurate facial feature extraction network to extract facial features from real face images, reliable facial points can be extracted. Based on these extracted facial points, accurate face analysis can be performed, which is a crucial preliminary step for subsequent 3D simulated face reconstruction and attribute analysis. Figure 8As shown, after extracting facial feature points from the real face image (8a), a schematic diagram of the extracted facial feature points (8b) can be obtained. It is understandable that the actual number of extracted facial feature points is not limited to the number shown in the schematic diagram (8b). It is also understandable that, to protect the privacy of the user's real face image, the eyes in the real face image (8a) are occluded. Therefore, when extracting feature points from the real face image, feature points from the eyes are also collected, and the schematic diagram (8b) of the extracted facial feature points also includes feature points from the eyes.

[0189] Furthermore, after extracting facial feature points from a real face image, the computer device further extracts image features from the real face image based on the extracted facial feature points. Specifically, the computer device can extract image features of corresponding regions in the real face image according to the positional distribution of facial feature points. For example, image features typically include some image features of different regions of the face, so image features of corresponding regions can be extracted based on the positional distribution of facial feature points.

[0190] In this embodiment, after first extracting the facial feature points and corresponding semantic information from the real face image, the image features of the real face image are further extracted based on the facial feature points, thereby enabling more accurate extraction of the image features in the real face image.

[0191] In one embodiment, the image features include native image features and additional image features; extracting image features from a real face image based on facial feature points includes: performing face alignment processing on the real face image to obtain an aligned real face image; identifying native image features from the aligned real face image based on facial feature points; and identifying additional image features from the aligned real face image based on facial feature points.

[0192] It's understandable that the face in the original real-life face image may be tilted relative to the image, meaning the face in the real-life face image is not a frontal view. Face alignment processing refers to correcting the face in the original real-life face image so that the face in the aligned real-life face image is a frontal view relative to the image.

[0193] Native image features refer to the inherent features of a real face in a real face image, such as the user's own hair, birthmarks, moles, and other image-related characteristics. Additional image features refer to the features corresponding to accessories worn by the user in a real face image. These additional image features in a real face image are the features of accessories included in the face region and the surrounding area. For example, additional image features include the features of accessories such as glasses, hats, and earrings.

[0194] After extracting facial feature points from a real face image, the computer device first performs face alignment processing on the original real face image based on the facial feature points. Specifically, the computer device can perform face alignment on the original real face image based on alignment key points or facial feature points in the face, so as to correct and align the position of the face in the real face image, thereby obtaining an aligned real face image.

[0195] In one embodiment, such as Figure 9 As shown, by performing face alignment processing on the real face image (9a), the face-aligned image (9b) can be obtained. It can be seen that the face in the face-aligned image (9b) is a frontal face relative to the image, thus enabling more accurate extraction of image features from the real face image.

[0196] In one embodiment, the computer device can align the original real face image during the process of extracting facial feature points from the real face image. Specifically, it can correct and align the position of the face in the original real face image based on the alignment key points or initial facial feature points in the face, so as to further extract facial feature points and thus extract facial feature points in the real face image more accurately.

[0197] After aligning a real face image, the computer device identifies the original facial features from the aligned image based on facial landmarks. Simultaneously, the computer device also identifies additional facial features from the aligned image based on these landmarks.

[0198] Specifically, computer equipment extracts native image features of corresponding regions in a real face image based on the location distribution of facial feature points. For example, native image features and additional image features in the region within and around the face region.

[0199] In this embodiment, by performing face alignment on the original real face image, the original image features and additional image features in the real face image can be extracted more accurately based on the positional distribution of facial feature points.

[0200] In one embodiment, the native image features include hairstyle features; identifying hairstyle features from an aligned real face image based on facial feature points includes: extracting hair features from the aligned real face image using a trained hairstyle recognition network, and generating a hair mask based on the hair features; dividing the hair mask into at least two mask sub-regions based on facial feature points; and obtaining the hairstyle features based on the distribution of hair features in the at least two mask sub-regions.

[0201] It's understandable that native facial features include hairstyle features, which in turn include hairstyle categories, such as short hair, medium hair, and long hair. More specifically, it's possible to further identify the gender characteristics of the face and then differentiate hairstyle features based on gender. For example, if the actual gender of the face is female, then hairstyle categories include short hair, medium hair, long hair, as well as straight hair and curly hair. If the actual gender of the face is male, then hairstyle categories include very short hair, short hair, and medium hair.

[0202] Computer devices can pre-train a hairstyle recognition network using a large number of sample images, enabling the trained hairstyle recognition network to recognize hairstyle features in real face images.

[0203] Specifically, the computer device extracts facial feature points from real face images and performs face alignment processing on the original real face images. Then, the face-aligned real face images are input into a trained hairstyle recognition network. The hairstyle recognition network extracts special hair features from the aligned real face images and generates a hair mask map based on the extracted hair features.

[0204] The computer device then divides the hair mask into at least two sub-regions based on facial feature points. For example, the face region can be divided according to the distribution of facial feature points within the face, thereby dividing the entire hair mask into multiple sub-regions. The computer device then obtains the hairstyle features based on the distribution of hair features within these multiple sub-regions.

[0205] In one embodiment, the computer device may first divide the face region according to the location distribution of facial feature points on the face, and then input the image of the divided face region into a trained hairstyle recognition network. After the computer device extracts the hair mask image through the hairstyle recognition network, it can directly divide the hair mask image into multiple mask sub-regions according to the division markers used to divide the face region.

[0206] In one embodiment, such as Figure 10 The image shown is the result of facial feature point extraction, face alignment, and face region segmentation from a real face image. Figure 10 The image shown includes extracted facial feature points, namely key points corresponding to at least one of the following facial features: eyebrows, eyes, nose, lips, chin, and jawline. The computer device divides the face region based on the distribution of these feature points within the face; specifically, it divides the face region into three equal parts and then... Figure 10 The image is divided into three sub-regions by marking with dividing markers 10a and 10b.

[0207] Computer equipment through Figure 10 After extracting hair features from the image, such as Figure 11 As shown, the computer device generates a hair mask based on the extracted hair features. Figure 11 The white areas represent hair features in a real human face image, while the black areas represent the background excluding hair features. Figure 11 The partition markers 11a and 11b in the middle are... Figure 10 The division marks 10a and 10b correspond to each other. Therefore, the hair mask can be divided into three mask sub-regions according to the division marks 11a and 11b.

[0208] The computer then determines the hairstyle features based on the distribution of hair features across multiple mask sub-regions. For example, Figure 11 Based on division marks 11a and 11b, the hair mask image is divided into three mask image sub-regions. Division mark 11a can be specifically obtained based on the ear root key point, and division mark 11b can be specifically obtained based on the chin key point. Therefore, the part above division mark 11a is the mask image sub-region Q1, the part between division marks 11a and 11b is the mask image sub-region Q2, and the part below division mark 11b is the mask image sub-region Q3.

[0209] If most hair features in the hair mask are concentrated in sub-region Q1, and almost none are distributed in sub-regions Q2 and Q3, then the hairstyle feature is determined to be short hair. If most hair features in the hair mask are distributed in sub-region Q1, with some also distributed in sub-region Q2, then the hairstyle feature is determined to be short hair. Similarly, if most hair features in the hair mask are distributed in sub-region Q1, with some also distributed in sub-regions Q2 and Q3, then the hairstyle feature is determined to be long hair. In one embodiment, if there are no hair features in the hair mask, considering aesthetics, the corresponding hairstyle feature can be determined to be short hair.

[0210] In this embodiment, after extracting hair features and generating a hair mask map through a trained hairstyle recognition network, the hair mask map is divided into at least two mask map sub-regions based on facial feature points. Then, by the distribution of hair features in at least two mask map sub-regions, the hairstyle features in real face images can be accurately identified.

[0211] In one embodiment, identifying additional image features from an aligned real face image based on facial feature points includes: extracting additional image region images from the aligned real face image according to the distribution location of facial feature points; identifying the additional image category of the additional image region image through a trained target classification network; and obtaining the additional image features in the real face image according to the additional image category.

[0212] The object classification network is a pre-trained deep learning network capable of classifying accessories in real-world face images. This network can employ the lightweight MobileNet network, whose basic unit is a deep-level separable convolution. In some embodiments, the object classification network can also employ machine learning models based on deep networks such as CNN (Convolutional Neural Networks), ResNet (Deep Residual Convolutional Network), DenseNet (Dense Convolutional Network), and DPN (Dual Path Network).

[0213] Computer equipment can pre-train a target classification network using a large number of sample images, enabling the trained target classification network to recognize additional image features in real human face images.

[0214] Specifically, the computer device extracts facial feature points from a real face image and performs face alignment processing on the original real face image. First, based on the distribution of the facial feature points, it extracts the accessory region image from the aligned real face image. Then, the computer device inputs the accessory region image into a trained target classification network, which then identifies the category of the accessory components in the accessory region image to obtain additional image features from the real face image based on the category of the accessory components.

[0215] In one embodiment, the additional image features include eyeglass features. For example, eyeglass features include at least one of the following: no glasses, black-rimmed glasses, metal-rimmed glasses, etc. Figure 12 The diagram shown illustrates a flowchart of classifying eyeglass features in one embodiment. Figure 9Taking a real face image as an example, after extracting facial feature points from the real face image, the eye area is extracted from the aligned face image based on the key points of the glasses in the face region, thus obtaining a glasses region image, which is used as input image 12a. The computer device then adjusts the glasses region image to a preset size, obtaining a resized image 12b, for example, the preset size can be 64*32. The resized image 12b is then input into the target classification network, specifically the MobileNet network 1202. The convolutional networks in the MobileNet network 1202 extract features from the resized image 12b, and then the fully connected layer 1204 identifies the category of accessory parts in the image and outputs the classification result 1206. The classification result is the additional image category, which can include any one of no glasses, black-rimmed glasses, and metal-rimmed glasses. The computer device then obtains additional image features in the real face image based on the additional image category.

[0216] In one embodiment, the target classification network uses the MobileNet lightweight network, whose basic unit is a depthwise separable convolution. The MobileNet lightweight network includes multiple network layers, each containing attributes such as layer type, stride, filter shape size, and input image size. For example, the layer type and corresponding stride can include at least one of Conv / s2, Conv dw / s1, Conv / s1, Conv dw / s2, Avg Pool / s1, FC / s1, and Softmax / s1. The filter shape size can include at least one of 3x3x3x32, 3x3x32dw, 1x1x32dw, 3x3x64dw, 1x1x64x128, 3x3x128dw, 1x1x64x128, 3x3x256dw, 3x3x512dw, and 3x3x1024dw. The input image size can include at least one of the following: 64x32x3, 112x112x32, 56x56x64, 56x56x128, 28x28x128, 28x28x256, 14x14x256, 14x14x512, 7x7x1024, and 1x1x1024. Specifically, the input image size for the initial network layer of the MobileNet lightweight network can be 64x32x3. The output categories of the FC layer are three, such as no glasses, black-rimmed glasses, and metal-rimmed glasses.

[0217] In one experimental test instance, over 10,000 sample images were acquired as a training set, including 3,500 images without glasses, 3,420 images with black-rimmed glasses, and 3,320 images with metal-rimmed glasses. A target classification network was trained using these sample images. Testing showed that the trained target classification network achieved an overall classification accuracy of 92% for the glasses category, demonstrating its effectiveness in training a target classification network with high accuracy.

[0218] In one embodiment, the image features include skin surface color features, native image features, and additional image features. The process of rendering a cartoon face model based on these image features to generate a 3D cartoon image includes: extracting pixels from the skin surface color region based on the distribution of facial feature points; obtaining skin surface color features based on the pixels in the skin surface color region; rendering the skin surface color of the 3D cartoon face model based on the skin surface color features; obtaining matching image materials based on the native and additional image features; and rendering the image materials onto the 3D cartoon face model after skin surface color rendering to obtain a 3D cartoon image with facial features and image features.

[0219] Among them, skin surface color features refer to the characteristics of the skin surface color of a real human face in a real face image. For example, skin surface color features can be represented by pixel color features or the color intensity of the skin surface.

[0220] In the process of extracting a real face image, the computer equipment also extracts pixels from the skin surface color region based on the distribution of facial feature points, and obtains skin surface color features based on the pixels of the skin surface color region. Specifically, the computer equipment can remove areas around the eyes and mouth that might affect the calculation of skin surface color based on the eye and lip key points in the real face image, in order to extract the skin surface color region that can be used for skin surface color features, that is, mainly the pixels of the cheek area. The computer equipment then compares the pixels of the skin surface color region with the pixels of a preset skin surface color chart, and obtains the skin surface color chart with the smallest pixel difference as the skin surface color feature of the real face image. Figure 13 The image shown is a schematic diagram of a skin surface color chart in one embodiment. (Refer to...) Figure 13 Different skin surface color characteristics can be distinguished by the depth of color. Figure 13 The image shows five different skin surface color charts representing different skin surface color characteristics.

[0221] The computer device extracts the image features from the real human face image, namely skin surface color features, original image features and additional image features, and constructs a corresponding three-dimensional cartoon face model. Then, the skin surface color of the cartoon face model is rendered according to the pixel color corresponding to the skin surface color features.

[0222] The computer equipment further acquires matching image materials based on the original and additional image features. The database includes image materials corresponding to various original and additional image features. The computer equipment then renders these image materials onto a 3D cartoon face model with rendered skin surface colors, thus effectively obtaining a 3D cartoon image with facial and overall image characteristics.

[0223] In one specific embodiment, with Figure 9 Taking a real human face image as an example, after constructing a corresponding 3D cartoon face model, the extracted image features are rendered onto the corresponding cartoon face model to obtain a 3D cartoon image. Figure 14 As shown, this is the effect of a 3D cartoon image obtained after reconstructing a real human face image 14 (R1) using 3D cartoonization. Figure 14 (C1). The original features of the real face in the real face image 14 (R1) include skin surface color features, hairstyle features, and mole marker features, as well as additional features and glasses features. Therefore, the constructed 3D cartoon image achieves its desired effect. Figure 14 (C1) also includes image features that correspond to the skin surface color features, hairstyle features, mole features, and glasses features of real human faces.

[0224] like Figure 15 The image shown is a rendering of a 3D cartoon character obtained by reconstructing a real human face image into a 3D cartoon form in another embodiment. Figure 15 The image shows three sets of real face images, namely real face image 15(R1), real face image 15(R2) and real face image 15(R3). After three-dimensional cartoon reconstruction of the real images, the corresponding three-dimensional cartoon images 15(C1), 15(C2) and 15(C3) are obtained respectively.

[0225] In this embodiment, by extracting the skin surface color features, original image features, and additional image features from real face images, and after constructing a 3D cartoon face model corresponding to a real face, the colors corresponding to the skin surface color features, as well as the materials corresponding to the original image features and additional image features, are rendered onto the cartoon face model. This allows for the construction of a highly accurate 3D cartoon image with real face features and image features, effectively improving the similarity between the 3D cartoon image and the real face.

[0226] In one embodiment, before extracting facial feature points and image features from a real face image, the method further includes: displaying a cartoon image selection interface, which includes three-dimensional image options; in response to the selection operation of the three-dimensional image options, acquiring a real face image of the user; and after rendering the cartoon face model according to the image features to generate a three-dimensional cartoon image, the above-mentioned face image cartoonization processing method further includes: displaying an image preview interface and displaying the real face image and the three-dimensional cartoon image in the image preview interface.

[0227] The computer device can be a terminal, running an application capable of reconstructing 3D cartoon characters. This application includes a cartoon character selection interface. The cartoon character selection interface is used to guide the user in choosing the type of cartoon character. For example, the interface might include both 2D and 3D character options.

[0228] After the user selects the desired cartoon character type in the character selection interface, an image acquisition interface is displayed to capture the user's real face image. Specifically, when the user selects the 3D character option in the character selection interface, the terminal responds to the selection operation for the 3D character option and then acquires the user's real face image. The terminal may also display an image acquisition interface, which can be a local image selection interface or an image capture interface. The local image selection interface is used to retrieve existing real face images from the terminal's local database. The image capture interface is used to capture the user's real face image in real time.

[0229] After acquiring a real face image, the terminal first extracts facial feature points and image features from the image. Then, based on these feature points, it constructs a 3D simulated face model corresponding to the real face image and obtains the difference features between the simulated face model and the 3D simulated face template. Further, based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, the difference features are transferred to the 3D cartoon face template, thus obtaining a 3D cartoon face model. Finally, the cartoon face model is rendered based on the extracted image features, resulting in a 3D cartoon image with both real facial features and image features. This allows for the accurate and effective construction of 3D cartoon images that are more similar to real faces.

[0230] After the terminal obtains a 3D cartoon image corresponding to a real face image, it displays an image preview interface, showing the original real face image and the constructed 3D cartoon image. This allows for effective comparison of the similarity between the real face image and the 3D cartoon image in the image preview interface.

[0231] In a specific embodiment, such as Figure 16 The diagram shown is a schematic of a cartoon character selection interface in one embodiment. For example, it could be a cartoon character selection interface in the "Centimeter Show" application. The cartoon character selection interface includes a two-dimensional cartoon character display area 2D and a three-dimensional cartoon character display area 3D. Two-dimensional character options 2D1A are displayed in the two-dimensional cartoon character display area 2D, and three-dimensional character options 3Db are displayed in the three-dimensional cartoon character display area 3D.

[0232] When the 3D model option (3Db) is selected, the image capture interface will be displayed. For example... Figure 17 The diagram shown illustrates an image capture interface in one embodiment. When the terminal displays the image capture interface, it captures a real image of the current user's face using its camera acquisition device, and uses a face detection box 17a to detect whether a face is present on the image capture interface. When a face is detected in the current frame, the real image of the current user's face is automatically captured and further processed to construct a 3D cartoon character.

[0233] After constructing a 3D cartoon image corresponding to a real human face image, a preview interface is displayed. For example... Figure 18 The diagram shown illustrates an image preview interface in one embodiment. The image preview interface includes a 3D cartoon image display area 18a and an original real human face image display area 18b. The image preview interface may also include a completion option for saving the currently generated 3D cartoon image.

[0234] Furthermore, after constructing and saving a 3D cartoon image corresponding to a real facial image, users can also customize the generated 3D cartoon image. Specifically, the terminal can display a 3D cartoon image adjustment interface, allowing users to customize the 3D cartoon image within this interface. Figure 19 The diagram shown illustrates a 3D cartoon character adjustment interface in one embodiment. The interface includes an adjustment pre-area 19a and a material selection area 19b. The material selection area 19b includes various facial features and other corresponding materials. Users can select appropriate materials in the material selection area 19b and add them to the corresponding parts of the 3D cartoon character, or replace the materials in the corresponding parts of the 3D cartoon character. This effectively enables customized adjustments to the automatically generated 3D cartoon character, significantly improving its editability and adaptability.

[0235] In another application scenario, computer equipment can also construct 3D cartoon images from real-life facial images corresponding to consecutive video frames. The computer equipment extracts facial feature points and image features from the real-life facial images, and then constructs a 3D simulated facial model based on these feature points. Next, by acquiring the difference features between the simulated facial model and a 3D simulated facial template, and based on a pre-defined semantic mapping relationship between the two templates, the difference features are transferred to the 3D cartoon facial template, resulting in a 3D cartoon facial model. Finally, the cartoon facial model is rendered based on the image features, transferring the image features from the real-life facial image into the cartoon facial model, thereby generating a 3D cartoon image that more closely resembles the real-life face in the real-life image.

[0236] Furthermore, after the computer device constructs a 3D cartoon image corresponding to the initial frame's real face image, it only needs to adjust the constructed 3D cartoon image based on the facial pose and expression in the subsequent frames' corresponding real face images. Specifically, the computer device compares the camera parameters, face shape parameters, and facial expression base parameters of the subsequent frames' real face images with the same parameters in the initial frame's real face image. If these parameters change, the corresponding parameters of the cartoon face model in the 3D cartoon image are directly adjusted based on the changed camera parameters, face shape parameters, and facial expression base parameters, ensuring that the 3D cartoon image's shape and expression are consistent with the corresponding real face image.

[0237] For example, a series of video frames, including real human faces, can be displayed on the terminal's interface. After the terminal generates a 3D cartoon image corresponding to the real face in each video frame, both the video containing the real face and the corresponding 3D cartoon image are simultaneously displayed on the interface. This allows for efficient cartoonization of real face images in video frames, enabling the rapid and accurate construction of 3D cartoon images that resemble real faces.

[0238] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0239] In one embodiment, such as Figure 20 As shown, a face image cartoonization processing device 2000 is provided. This device can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes: a feature extraction module 2002, a 3D face reconstruction module 2004, a difference feature extraction module 2006, a difference feature transfer module 2008, and a 3D cartoon character generation module 2010, wherein:

[0240] The feature extraction module 2002 is used to extract facial feature points and image features from real face images;

[0241] The 3D face reconstruction module 2004 is used to construct a 3D simulated face model corresponding to a real face image based on facial feature points.

[0242] The difference feature extraction module 2006 is used to obtain the difference features between the simulated human face model and the three-dimensional simulated human face template;

[0243] The differential feature transfer module 2008 is used to transfer differential features to the 3D cartoon face template based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, so as to obtain a 3D cartoon face model.

[0244] The 3D Cartoon Character Generation Module 2010 is used to render cartoon face models based on their features and generate 3D cartoon characters.

[0245] In one embodiment, the facial feature points are two-dimensional facial feature points; the three-dimensional face reconstruction module 2004 is also used to obtain the feature point mapping matrix of the two-dimensional facial feature points in the three-dimensional simulated face template; perform parameter estimation based on the facial feature points and the feature point mapping matrix to obtain three-dimensional face parameters; and construct a three-dimensional simulated face model corresponding to the real face image based on the three-dimensional face parameters.

[0246] In one embodiment, the 3D face reconstruction module 2004 is further configured to perform iterative estimation of camera parameters based on face feature points and feature point mapping matrix, and obtain camera parameters after satisfying the first iteration condition; perform iterative estimation of face parameters based on face feature points, feature point mapping matrix and camera parameters, and obtain face shape basis parameters and face expression basis parameters after satisfying the second iteration condition; and construct a 3D simulated face model corresponding to the real face image based on the camera parameters, face shape basis parameters and face expression basis parameters.

[0247] In one embodiment, the topological structure of the simulated face model includes multiple triangular faces; the difference feature extraction module 2006 is further used to obtain the first deformation gradient of each triangular face in the simulated face model on the corresponding triangular face in the three-dimensional simulated face template; based on the first deformation gradient, the affine transformation mapping matrix between each triangular face in the simulated face model and the corresponding triangular face in the three-dimensional simulated face template is obtained; the affine transformation mapping matrix is ​​used to characterize the difference features between the simulated face model and the three-dimensional simulated face template.

[0248] In one embodiment, the topological structure of the 3D cartoon face template includes multiple triangular faces; the difference feature transfer module 2008 is further used to align the simulated face model and the 3D cartoon face template based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template; in each triangular face in the aligned 3D cartoon face template, a triangular face matching each triangular face in the simulated face model is found to obtain the triangular face mapping relationship between each triangular face in the simulated face model and the triangular face in the 3D cartoon face template; the difference features are transferred to the 3D cartoon face template according to the triangular face mapping relationship to obtain the 3D cartoon face model.

[0249] In one embodiment, the difference feature transfer module 2008 is further configured to deform the original triangular faces in the 3D cartoon face template according to the triangular face mapping relationship and the affine transformation mapping matrix, and determine the second deformation gradient between the original triangular faces and the deformed triangular faces in the 3D cartoon face template; in the direction that minimizes the difference between the second deformation gradient and the first deformation gradient, iteratively deform each triangular face in the aligned 3D cartoon face template according to the triangular face mapping relationship and the affine transformation mapping matrix; and after satisfying the iteration stopping condition, obtain the 3D cartoon face model.

[0250] In one embodiment, the feature extraction module 2002 is further configured to extract facial feature key points, contour key points, and corresponding semantic information from a real face image; obtain facial feature points based on the facial feature key points and contour key points and corresponding semantic information; and extract image features of the real face image based on the facial feature points.

[0251] In one embodiment, the image features include original image features and additional image features; the feature extraction module 2102 is also used to perform face alignment processing on the real face image to obtain an aligned real face image; based on face feature points, the original image features are identified from the aligned real face image; based on face feature points, the additional image features are identified from the aligned real face image.

[0252] In one embodiment, the native image features include hairstyle features; the feature extraction module 2102 is further configured to extract hair features from the aligned real face image through a trained hairstyle recognition network, and generate a hair mask based on the hair features; divide the hair mask into at least two mask sub-regions based on face feature points; and obtain the hairstyle features based on the distribution of hair features in the at least two mask sub-regions.

[0253] In one embodiment, the feature extraction module 2002 is further configured to extract additional image region images from the aligned real face image based on the distribution location of facial feature points; identify the additional image category of the additional image region image through a trained target classification network; and obtain additional image features in the real face image based on the additional image category.

[0254] In one embodiment, the image features include skin surface color features, native image features, and additional image features; the 3D cartoon image generation module 2010 is also used to extract pixels of the skin surface color region based on the distribution position of facial feature points, obtain skin surface color features based on the pixels of the skin surface color region; render the skin surface color of the 3D cartoon face model based on the skin surface color features; obtain matching image materials based on the native image features and additional image features, and render the image materials onto the 3D cartoon face model after the skin surface color rendering to obtain a 3D cartoon image with facial features and image features.

[0255] In one embodiment, the device further includes a display module for displaying a cartoon character selection interface, which includes three-dimensional character options; in response to the selection operation of the three-dimensional character options, acquiring a real face image of the user; displaying a character preview interface, and displaying the real face image and the three-dimensional cartoon character in the character preview interface.

[0256] Specific limitations regarding the face image cartoonization processing device can be found in the limitations of the face image cartoonization processing method described above, and will not be repeated here. Each module in the aforementioned face image cartoonization processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0257] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 21 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as real human face images, 3D simulated human face templates, and 3D cartoon human face templates. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for cartoonizing human face images.

[0258] In one embodiment, another computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 22As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for cartoonizing facial images. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0259] Those skilled in the art will understand that Figure 21 and Figure 22 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0260] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0261] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0262] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0263] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0264] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0265] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cartoonizing facial images, characterized in that, The method includes: Facial feature points and image features are extracted from real face images; the image features include skin surface color features, original image features, and additional image features; the original image features include hairstyle features; the hairstyle features are obtained based on the distribution of hair features in the real face image in at least two mask sub-regions divided in a hair mask image, and the hair mask image is generated based on the hair features; A three-dimensional simulated face model corresponding to the real face image is constructed based on the facial feature points; the topological structure of the simulated face model includes multiple triangular faces; Obtain the difference features between the simulated human face model and the three-dimensional simulated human face template; Based on the semantic mapping relationship between the 3D simulated face template and the 3D cartoon face template, the simulated face model and the 3D cartoon face template are aligned; the topological structure of the 3D cartoon face template includes multiple triangular faces; In each of the triangular faces in the aligned 3D cartoon face template, find the triangular face that matches each of the triangular faces in the simulated face model, and obtain the triangular face mapping relationship between each of the triangular faces in the simulated face model and the triangular faces in the 3D cartoon face template. The difference features are transferred to the 3D cartoon face template according to the triangle mapping relationship to obtain a 3D cartoon face model; the 3D cartoon face model is obtained by iterative deformation processing of each triangle face in the aligned 3D cartoon face template. The cartoon face model is rendered based on the described features to generate a three-dimensional cartoon image.

2. The method according to claim 1, characterized in that, The facial feature points are two-dimensional facial feature points; the construction of a three-dimensional simulated facial model corresponding to the real facial image based on the facial feature points includes: Obtain the feature point mapping matrix of the two-dimensional facial feature points in the three-dimensional simulated facial template; Based on the facial feature points and the feature point mapping matrix, parameter estimation is performed to obtain the three-dimensional facial parameters. Based on the three-dimensional face parameters, a three-dimensional simulated face model corresponding to the real face image is constructed.

3. The method according to claim 2, characterized in that, The parameter estimation based on the facial feature points and the feature point mapping matrix to obtain the three-dimensional facial parameters includes: Based on the facial feature points and the feature point mapping matrix, the camera parameters are iteratively estimated, and the camera parameters are obtained after the first iteration condition is met. Based on the facial feature points, the feature point mapping matrix, and the camera parameters, iterative estimation of facial parameters is performed. After satisfying the second iteration condition, facial shape basis parameters and facial expression basis parameters are obtained. The step of constructing a three-dimensional simulated face model corresponding to the real face image based on the three-dimensional face parameters includes: Based on the camera parameters, the facial shape base parameters, and the facial expression base parameters, a three-dimensional simulated facial model corresponding to the real facial image is constructed.

4. The method according to claim 1, characterized in that, The step of obtaining the difference features between the simulated face model and the 3D simulated face template includes: Obtain the first deformation gradient of each triangular face in the simulated face model and the corresponding triangular face in the three-dimensional simulated face template; Based on the first deformation gradient, an affine transformation mapping matrix is ​​obtained between each triangular facet in the simulated facet model and the corresponding triangular facet in the three-dimensional simulated facet template; the affine transformation mapping matrix is ​​used to characterize the difference features between the simulated facet model and the three-dimensional simulated facet template.

5. The method according to claim 4, characterized in that, The step of transferring the difference features to the 3D cartoon face template according to the triangular face mapping relationship to obtain a 3D cartoon face model includes: Based on the triangular face mapping relationship and the affine transformation mapping matrix, the original triangular faces in the three-dimensional cartoon face template are deformed to determine the second deformation gradient between the original triangular faces and the deformed triangular faces in the three-dimensional cartoon face template. In the direction of minimizing the difference between the second deformation gradient and the first deformation gradient, each triangle in the aligned 3D cartoon face template is iteratively deformed according to the triangle mapping relationship and the affine transformation mapping matrix. After the iteration stopping condition is met, a three-dimensional cartoon face model is obtained.

6. The method according to claim 1, characterized in that, The extraction of facial feature points and image features from real facial images includes: Extract facial feature key points, contour key points, and their corresponding semantic information from real facial images; Based on the key points of the facial features and the key points of the contour, as well as the corresponding semantic information, facial feature points are obtained; Based on the facial feature points, the image features of the real face image are extracted.

7. The method according to claim 6, characterized in that, The step of extracting the image features of the real face image based on the facial feature points includes: The real face image is subjected to face alignment processing to obtain an aligned real face image; Based on the facial feature points, the original image features are identified from the aligned real face image; Based on the facial feature points, additional image features are identified from the aligned real face image.

8. The method according to claim 7, characterized in that, The step of identifying native image features from the aligned real face image based on the facial feature points includes: The hair features in the aligned real face image are extracted using a trained hair recognition network, and a hair mask is generated based on the hair features. The hair mask is divided into at least two mask sub-regions based on the facial feature points; Hairstyle features are obtained based on the distribution of the hair features in the at least two mask sub-regions.

9. The method according to claim 7, characterized in that, The step of identifying additional image features from the aligned real face image based on the facial feature points includes: Based on the distribution location of the facial feature points, an additional image region image is extracted from the aligned real face image; The additional image category of the additional image region image is identified using a trained target classification network. Based on the additional image category, additional image features in the real face image are obtained.

10. The method according to claim 1, characterized in that, The step of rendering the cartoon face model based on the image features to generate a three-dimensional cartoon image includes: Based on the distribution location of the facial feature points, pixels of the skin surface color region are extracted, and skin surface color features are obtained based on the pixels of the skin surface color region. The skin surface color of the three-dimensional cartoon face model is rendered based on the skin surface color features. Based on the original image features and the additional image features, obtain matching image materials, and render the image materials onto a three-dimensional cartoon face model after the skin surface color rendering, to obtain a three-dimensional cartoon image with facial features and image features.

11. The method according to any one of claims 1 to 10, characterized in that, Before extracting facial feature points and image features from real face images, the method further includes: A cartoon character selection interface is displayed, which includes three-dimensional character options; In response to the selection of the three-dimensional image option, a real facial image of the user is captured; After rendering the cartoon face model based on the image features to generate a three-dimensional cartoon image, the method further includes: The system displays a preview interface, showing the real human face image and the 3D cartoon character.

12. A face image cartoonization processing device, characterized in that, The device includes: The feature extraction module is used to extract facial feature points and image features from real face images; the image features include skin surface color features, original image features, and additional image features; the original image features include hairstyle features; the hairstyle features are obtained based on the distribution of hair features in the real face image in at least two mask sub-regions divided in a hair mask image, and the hair mask image is generated based on the hair features; A 3D face reconstruction module is used to construct a 3D simulated face model corresponding to the real face image based on the face feature points; the topological structure of the simulated face model includes multiple triangular faces; The difference feature extraction module is used to obtain the difference features between the simulated face model and the three-dimensional simulated face template; The differential feature transfer module is used to align the simulated face model and the 3D cartoon face template based on the semantic mapping relationship between them. The topological structure of the 3D cartoon face template includes multiple triangular faces. Among the triangular faces in the aligned 3D cartoon face template, triangular faces that match those in the simulated face model are found, thus obtaining the triangular face mapping relationship between the triangular faces in the simulated face model and the triangular faces in the 3D cartoon face template. The differential features are then transferred to the 3D cartoon face template according to this triangular face mapping relationship to obtain a 3D cartoon face model. The 3D cartoon face model is obtained by iteratively deforming the triangular faces in the aligned 3D cartoon face template. The 3D cartoon character generation module is used to render the cartoon face model based on the character features to generate a 3D cartoon character.

13. The face image cartoonization processing device according to claim 12, characterized in that, The facial feature points are two-dimensional facial feature points; the three-dimensional face reconstruction module is also used to obtain the feature point mapping matrix of the two-dimensional facial feature points in the three-dimensional simulated face template; based on the facial feature points and the feature point mapping matrix, parameter estimation is performed to obtain the three-dimensional face parameters; Based on the three-dimensional face parameters, a three-dimensional simulated face model corresponding to the real face image is constructed.

14. The face image cartoonization processing device according to claim 13, characterized in that, The 3D face reconstruction module is further configured to perform iterative estimation of camera parameters based on the facial feature points and the feature point mapping matrix, and obtain camera parameters after satisfying the first iteration condition; perform iterative estimation of face parameters based on the facial feature points, the feature point mapping matrix, and the camera parameters, and obtain face shape basis parameters and face expression basis parameters after satisfying the second iteration condition; and construct a 3D simulated face model corresponding to the real face image based on the camera parameters, the face shape basis parameters, and the face expression basis parameters.

15. The face image cartoonization processing device according to claim 12, characterized in that, The differential feature extraction module is further used to obtain the first deformation gradient of each triangular face in the simulated face model on the corresponding triangular face in the three-dimensional simulated face template; and to obtain the affine transformation mapping matrix between each triangular face in the simulated face model and the corresponding triangular face in the three-dimensional simulated face template based on the first deformation gradient. The affine transformation mapping matrix is ​​used to characterize the differences between the simulated face model and the three-dimensional simulated face template.

16. The face image cartoonization processing device according to claim 15, characterized in that, The difference feature transfer module is further configured to deform the original triangular faces in the 3D cartoon face template according to the triangular face mapping relationship and the affine transformation mapping matrix, and determine the second deformation gradient between the original triangular faces and the deformed triangular faces in the 3D cartoon face template; in the direction that minimizes the difference between the second deformation gradient and the first deformation gradient, iteratively deform each triangular face in the aligned 3D cartoon face template according to the triangular face mapping relationship and the affine transformation mapping matrix; after satisfying the iteration stopping condition, a 3D cartoon face model is obtained.

17. The face image cartoonization processing device according to claim 12, characterized in that, The feature extraction module is further used to extract facial feature key points, contour key points, and corresponding semantic information from real face images; obtain facial feature points based on the facial feature key points, contour key points, and corresponding semantic information; and extract image features of the real face image based on the facial feature points.

18. The face image cartoonization processing device according to claim 17, characterized in that, The feature extraction module is also used to perform face alignment processing on the real face image to obtain an aligned real face image; and to identify original image features from the aligned real face image based on the face feature points. Based on the facial feature points, additional image features are identified from the aligned real face image.

19. The face image cartoonization processing device according to claim 18, characterized in that, The feature extraction module is further configured to extract hair features from the aligned real face image through a trained hairstyle recognition network, and generate a hair mask based on the hair features; divide the hair mask into at least two mask sub-regions based on the face feature points; and obtain hairstyle features based on the distribution of the hair features in the at least two mask sub-regions.

20. The face image cartoonization processing device according to claim 18, characterized in that, The feature extraction module is further configured to extract additional image region images from the aligned real face image based on the distribution location of the facial feature points; identify the additional image category of the additional image region image through a trained target classification network; and obtain the additional image features in the real face image based on the additional image category.

21. The face image cartoonization processing device according to claim 12, characterized in that, The 3D cartoon character generation module is also used to extract pixels of the skin surface color region based on the distribution position of the facial feature points, and to obtain skin surface color features based on the pixels of the skin surface color region. The skin surface color of the three-dimensional cartoon face model is rendered according to the skin surface color features; matching image materials are obtained according to the original image features and the additional image features, and the image materials are rendered into the three-dimensional cartoon face model after the skin surface color rendering, so as to obtain a three-dimensional cartoon image with facial features and image features.

22. The face image cartoonization processing apparatus according to any one of claims 12 to 21, characterized in that, The device also includes a display module for displaying a cartoon character selection interface, which includes three-dimensional character options; in response to the selection operation of the three-dimensional character options, it acquires a real face image of the user; and displays a character preview interface, in which the real face image and the three-dimensional cartoon character are displayed.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 11.

24. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 11.

25. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1 to 11.

Images