Method, apparatus, computer device, and storage medium for processing three-dimensional cartoon image
Patent Information
- Application Number
- HK42023069076
- Authority / Receiving Office
- HK · HK
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-02-27
- Publication Date
- 2026-08-28
AI Technical Summary
The existing technology does not produce 3D cartoon images that match the expressions of real human faces very well, and cannot accurately simulate the changes in facial expressions.
By acquiring a basic 3D cartoon image with removed facial expressions and then tracking the user's facial expressions in real time within the camera's field of view, a 3D cartoon image synchronized with the user's facial expressions is generated using facial feature prediction networks and machine learning techniques.
It achieves a high degree of matching and synchronous changes between 3D cartoon characters and real human facial expressions, generating cartoon characters that match the user's real-time expressions.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer equipment, and storage medium for processing three-dimensional cartoon characters. Background Technology
[0002] With the continuous development of computer image processing technology, a technique for 3D reconstruction based on information from 2D images has emerged. 3D reconstruction is increasingly widely used in various scenarios, such as creating corresponding cartoon characters based on a user's real face using 3D reconstruction.
[0003] In related technologies, users are typically required to input a facial image. Then, a corresponding 3D facial model is constructed based on the location information of facial feature points detected in the input image, generating a 3D cartoon face with a fixed expression. However, the fixed expressions of the cartoon faces constructed in this way differ significantly from the real expressions of real faces, resulting in low accuracy. Summary of the Invention
[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for processing three-dimensional cartoon images that can accurately generate three-dimensional cartoon images with a high degree of matching with the expressions of real human faces, in order to address the above-mentioned technical problems.
[0005] A method for processing three-dimensional cartoon characters, the method comprising:
[0006] Obtain a basic 3D cartoon image after facial expression removal;
[0007] When the target user is within the camera's field of view and the target user has an expression, a 3D cartoon image of the target user, derived from the base 3D cartoon image, is displayed; the expression of the 3D cartoon image matches the expression of the target user.
[0008] When the target user's facial expression changes within the camera's field of view, a 3D cartoon image that is synchronized with the target user's facial expression is displayed, based on the changes in the base 3D cartoon image.
[0009] A three-dimensional cartoon character processing device, the device comprising:
[0010] The 3D cartoon character acquisition module is used to acquire a basic 3D cartoon character after facial expressions have been removed.
[0011] A 3D cartoon character display module is used to display a 3D cartoon character derived from the base 3D cartoon character when the target user is within the camera's field of view and the target user has an expression; the expression of the 3D cartoon character matches the expression of the target user.
[0012] The 3D cartoon image acquisition module is also used to display a 3D cartoon image that is synchronized with the target user's expression, based on the changes in the basic 3D cartoon image, when the target user's expression changes within the camera's field of view.
[0013] In one embodiment, the aforementioned 3D cartoon character processing device further includes a face scanning module for displaying a face scanning interface; the 3D cartoon character display module is also used to capture the current face image of the target user when the target user is within a first preset acquisition area in the face scanning interface; the first preset acquisition area is an area in the face scanning interface that is located within the camera's field of view; if the target user in the current face image has an expression, then the target user's current face image and 3D cartoon character are displayed in the cartoon character display interface; the 3D cartoon character is obtained by controlling the changes of the basic 3D cartoon character based on the target user's expression.
[0014] In one embodiment, the cartoon character display interface includes a second preset acquisition area and a character display area; the current face image is displayed in the second preset acquisition area; the three-dimensional cartoon character is displayed in the character display area; the second preset acquisition area is the area in the cartoon character display interface that is located within the camera's field of view.
[0015] In one embodiment, the 3D cartoon character display module is further configured to, when the target user's facial expression changes within the second preset acquisition area, follow the target user's facial expression changes and display a 3D cartoon character in the character display area that is synchronized with the target user's facial expression, based on the changes in the basic 3D cartoon character.
[0016] In one embodiment, the 3D cartoon image acquisition module is further configured to acquire an input face image and generate a 3D cartoon image corresponding to the face image; the 3D cartoon image has the face image in the face image; the 3D cartoon image display module is further configured to display a basic 3D cartoon image obtained by removing facial expressions from the 3D cartoon image.
[0017] In one embodiment, the above-mentioned three-dimensional cartoon character processing device further includes a three-dimensional cartoon character generation module, which is used to generate three-dimensional facial features corresponding to the face image through a trained facial feature prediction network; map the three-dimensional facial features to a three-dimensional cartoon face template to obtain a three-dimensional cartoon face model; add the image features from the face image to the three-dimensional cartoon face model to generate a three-dimensional cartoon character corresponding to the face image.
[0018] In one embodiment, the face feature prediction network is obtained through a model training step. The aforementioned 3D cartoon character processing device further includes a model training module for acquiring sample data. The sample data includes sample images and facial key points corresponding to the sample images. Using the face feature prediction network to be trained, based on preset 3D face data and the 2D face features of the sample images, the 3D face features corresponding to the sample images are predicted to reconstruct the sample 3D face. The sample 3D face is mapped and rendered onto the sample images to obtain a 2D mapped image. A first loss is determined based on the difference between the key points of the sample 3D face and the facial key points. A second loss is determined based on the pixel difference between the 2D mapped image and the sample image. Based on the first loss and the second loss, the parameters of the face feature prediction network are adjusted and training continues until the training stop condition is met, resulting in a trained face feature prediction network.
[0019] In one embodiment, the sample data further includes the face segmentation result corresponding to the sample image; the model training module is further configured to determine a third loss based on the difference between the two-dimensional mapping image and the face segmentation result; and adjust the parameters of the face feature prediction network and continue training based on the first loss, the second loss and the third loss.
[0020] In one embodiment, the 3D cartoon character generation module is further configured to obtain a facial feature difference matrix corresponding to the 3D cartoon face template; process the 3D facial features according to the facial feature difference matrix; and map the processed 3D facial features onto the 3D cartoon face template to obtain a 3D cartoon face model.
[0021] In one embodiment, the 3D cartoon character generation module is further configured to extract facial key points from the face image; extract image regions corresponding to each local image in the face image based on the facial key points; identify the image features of each image region through a recognition network corresponding to each local image; and add the image features to the 3D cartoon face model to obtain a 3D cartoon character.
[0022] In one embodiment, the 3D cartoon character generation module is further used to adjust the expression coefficients in the 3D cartoon character so that the expression coefficients of the 3D cartoon character match the neutral expression coefficients, thereby obtaining a basic 3D cartoon character after expression removal; and displaying the basic 3D cartoon character obtained after expression removal.
[0023] In one embodiment, the 3D cartoon character generation module is further configured to generate multiple cartoon facial expression bases corresponding to the basic 3D cartoon character; when the target user is within the camera's field of view and the target user has an expression, the module acquires the target user's facial image; performs 3D reconstruction processing on the facial image to obtain corresponding 3D facial expression features; processes the 3D facial expression features according to the cartoon facial expression bases; and maps the processed 3D facial expression features onto the basic 3D cartoon character to obtain and display a 3D cartoon character that matches the target user's expression.
[0024] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the three-dimensional cartoon character processing method of the various embodiments of this application.
[0025] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the three-dimensional cartoon character processing method of the various embodiments of this application.
[0026] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium; a processor of the computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements the steps in the three-dimensional cartoon character processing method of the various embodiments of this application.
[0027] The aforementioned 3D cartoon image processing method, apparatus, computer equipment, and storage medium acquire a basic 3D cartoon image constructed from a real human face image and after expression removal. When the target user is within the camera's field of view and the target user has an expression, the basic 3D cartoon image is deformed based on the expression in the target user's face image. This effectively generates and displays a 3D cartoon image that matches the target user's expression. When the target user's expression changes within the camera's field of view, the basic 3D cartoon image is deformed in real time based on the change in the target user's face image, resulting in a 3D cartoon image that is synchronized with the target user's expression. This effectively generates and displays a 3D cartoon image that matches the user's real-time expression after deformation of the basic 3D cartoon image. Therefore, the basic 3D cartoon image after expression removal can be effectively driven in real time according to the target user's real-time expression, thus accurately generating a 3D cartoon image with a high degree of matching with the expression of a real human face. Attached Figure Description
[0028] Figure 1 This is an application environment diagram of a three-dimensional cartoon character processing method in one embodiment;
[0029] Figure 2 This is a flowchart illustrating a method for processing 3D cartoon characters in one embodiment;
[0030] Figure 3 This is a schematic diagram of a 3D cartoon image after changes to a basic 3D cartoon image in one embodiment;
[0031] Figure 4 This is a schematic diagram of a face scanning interface in one embodiment;
[0032] Figure 5 This is a schematic diagram of a cartoon character display interface in one embodiment;
[0033] Figure 6 This is a schematic diagram illustrating the interface for adjusting a 3D cartoon character in one embodiment;
[0034] Figure 7 This is a schematic diagram illustrating a user cartoon avatar interface in one embodiment;
[0035] Figure 8 This is a schematic diagram illustrating the generation of a basic 3D cartoon character with facial expression removal in one embodiment;
[0036] Figure 9 This is a schematic diagram of the process of training a face feature prediction network in one embodiment;
[0037] Figure 10This is a schematic diagram of a 3D cartoon face template and a 3D simulated face template in one embodiment;
[0038] Figure 11 This is a schematic diagram illustrating the generation of a 3D cartoon face model in one embodiment;
[0039] Figure 12 This is a schematic diagram of a color chart for a cartoon character in one embodiment;
[0040] Figure 13 This is a schematic diagram illustrating the eyelash deformation process in one embodiment.
[0041] Figure 14 This is a schematic diagram of the cartoon face expression base corresponding to the 3D cartoon face template in one embodiment;
[0042] Figure 15 This is a flowchart illustrating the generation of a 3D cartoon character that matches the facial expression of a target user, as shown in one embodiment.
[0043] Figure 16 This is a structural block diagram of a three-dimensional cartoon character processing device in one embodiment;
[0044] Figure 17 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0046] The 3D cartoon character processing method provided in this application can be applied to computer equipment. The computer equipment can be a terminal or a server. It is understood that the 3D cartoon character processing method provided in this application can be applied to a terminal, a server, or a system including both a terminal and a server, and can be implemented through interaction between the terminal and the server.
[0047] The 3D cartoon character processing method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 uploads a facial image to server 104, which then generates a basic 3D cartoon image corresponding to the facial image after removing facial expressions and returns it to terminal 102. When the target user is within the camera's field of view and the target user has an expression, terminal 102 captures the target user's facial image with the current expression and uploads it to server 104. Server 104 then deforms the basic 3D cartoon image based on the expression in the current facial image, obtaining a deformed 3D cartoon image that matches the target user's expression and returns it to terminal 102. Terminal 102 then displays the 3D cartoon image obtained based on the deformation of the basic 3D cartoon image; the expression of the 3D cartoon image matches the target user's expression. When the target user's expression changes within the camera's field of view, terminal 102 follows the target user's expression change and displays a 3D cartoon image obtained based on the deformation of the basic 3D cartoon image and synchronized with the target user's expression.
[0048] The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.
[0049] Cloud computing refers to the delivery and usage model of IT (Internet Technology) infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the convergence and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire internet model and enterprise management model.
[0050] It is understood that the 3D cartoon image processing methods in the various embodiments of this application employ computer vision and machine learning technologies from artificial intelligence, effectively enabling the automatic generation of 3D cartoon images similar to real human faces. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions.
[0051] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision researches related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. This application, therefore, uses computer vision technology to cartoonize facial images to generate 3D cartoon characters that resemble real human faces.
[0052] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. It can be understood that the face feature prediction network and recognition network used in some embodiments of this application are trained using machine learning techniques. Based on this machine learning training, they can more accurately predict three-dimensional facial features in face images and recognize image features.
[0053] In one embodiment, such as Figure 2 As shown, a method for processing three-dimensional cartoon characters is provided, which can be applied to... Figure 1 This embodiment will be illustrated using a terminal as an example. The steps included are as follows:
[0054] Step S202: Obtain the basic 3D cartoon image after facial expression removal.
[0055] It can be understood that a 3D cartoon character is a non-realistic cartoon image with both cartoonish and realistic facial features. Specifically, a 3D cartoon character is generated by cartoonizing a real human face image. The resulting 3D cartoon character thus possesses the facial features found in the original face image.
[0056] Among them, real face images are images of faces captured in real-world scenarios. They are two-dimensional face images that include the user's face.
[0057] As we can understand, facial expressions usually refer to the facial expressions and postures of a human face, which can be produced by changes in facial features. The facial expressions of a 3D cartoon character refer to the facial expressions and postures corresponding to that 3D cartoon character.
[0058] Expression removal can refer to the process of transforming the facial expressions of a 3D cartoon character into neutral expressions, resulting in a visually expressionless 3D cartoon character. This expressionless state can also refer to a state with neutral expressions. Neutral expressions can mean that the changes in facial expressions and postures are within a preset range, such as changes in facial features being within a preset range, specifically, the mouth being closed and the face being in a frontal posture.
[0059] The terminal first acquires a face image, then performs 3D face reconstruction processing based on the face image to obtain a 3D cartoon face model corresponding to the face image. Next, it adds the facial features from the face image to the 3D cartoon face model and performs expression removal processing, thus obtaining a basic 3D cartoon image with expression removal.
[0060] It is understandable that the initially generated 3D cartoon image corresponding to the face image has a facial image and expression image that match the face image. Among them, the facial image corresponds to the basic facial features in the face image, and the expression image corresponds to the facial expression features in the face image.
[0061] Since the generated 3D cartoon face model has the facial features of a real face image, the image features of the cartoon face model are further rendered according to the image features to render the image features of the real face image into the cartoon face model, thereby effectively generating a 3D cartoon image similar to a real face image.
[0062] In one embodiment, the terminal obtains a basic 3D cartoon image after removing facial expressions and displays the basic 3D cartoon image on the terminal's interface.
[0063] It is understandable that the basic 3D cartoon image after facial expression removal is used to respond to the facial image with facial expression input by the target user. The basic 3D cartoon image is then subjected to expression-driven processing, thereby deforming the basic 3D cartoon image to obtain a 3D cartoon image that matches the facial expression of the target user.
[0064] Step S204: When the target user is within the camera's field of view and the target user has an expression, display the three-dimensional cartoon image of the target user obtained by changing the basic three-dimensional cartoon image.
[0065] It is understandable that "target user within the camera's field of view" refers to the image captured within the field of view of the terminal's camera device. "Target user exhibiting facial expressions" can mean that changes in the target user's facial features or posture exceed a preset range, indicating that the target user is exhibiting facial expressions.
[0066] It is understandable that the target user, the user in the facial image corresponding to the basic 3D cartoon character, can be the same user or a different user.
[0067] After the terminal acquires a basic 3D cartoon image with facial expressions removed, the target user can input a facial image with expressions in real time via the terminal. It can be understood that the terminal has a corresponding field of view when capturing images. The field of view refers to the area of the camera's field of view when the terminal captures images.
[0068] Specifically, the target user can have their facial image captured through the corresponding terminal. During the image capture process, when the target user is within the camera's field of view and is displaying an expression, the terminal will capture the target user's facial image.
[0069] The terminal then extracts the corresponding 3D facial expression features from the acquired facial images, maps these features onto a basic 3D cartoon character, and deforms the character to obtain a transformed 3D cartoon character. This transformed 3D cartoon character possesses the facial expression features from the real-time input facial image. Therefore, the resulting 3D cartoon character's expression matches the target user's expression.
[0070] Step S206: When the target user's facial expression changes within the camera's field of view, a 3D cartoon image that is synchronized with the target user's facial expression is displayed, based on the changes in the basic 3D cartoon image.
[0071] Among them, expression change can refer to the change in the expression of the target user's face in the current captured image compared to the expression in the previous captured image; or the change in the expression in the current face image compared to the previous face image exceeding a preset range, which indicates that the target user's expression has changed within the camera's field of view.
[0072] It is understandable that the target user can continuously capture their corresponding facial images in real time through the terminal. For example, corresponding facial image frames can be captured continuously.
[0073] Specifically, during the process of acquiring the target user's facial image, if the target user's expression changes within the camera's field of view, the terminal acquires the facial image after the expression change.
[0074] The terminal extracts the 3D facial expression features corresponding to the changed facial image, then maps these features onto a basic 3D cartoon character. This basic 3D cartoon character is then deformed to obtain the transformed 3D cartoon character, which is then displayed. The transformed 3D cartoon character's facial expression matches that of the target user.
[0075] Therefore, the terminal can capture facial images after facial expression changes in real time, and follow the target user's facial expression changes to display a 3D cartoon image that is based on the changes of the basic 3D cartoon image and keeps synchronized with the target user's facial expression.
[0076] For example, such as Figure 3 The diagram illustrates a 3D cartoon image derived from a base 3D cartoon image in one embodiment. The base 3D cartoon image 32 is generated from an initial facial image. When the target user is within the camera's field of view and displays an expression, the target user's current facial image 34 is captured. Based on the 3D facial expression features in the facial image 34, the base 3D cartoon image 32 is deformed to obtain a modified 3D cartoon image 36 that matches the target user's expression. It is understood that, to protect the privacy of the user's real facial image, the eyes in the real facial image 34 are occluded; in fact, the eyes are also captured when the real facial image is acquired.
[0077] In the aforementioned 3D cartoon image processing method, the terminal first acquires a basic 3D cartoon image constructed from a real human face image and after expression removal. When the target user is within the camera's field of view and the target user has an expression, the basic 3D cartoon image is deformed based on the expression in the target user's face image. This effectively generates and displays a 3D cartoon image that matches the target user's expression. When the target user's expression changes within the camera's field of view, the basic 3D cartoon image is deformed in real time based on the change in the target user's face image, resulting in a 3D cartoon image that is synchronized with the target user's expression. This effectively generates and displays a 3D cartoon image that matches the user's real-time expression after deformation of the basic 3D cartoon image. Therefore, the basic 3D cartoon image after expression removal can be effectively driven in real time according to the target user's real-time expression, thus accurately generating a 3D cartoon image with a high degree of matching with the expression of a real human face.
[0078] In one embodiment, the above-mentioned 3D cartoon image processing method further includes: displaying a face scanning interface. When the target user is within the camera's field of view and the target user has an expression, a 3D cartoon image of the target user based on changes to a basic 3D cartoon image is displayed, including: when the target user is within a first preset acquisition area in the face scanning interface, acquiring the target user's current face image; if the target user in the current face image has an expression, then displaying the target user's current face image and the 3D cartoon image on the cartoon image display interface.
[0079] It can be understood that a face scanning interface is a display interface used to show the images captured by the terminal's camera device within the camera's field of view. The face scanning interface may also include a first preset acquisition area, which is an area within the face scanning interface and located within the camera's field of view, used to capture the target user's face within the face scanning interface. Specifically, the first preset acquisition area can be a face detection area.
[0080] The terminal can also display a face scanning interface, showing the image captured by the terminal's camera. The terminal can also detect whether the target user's face is within the camera's field of view through a first preset acquisition area in the face scanning interface. When the target user's face is within the first preset acquisition area in the face scanning interface, the terminal captures the target user's current face image.
[0081] Furthermore, if an expression is detected in the target user's current facial image, the basic 3D cartoon character is subjected to expression deformation processing based on the expression in the current facial image to generate a 3D cartoon character that matches the expression in the current facial image. Both the current facial image and the 3D cartoon character are then displayed simultaneously on the cartoon character display interface. It can be understood that the 3D cartoon character is obtained by controlling the changes in the basic 3D cartoon character based on the target user's facial expression.
[0082] like Figure 4 The diagram shown illustrates a face scanning interface in one embodiment. The face scanning interface 42 includes a face detection area 4a, which is the first preset acquisition area. During the process of the terminal acquiring the target user's current face image through the camera device, if it detects whether the target user's face is within the face detection area 4a, then the target user's current face image is acquired. If the current face image contains an expression, the basic 3D cartoon image is further modified based on the target user's expression to obtain a modified 3D cartoon image.
[0083] In one embodiment, the cartoon character display interface includes a second preset acquisition area and a character display area; the current face image is displayed in the second preset acquisition area; the three-dimensional cartoon character is displayed in the character display area; the second preset acquisition area is the area in the cartoon character display interface that is located within the camera's field of view.
[0084] It is understood that the cartoon character display interface can include two parts: a second preset acquisition area and a character display area. The second preset acquisition area can be used to display the images captured by the camera device within the camera's field of view; the character display area is used to display the generated 3D cartoon character.
[0085] Specifically, the terminal collects the target user's current face image in the face scanning interface, and generates a 3D cartoon image that matches the expression in the current face image based on the basic 3D cartoon image. Then, it jumps to the cartoon image display interface, displays the target user's current face image in the second preset collection area of the cartoon image display interface, and displays the generated 3D cartoon image in the image display area of the cartoon image display interface.
[0086] like Figure 5 The diagram illustrates a cartoon character display interface in one embodiment. The cartoon character display interface 52 includes a second preset acquisition area 52a and a character display area 5ab. The terminal displays the captured current facial image of the target user in the second preset acquisition area 52a, and displays the generated 3D cartoon character in the character display area 5ab. In one embodiment, the terminal can continuously display the target user's facial image captured in real-time by the camera device in the second preset acquisition area 52a, and simultaneously display the generated 3D cartoon character in the character display area 5ab that matches the expression in the target user's facial image.
[0087] In this embodiment, by simultaneously displaying the target user's current facial image and the corresponding generated 3D cartoon image in the second preset acquisition area and image display area of the cartoon image display interface, it is possible to effectively display a 3D cartoon image that matches the expression in the current facial image, and effectively visually represent the generated 3D cartoon image with a high degree of matching with the expression in the real facial image.
[0088] In one embodiment, when the target user's facial expression changes within the camera's field of view, a 3D cartoon image that is based on changes in a basic 3D cartoon image and remains synchronized with the target user's facial expression is displayed, following the changes in the target user's facial expression. This includes: when the target user's facial expression changes within a second preset acquisition area, a 3D cartoon image that is based on changes in a basic 3D cartoon image and remains synchronized with the target user's facial expression is displayed in the image display area, following the changes in the target user's facial expression.
[0089] The terminal can capture continuous facial images of a target user in real time using a camera device. For example, it can continuously capture facial image frames corresponding to the target user.
[0090] During the process of acquiring the target user's facial image, if the target user's facial expression changes within the camera's field of view, the terminal acquires the current facial image after the expression change in real time. It then maps the 3D facial expression features from the current facial image onto a basic 3D cartoon image for deformation processing, outputting a 3D cartoon image synchronized with the expression in the current facial image. The cartoon image display interface simultaneously displays the real-time acquired current facial image and the real-time output 3D cartoon image synchronized with the target user's expression.
[0091] by Figure 5 Taking the second preset acquisition area 52a and the image display area 5ab included in the cartoon character display interface 52 as an example, when a face image with a changed expression is acquired in the second preset acquisition area 52a, a three-dimensional cartoon character synchronized with the expression in the face image is displayed in real time in the image display area 5ab.
[0092] Thus, the terminal can capture facial images of the target user in real time during the process of facial expression changes, and dynamically display a 3D cartoon image that is based on the changes of the basic 3D cartoon image and keeps synchronized with the target user's facial expression. In other words, it can display a 3D cartoon image animation in which the 3D cartoon image's expression follows the changes of the target user's facial expression.
[0093] In one embodiment, after the terminal constructs and saves a 3D cartoon avatar corresponding to a real facial image, the user can further customize and dress up the generated 3D cartoon avatar. Specifically, the terminal can display a 3D cartoon avatar adjustment interface to allow for customized adjustments to the 3D cartoon avatar. Figure 6The diagram illustrates a 3D cartoon character adjustment interface in one embodiment. The 3D cartoon character adjustment interface 62 includes an adjustment preview area 62a and a material selection area 62b. The material selection area 62b includes various facial features and other corresponding materials. Users can select appropriate materials in the material selection area 62b and add them to the corresponding parts of the 3D cartoon character, or replace the materials in the corresponding parts of the 3D cartoon character. The adjusted 3D cartoon character is then displayed in the adjustment preview area 62a. This effectively enables customized adjustments to automatically generated 3D cartoon characters, significantly improving their editability and adaptability.
[0094] In one embodiment, after constructing and saving a 3D cartoon avatar corresponding to a real facial image, the terminal can also display the final generated complete 3D cartoon avatar on the user's cartoon avatar interface. The complete 3D cartoon avatar may include a facial image and a body structure and costume. For example... Figure 7 The image shown is a schematic diagram illustrating a user cartoon avatar interface in one embodiment. (Refer to...) Figure 7 The complete 3D cartoon character 72a displayed in the user cartoon character interface 72 includes the facial image and the body structure and costume image.
[0095] In this embodiment, by synchronously displaying the real-time captured current face image and the real-time output of a 3D cartoon image synchronized with the target user's expression on the same cartoon image display interface, it is possible to effectively and dynamically display a 3D cartoon image that is synchronized with the expression in the target user's face image and highly matched with the target user's expression.
[0096] In one embodiment, obtaining a basic 3D cartoon image after facial expression removal includes: obtaining an input face image, generating a 3D cartoon image corresponding to the face image, and displaying the basic 3D cartoon image obtained by facial expression removal processing of the 3D cartoon image.
[0097] It is understood that the generated 3D cartoon character possesses a facial image from a human face image. Here, a human face image refers to a face image including a real human face. The input human face image can be a real-time photograph of a face taken by the terminal's camera device, or a pre-collected human face image including a face obtained by the terminal from its local storage or the internet. In this embodiment, the human face image can be a 2D human face image with any expression and any pose.
[0098] The terminal can first acquire a face image, then perform 3D face reconstruction processing based on the face image to obtain 3D face features corresponding to the face image, and then map the 3D face features to a preset 3D cartoon face template to obtain a 3D cartoon face model corresponding to the face image.
[0099] Then, the terminal adds facial features from the image to the 3D cartoon face model and performs expression removal processing on the 3D cartoon face model, thus obtaining a basic 3D cartoon image with expression removal.
[0100] Specifically, the terminal can first add facial features from a human face image to a 3D cartoon face model to obtain a corresponding 3D cartoon image, and then perform expression removal processing on the 3D cartoon image to obtain a basic 3D cartoon image with expression removal. In another embodiment, the terminal can also first perform expression removal processing on the 3D cartoon face model to obtain an expression-removed 3D cartoon face model, and then add facial features from the human face image to the expression-removed 3D cartoon face model to obtain a basic 3D cartoon image with expression removal.
[0101] like Figure 8 The image shown is a schematic diagram illustrating the generation of a basic 3D cartoon character with facial expression removal in one embodiment. (Refer to...) Figure 8 After acquiring the initial face image 82, the terminal performs 3D face reconstruction based on the face image 82 to obtain the 3D face features and 3D face model 84 corresponding to the face image 82. Then, the terminal maps the 3D face features to a 3D cartoon face template to obtain the corresponding 3D cartoon face model, and adds image features from the face image to obtain a 3D cartoon image 86 corresponding to the face image 82. The terminal then performs expression removal processing on the 3D cartoon image 86 to obtain a basic 3D cartoon image 86 with expression removal.
[0102] In this embodiment, after first generating a 3D cartoon image corresponding to a human face image, the 3D cartoon image undergoes expression removal processing to obtain a basic 3D cartoon image after expression removal. This allows the target user to effectively drive the basic 3D cartoon image with facial expressions, thereby deforming the basic 3D cartoon image to obtain a 3D cartoon image that matches the target user's facial expression.
[0103] In one embodiment, generating a 3D cartoon image corresponding to a face image includes: generating 3D face features corresponding to the face image through a trained face feature prediction network; mapping the 3D face features to a 3D cartoon face template to obtain a 3D cartoon face model; and adding image features from the face image to the 3D cartoon face model to generate a 3D cartoon image corresponding to the face image.
[0104] It is understood that a face feature prediction network is a trained deep neural network capable of predicting the three-dimensional face features corresponding to a two-dimensional face image. This face feature prediction network can be trained using a machine learning model based on ResNet (Deep Residual Convolutional Network). In some embodiments, the face feature prediction network can also be trained using machine learning models based on deep networks such as CNN (Convolutional Neural Networks), DenseNet (Dense Convolutional Network), and DPN (Dual Path Network), but this application does not limit the specific methods used.
[0105] Among them, 3D facial features refer to the features corresponding to a 3D facial model. Specifically, 3D facial features can include 3D facial feature basis vectors and their corresponding eigenvalues, where the eigenvalues are the coefficients of each basis vector. The basic attributes of a face include shape, expression, and texture. Similarly, 3D facial features can include 3D facial shape features, 3D facial expression features, and 3D facial texture features.
[0106] It is understandable that in the process of reconstructing a 3D face based on a 2D face image, 3D vertices are typically used to represent the face. The 3D face features of a 3D face model can be matched in 3D space and can be generated by weighted linear addition of multiple orthogonal face bases. Each 3D face model can be represented by the linear superposition of a 3D face shape vector, a 3D face expression vector, and a 3D face texture vector.
[0107] For example, each point (x, y, z) in 3D space is obtained by weighted summation of the basis vectors (1, 0, 0), (0, 1, 0), and (0, 0, 1) in three directions, with weights of x, y, and z, respectively. Any 3D face model can be obtained by weighted combination of multiple face models in a dataset. Predicting the 3D face features of a 3D face model essentially involves predicting the coefficients of each basis vector. After predicting the 3D face features corresponding to a face image, a 3D face model matching the face in the face image can be directly reconstructed based on the obtained 3D face features.
[0108] The terminal can pre-train a face feature prediction network using a large amount of sample data, enabling the trained face feature prediction network to predict the three-dimensional face features corresponding to a two-dimensional face image.
[0109] Specifically, after the terminal acquires the target user's current face image, it performs feature extraction on the current face image to extract two-dimensional face features. These two-dimensional face features are then input into a trained face feature prediction network, which predicts the corresponding three-dimensional face features based on the two-dimensional features and outputs the prediction.
[0110] In this context, the face feature prediction network, when predicting corresponding 3D face features based on 2D face features, can use pre-trained preset 3D face data to predict the 3D face features corresponding to the 2D face features. This preset 3D face data, also known as 3D face prior data, can be a 3D simulated face template obtained through fitting and analysis of a large amount of 3D face data, along with corresponding face shape basis vectors, face expression basis vectors, average texture, and texture basis vectors. In other words, the 3D simulated face template can also be a 3D average face model obtained through fitting and analysis of a large amount of 3D face data. During prediction, the face feature prediction network only needs to predict the corresponding coefficients under each face shape basis vector, face expression basis vector, and texture basis vector corresponding to the 2D face features, thereby obtaining the 3D face features corresponding to the 2D face features.
[0111] The terminal then maps the 3D facial features onto a 3D cartoon face template to generate a 3D cartoon face model corresponding to the current facial image. The terminal can also extract facial features from the face image.
[0112] It is understandable that image features refer to the image characteristics of a real face in a real face image, such as the user's hair, birthmarks, moles, eyebrows, eyes, glasses, and other image-related features in the face image.
[0113] Then, the terminal adds facial features from the face image to the 3D cartoon face model, thereby generating a 3D cartoon image corresponding to the face image. This enables the accurate and effective generation of 3D cartoon images that resemble face images.
[0114] In one embodiment, the face feature prediction network is obtained through a model training step, which includes: acquiring sample data; using the face feature prediction network to be trained, based on preset 3D face data and 2D face features of sample images, predicting the 3D face features corresponding to the sample images to reconstruct the sample 3D face; mapping and rendering the sample 3D face onto the sample image to obtain a 2D mapped image; determining a first loss based on the difference between the key points of the sample 3D face and the face key points; determining a second loss based on the pixel difference between the 2D mapped image and the sample image; adjusting the parameters of the face feature prediction network based on the first loss and the second loss and continuing training until the training stopping condition is met to obtain a trained face feature prediction network.
[0115] The sample data includes sample images and corresponding facial landmarks. Sample images are two-dimensional images containing real human faces; the corresponding facial landmarks are the facial landmarks obtained by performing facial landmark detection processing on the sample images.
[0116] It is understandable that before training a facial feature prediction network, a large amount of sample 3D facial data needs to be acquired to train 3D facial prior data, i.e., preset 3D facial data. Specifically, a preset number of large amounts of 3D facial data can be obtained from a preset database, or a large amount of pre-collected 3D facial data can be obtained.
[0117] For example, 1000 3D face data points can be obtained from a specific BFM (Basel Face Model, a 3D face model dataset) database; another 1000 pre-collected 3D face data points can also be obtained, resulting in 2000 3D face data points used to construct the required sample 3D face data. Each 3D face data point includes multiple vertices, such as 53215 vertices. The terminal can preprocess the large amount of acquired 3D face data to construct a model dataset with a unified vertex and network topology structure.
[0118] The sample 3D face data includes 3D face feature data processed by feature analysis, such as PCA (Principal Component Analysis). The processed 3D face feature data can be represented by the following expression:
[0119] ;
[0120] Among them, S 3DLet M represent the average face data, V represent the 3D face shape feature vector, and E represent the feature value corresponding to the 3D face shape feature vector. Specifically, the 3D face structure data can be reconstructed by parameterizing the feature values, for example, by randomly sampling to obtain the positive and negative standard deviations of the feature values.
[0121] First, a preset number of vertices are selected from the 3D face structure data obtained from a specific database. For example, 32,000 mesh vertices of the facial region are selected from the head mesh, and the vertex data of the neck and ear regions are removed to obtain a simple connected facial topology structure composed of 32,000 vertices.
[0122] Then, a neutral face model is obtained from a pre-defined database, such as the BFM database. Based on pre-collected 3D face data, the neutral face model is deformed to obtain the corresponding deformed 3D face model data. For example, non-rigid registration can be used to deform the neutral face model to correspond to 1000 collected 3D face model data, thus obtaining 3D average face model data with a topological structure consistent with 1000 face models in a specific database. This allows the construction of a face dataset containing 2000 3D average face model data.
[0123] For example, the terminal can obtain an average face model from a preset dataset, specifically from the FACEWAREHOUSE database. A triangular mesh deformation transfer method can be used to transfer multiple expression models corresponding to the average face model (e.g., 46 expression models) to 2000 neutral face models, thereby constructing 47 expression bases for each 3D face model data. In this way, after deforming the 2000 3D average face model data in the face dataset, each 3D average face model data has corresponding 47 expression bases. Alignment is then performed on all vertex data in the dataset using Protodyakonov analysis. Furthermore, the neutral face model in the BFM database includes 199 principal components to represent albedo information, specifically albedo components that span the surface albedo space. Because the mesh connectivity of the neutral face model in the BFM database differs from that of the average face in the FACEWAREHOUSE database—that is, the structure of the 3D face mesh model is different—this principal albedo information cannot be directly used in the newly generated 3D face model data. Therefore, a non-rigid registration method can be used to determine the correspondence, transferring the albedo information of the neutral face model to the mesh representation of the newly generated 3D face model data. This allows for the construction of multiple principal albedo components for each 3D average face model in the face dataset, for example, 199 principal albedo components. This results in a face dataset comprising 2000 3D average face model data points, each with 47 expression bases.
[0124] Then, the 3D facial features corresponding to the 3D average face model data are trained. These 3D facial features can include 3D face shape vectors, 3D face expression vectors, and texture expression vectors.
[0125] Specifically, firstly, based on multiple 3D average face model data in the face dataset, the average face model corresponding to all 3D face model data is calculated. The average face model can be understood as the 3D simulated face template. The expression for the 3D simulated face template can be as follows:
[0126] ;
[0127] in, Represented as a 3D simulated human face template, S i This represents the data of each 3D average face model in the face dataset, where N represents the number of 3D average face models in the face dataset. For example, N can be 2000.
[0128] Then, subtract the 3D simulated face template data from each 3D average face model data in the face dataset to obtain the difference between each 3D average face model data and the 3D simulated face template. For example, this can be represented by the following expression: ; where △S i This indicates the difference between the average 3D face model data and the 3D simulated face template.
[0129] Then, principal component analysis can be used to compress the differential data, obtaining the covariance matrix of the face shape corresponding to each 3D average face model data, and the orthogonal basis formed by the eigenvectors of the corresponding covariance matrix, which can be expressed by the following expression:
[0130] ;
[0131] in, ; ;
[0132] Among them, S model S represents the orthogonal basis corresponding to each 3D average face model data; i Let represent the eigenvector of the covariance matrix corresponding to the face shape covariance matrix; α is the face shape basis coefficient; p(α) is the probability of α; and σ is the eigenvalue of the face shape covariance matrix. i represents the nth 3D average face model data in the face dataset, and m represents the number of eigenvalues.
[0133] This allows us to obtain a 3D simulated face template composed of the 3D average face model data from the face dataset. Set of facial shape feature vectors and the corresponding set of eigenvalues , where n represents the number of vertices in each 3D average face model data.
[0134] Similarly, to obtain expression fitting data, multiple expression bases are constructed for each 3D average face model in the face dataset; for example, there could be 47 expression bases. Similar to the method described above for constructing the set of face shape feature vectors, facial expression base vectors and facial texture vectors are constructed for each 3D average face model.
[0135] After the terminal constructs the required 3D average face model data, corresponding to the face shape feature vector, face expression basis vector, and face texture vector, it constructs a face feature prediction network based on this 3D average face model data and trains the face feature prediction network.
[0136] Specifically, the terminal can acquire sample data for training the face feature prediction network. This sample data includes two-dimensional sample images and corresponding facial landmarks. The facial landmarks in the sample images can be obtained by pre-processing the sample images with landmark detection. The sample images and facial landmarks can serve as training labels. Unsupervised training can be used to train the face feature prediction network.
[0137] The terminal inputs sample images into the face feature prediction network to be trained, or it can input two-dimensional face features extracted from the sample images. Then, based on pre-constructed preset three-dimensional face data, the face feature prediction network predicts the corresponding three-dimensional face features from the sample images, according to the two-dimensional face features in the sample images. These three-dimensional face features can include face shape parameters, face expression coefficients, face texture coefficients, pose parameters, texture parameters, and illumination parameters. A corresponding sample three-dimensional face is then constructed based on the predicted three-dimensional face features. This sample three-dimensional face also contains facial landmarks.
[0138] The terminal then projects the generated sample 3D face onto a 2D plane, and then maps and renders the projected 2D sample 3D face onto the corresponding 2D sample image. For example, a differentiable renderer can be used to map the reconstructed sample 3D face onto the 2D plane. Specifically, based on the correspondence between the facial key points of the sample 3D face and the corresponding facial key points in the sample image, the sample 3D face can be mapped and rendered onto the corresponding 2D sample image to obtain the mapped 2D image.
[0139] The terminal then compares the key points of the 3D face in the sample image with the corresponding key points in the sample image. Based on the differences between the key points, it determines the first loss, i.e., the key point loss. Simultaneously, the terminal compares the 2D mapped image with the sample image to determine the pixel differences between them. Based on these pixel differences, it determines the second loss, i.e., the pixel consistency loss, thereby obtaining pixel-level supervision signals. The terminal then adjusts the parameters of the face feature prediction network based on the first and second losses and continues training. Training stops when the training stopping condition is met, resulting in a well-trained face feature prediction network.
[0140] In this embodiment, during the training of the face feature prediction network, after predicting the three-dimensional face features corresponding to the sample image and reconstructing the sample three-dimensional face using the face feature prediction network, the sample three-dimensional face is mapped and rendered onto the two-dimensional sample image to obtain a two-dimensional mapped image. Then, the first loss corresponding to the difference between the key points of the sample three-dimensional face and the face key points is calculated, and the second loss corresponding to the pixel difference between the two-dimensional mapped image and the sample image is calculated. Then, based on the first loss and the second loss, the parameters of the face feature prediction network are adjusted and training continues. This effectively utilizes the face key points and pixel-level supervision signals to train the face feature prediction network, thereby effectively improving the accuracy of the face feature prediction network.
[0141] In one embodiment, the above model training step further includes: determining a third loss based on the difference between the two-dimensional mapped image and the face segmentation result. The step of adjusting the parameters of the face feature prediction network and continuing training based on the first and second losses includes: adjusting the parameters of the face feature prediction network and continuing training based on the first, second, and third losses.
[0142] The sample data also includes the face segmentation results corresponding to the sample images. It can be understood that the face segmentation results corresponding to the sample images can be obtained by pre-segmenting the sample images, and these face segmentation results can be used as training labels in the training data.
[0143] like Figure 9 The diagram illustrates the process of training a face feature prediction network in one embodiment. During training, the terminal uses the network to predict the 3D face features corresponding to the sample image and reconstruct the corresponding 3D face. After determining the first and second losses, it compares the 2D mapped image with the face segmentation result, calculating the difference between the face region in the 2D mapped image and the face region in the face segmentation result to determine the third loss, the face region loss. The terminal then adjusts the parameters of the face feature prediction network based on the first, second, and third losses and continues training. Training stops when the training termination condition is met, resulting in a well-trained face feature prediction network.
[0144] In this embodiment, during the training of the face feature prediction network, the first loss corresponding to the difference between the key points of the sample 3D face and the face key points, the second loss corresponding to the pixel difference between the 2D mapping image and the sample image, and the third loss corresponding to the difference between the 2D mapping image and the face segmentation result are calculated. Then, the parameters of the face feature prediction network are adjusted according to these three losses. This can effectively utilize the supervision signals of face key points, pixel level, and face region to train the face feature prediction network, thereby effectively improving the accuracy of the face feature prediction network.
[0145] In one embodiment, mapping three-dimensional facial features to a three-dimensional cartoon face template to obtain a three-dimensional cartoon face model includes: obtaining a facial feature difference matrix corresponding to the three-dimensional cartoon face template; processing the three-dimensional facial features according to the facial feature difference matrix; and mapping the processed three-dimensional facial features to the three-dimensional cartoon face template to obtain a three-dimensional cartoon face model.
[0146] The 3D cartoon face template refers to a pre-defined, universal cartoon face model, also known as a 3D cartoon average face model. Specifically, it can be obtained by averaging the cartoon faces corresponding to a large number of simulated face models. It can be understood that both 3D simulated face templates and 3D simulated face templates are 3D face models with topological structures. Furthermore, there are corresponding index key points between 3D simulated face templates.
[0147] It can be understood that the facial feature difference matrix corresponding to the 3D cartoon face template refers to the feature differences between the multi-dimensional facial features obtained by deforming the 3D cartoon face template and the facial features of the 3D cartoon face template. The facial feature difference matrix corresponding to the 3D cartoon face template may also include index keypoints.
[0148] The 3D facial features can be predicted from the 2D facial features in the facial image based on the 3D prior data corresponding to the 3D simulated facial template. The resulting 3D facial features may also include index keypoints. Therefore, the terminal can process the 3D facial features based on the index keypoints and the facial feature difference matrix, and then map the processed 3D facial features onto the 3D cartoon facial template to obtain a 3D cartoon facial model corresponding to the facial image.
[0149] In order to directly map the obtained 3D facial features onto the 3D cartoon face template, a multi-dimensional 3D facial shape vector basis and a multi-dimensional 3D facial expression vector basis corresponding to the 3D cartoon face template can be pre-constructed.
[0150] Specifically, after the terminal constructs a three-dimensional simulated face template based on a large amount of sample three-dimensional face data, it first constructs a multi-dimensional three-dimensional face shape vector basis and a multi-dimensional three-dimensional face expression vector basis corresponding to the three-dimensional simulated face template.
[0151] It can be understood that each dimension of the 3D face shape vector basis can represent a corresponding 3D face shape model; similarly, each dimension of the 3D face expression vector basis can represent a corresponding 3D face expression model. For example, specifically, there can be an 80-dimensional 3D face shape vector basis and a 64-dimensional 3D face expression vector basis. That is, the 3D simulation face template and the 3D cartoon face template each include the corresponding 80-dimensional 3D face shape vector basis and 64-dimensional 3D face expression vector basis, respectively.
[0152] The terminal first constructs a multi-dimensional 3D face shape vector basis corresponding to the 3D simulated face template. Specifically, it models the principal component vectors of the 3D face shape basis of the 3D simulated face template to generate 80 3D face shape models corresponding to the 3D face shape vectors. The specific expression can be:
[0153] ;
[0154] ;
[0155] ;
[0156] Among them, S new This represents the newly constructed 3D face shape model; V represents a 3D simulated human face template. id This represents the face shape vector machine; i represents the nth dimension, m represents the number of face shape basis vectors, for example, it can be 80; v represents the vertex coordinates; The vertex coordinates of the 3D simulated human face template are represented by t, and the texture annotation is represented by t. 'b' represents the average face texture data; 'b' represents the corresponding texture feature vector data. This allows us to obtain a multi-dimensional 3D face shape model corresponding to the face shape vector, for example, an 80-dimensional 3D face shape model.
[0157] Similarly, the principal component vectors corresponding to the facial expression vector machine in the three-dimensional facial features can be modeled to obtain a three-dimensional facial expression model with multiple dimensions corresponding to the facial expression vector, such as a 64-dimensional three-dimensional facial expression model.
[0158] After obtaining 80 three-dimensional face shape models and 64 three-dimensional face expression models corresponding to the three-dimensional simulated face template, deformation transfer is used to generate 80 three-dimensional cartoon face shape models and 64 three-dimensional cartoon face expression models corresponding to the three-dimensional cartoon face template.
[0159] Deformation transfer can be understood as transferring deformation from the source model to the target model using deformation gradients. The source model consists of 80 3D face shape models and 64 3D face expression models corresponding to a 3D simulated face template, while the target model consists of 80 3D cartoon face shape models and 64 3D cartoon face expression models corresponding to a 3D cartoon face template. Deformation transfer requires pre-marking a set of corresponding key points on both the source and target models; specifically, this can be indexed keypoints. For example, corresponding indexed keypoints can be pre-marked on the 3D simulated face template and the 3D cartoon face template. Figure 10 The diagram shown is a schematic representation of the index key points corresponding to a 3D simulated face template and a 3D cartoon face template in one embodiment. (Refer to...) Figure 10 The 3D simulation face template 10(A) includes index key points, specifically facial key points. The 3D cartoon face template 10(B) has index key points corresponding to those of the 3D simulation face template 10(A).
[0160] The terminal uses a deformation migration algorithm to migrate the 80 three-dimensional face shape models and 64 three-dimensional face expression models corresponding to the three-dimensional simulated face template to the 80 three-dimensional cartoon face shape models and 64 three-dimensional cartoon face expression models corresponding to the three-dimensional cartoon face template.
[0161] Then, the terminal subtracts the vertex data from the 3D cartoon face template from the 80 deformed and transferred 3D cartoon face shape models and 64 3D cartoon face expression models to obtain the face feature difference matrix. The face feature difference matrix includes an 80-dimensional cartoon face shape vector basis and a 60-dimensional cartoon face expression vector basis. The face feature difference matrix can be pre-constructed prior data.
[0162] When generating a 3D cartoon face corresponding to a face image, the terminal can directly map the 3D face features corresponding to the face image to the 3D cartoon face template based on the face feature difference matrix corresponding to the 3D cartoon face template, so as to generate the corresponding 3D face model.
[0163] The facial feature difference matrix includes an 80-dimensional cartoon face shape vector basis S. i (i=0,1,2…80), and a 60-dimensional cartoon face expression vector basis u i(i=0,1…64). Specifically, after the terminal obtains the 3D facial features corresponding to the face image, it can generate a 3D cartoon face model similar to the input face image using the following expression:
[0164] ;
[0165] Where S represents the generated 3D cartoon face model, 3D cartoon face template, S i Represents the vector basis of the cartoon face shape, u i Let represent the vector basis of cartoon face expressions, m represent the dimension of the vector basis of cartoon face shapes, p represent the dimension of the vector basis of cartoon face expressions, and α represent the dimension of the vector basis of cartoon face expressions. i β represents the basic coefficient of facial image in three-dimensional facial features. i This represents the facial expression basis coefficients in 3D facial features. This enables the accurate and efficient generation of 3D cartoon face models that resemble the input face image.
[0166] like Figure 11 The image shown is a schematic diagram illustrating the generation of a 3D cartoon face model in one embodiment. (Refer to...) Figure 11 After acquiring the initial face image 11a, the terminal performs 3D face reconstruction based on the face image 11a to obtain the 3D face features corresponding to the face image 11a and the corresponding 3D face model 11b. Then, the terminal maps the 3D face features to a 3D cartoon face template to obtain the corresponding 3D cartoon face model 11c.
[0167] In this embodiment, by pre-deforming and transferring the multi-dimensional 3D face shape model and 3D face expression model corresponding to the 3D simulated face template, a multi-dimensional 3D cartoon face shape model and 3D cartoon face expression model corresponding to the 3D cartoon face template are constructed to generate prior data for the face feature difference matrix corresponding to the 3D cartoon face template. This allows for the direct mapping of the 3D face features corresponding to the face image to the 3D cartoon face template based on the face feature difference matrix, thus generating the corresponding 3D face model. Therefore, when generating a 3D cartoon face corresponding to a face image, there is no need to perform deformation and transfer processing on the 3D face model, greatly improving the efficiency of generating a 3D face model corresponding to a face image and effectively ensuring the accuracy of the generated 3D face model.
[0168] In one embodiment, adding image features from a face image to a 3D cartoon face model to generate a 3D cartoon image corresponding to the face image includes: extracting facial key points from the face image; extracting image regions corresponding to each local image in the face image based on the facial key points; identifying image features of each image region through a recognition network corresponding to each local image; and adding the image features to the 3D cartoon face model to obtain the 3D cartoon image.
[0169] Facial key points can refer to key points corresponding to key parts of the face, such as facial features, specifically including key points corresponding to at least one of the following: eyebrows, eyes, nose, and lips. Local features can refer to the features corresponding to specific parts of the face, such as gender, hair, eyebrows, eyes, glasses, nose, and lips. Corresponding feature characteristics can include gender characteristics, hair characteristics, eyebrow characteristics, eye characteristics, glasses characteristics, nose characteristics, and lip characteristics.
[0170] Specifically, the terminal also performs keypoint detection on the facial image to extract facial keypoints. Then, based on the keypoints, the terminal extracts the image regions corresponding to each local feature in the facial image. For example, image regions can be segmented based on facial keypoints to correspond to each local feature. These image regions are the areas in the facial image that correspond to each local feature.
[0171] The terminal then identifies the image features of each image region through a recognition network corresponding to each local image. The recognition network corresponding to each local image is a pre-trained neural network model capable of extracting and recognizing features of each local image. This recognition network can be a machine learning model trained using networks such as CNN (Convolutional Neural Networks), ResNet (Deep Residual Convolutional Network), DenseNet (Dense Convolutional Network), and DPN (Dual Path Network).
[0172] For example, recognition networks corresponding to various partial images, including gender recognition networks, hairstyle recognition networks, eye recognition networks, glasses recognition networks, eyebrow recognition networks, etc.
[0173] The terminal then inputs the corresponding image regions for each local image into the corresponding recognition network for identification and classification to obtain the corresponding image features. Based on the identified image features, it then obtains the matching cartoon image components and adds them to the 3D cartoon face model to obtain the 3D cartoon image.
[0174] For example, specific image features can include gender, hair type, double eyelid type, eyebrow type, and glasses type. Gender includes male and female. Hair type can include male hairstyles such as buzz cut, crew cut, slicked-back hair, parted hair, punk hairstyle, quirky hairstyle, pompadour, and side-parted slicked-back hair, and female hairstyles such as long hair, short hair, medium-short hair, curly hair, big waves, and bun. Double eyelid type can include no double eyelids and double eyelids, as well as an intensity value, such as 0~1, which can be output as a parameter of 0~1 by the recognition network. Glasses type can include round black-rimmed glasses, square black-rimmed glasses, light-rimmed round glasses, light-rimmed square glasses, round sunglasses, and square sunglasses, etc.
[0175] Then, different recognition networks are used for recognition and classification to obtain the image features corresponding to the facial image. Specifically, the terminal performs keypoint detection on the input facial image, for example, obtaining 256 facial keypoints. Then, based on the facial keypoints, the image corresponding to each local image is cropped. The cropped image is then scaled to a preset size, for example, 224*224. After normalization and mean subtraction operations on the scaled image, it is input into the recognition network corresponding to each local image. Features are extracted from the image of the specific image region to obtain the feature vector corresponding to each part. Then, the obtained feature vectors are concatenated through a specific network to obtain the corresponding vector. Finally, the classification layer of the convolutional network classifies the category of the target part to obtain the image features corresponding to each local image. For example, it can include category attributes such as gender features, hair features, double eyelid features, eyebrow features, and glasses features.
[0176] In one embodiment, the 3D facial features obtained by the terminal corresponding to the facial image also include texture coefficients, which are parameters of the facial skin. The terminal then adds texture color information corresponding to the texture coefficients to the generated 3D cartoon facial model, making the generated 3D cartoon image more closely match the facial texture and color in the facial image. Specifically, the texture image corresponding to the 3D face can be obtained using the following formula:
[0177] ;
[0178] Where Tg represents the texture image to be generated. T represents average texture. i Here, δ represents the texture feature vector, and δ is the texture coefficient in the 3D face feature. After extracting the mean and variance from the texture image, a matching texture color can be found in a preset color swatch for cartoon characters. This color swatch for cartoon characters corresponds to various texture color information. For example... Figure 12The image shown is a schematic diagram of a color chart for a cartoon character in one embodiment. The different color characteristics can be distinguished by their shades and intensities. Figure 12 The document displays ten color charts with different color characteristics, each corresponding to a different texture color. This texture color information is then mapped onto the generated 3D cartoon face model, resulting in a 3D cartoon image that matches the texture and color of the face in the image.
[0179] In one embodiment, a mapping table corresponds to the color swatches of various cartoon characters, storing the mapping relationships between the color swatches of different cartoon characters and their texture color information. After the terminal obtains the corresponding texture image based on the texture coefficients in the 3D facial features, it compares the obtained texture image with the preset color swatches of various cartoon characters to find the closest color. Specifically, it compares pixel color values and determines the color swatch of the cartoon character with the closest color value as the matching color swatch for the texture image. Then, the terminal retrieves the texture color information of the cartoon character corresponding to the matching color swatch from the mapping table. Finally, it maps and renders the retrieved texture color information of the cartoon character onto the 3D cartoon face model. This allows the mapping table to map the color of the current user's facial skin onto the 3D cartoon face model, thereby generating a more accurate 3D cartoon character similar to the current user.
[0180] In one embodiment, the terminal can also add eyelash features to the generated 3D cartoon face model. Specifically, the terminal can obtain a preset average eyelash model, and then, based on the eye features of the generated 3D cartoon face model, deform the average eyelash model to obtain an eyelash model that matches the eye features, and add it to the 3D cartoon face model. For example, a mesh deformation algorithm can be used to deform the average eyelash model.
[0181] Specifically, a corresponding number of index keypoints can be pre-marked at the eyelash positions of the 3D cartoon face template and at the corresponding positions of the average eyelash model; for example, 16 keypoint coordinate indices can be marked. Then, based on the coordinates of the eyelash vertices (i.e., the index keypoints) in the currently generated 3D cartoon face model, the average eyelash model is deformed. The deformation goal is to transform the original coordinate vertices of the index keypoints in the average eyelash model to the corresponding eyelash vertices in the 3D cartoon face model. For example, a capability function can be used to deform the eyelash vertex data.
[0182] ;
[0183] Among them, Ci and These represent the vertices p of the model before deformation. i and the vertices of the deformed model The corresponding deformation element, N(i), represents the model vertex p. i The index of the 1-neighborhood point, and р j and These represent the model vertices p respectively. i and model vertices The vertices of R's 1-neighborhood i Indicates from C i arrive The optimal rotation matrix, w ij Representing edge e ij =(p i ,p j The weight of ).
[0184] For example, such as Figure 13 The diagram shown illustrates the eyelash deformation processing in one embodiment, wherein the average eyelash model 13(A) includes eyelash vertices p. i and the tip of the eyelashes j Through rotation matrix R i After deforming the average eyelash model 13(A), the deformed eyelash model 13(B) and the deformed eyelash apex are obtained. and the tip of the eyelashes This allows for the accurate deformation of the eyelash vertex coordinates of the average eyelash model to match the position of the 3D cartoon face model, thus enabling the precise generation of eyelashes that match the 3D cartoon face model.
[0185] In one embodiment, displaying a basic 3D cartoon image obtained by removing facial expressions from a 3D cartoon image includes: adjusting the facial expression coefficients in the 3D cartoon image so that the facial expression coefficients of the 3D cartoon image match the neutral facial expression coefficients, thereby obtaining a basic 3D cartoon image after facial expression removal; and displaying the basic 3D cartoon image obtained after facial expression removal.
[0186] Among them, the expression coefficient refers to the coefficient corresponding to the facial expression features of a 3D cartoon character. The neutral expression coefficient refers to the coefficient corresponding to the facial expression features of a 3D cartoon character template with a neutral expression. The neutral expression coefficient can be obtained from prior data trained using a large number of sample 3D cartoon face data.
[0187] It's understandable that removing facial expressions from 3D cartoon characters is essentially the process of adjusting the facial expression coefficients in the 3D cartoon character to match the neutral facial expression coefficients.
[0188] After the terminal generates a 3D cartoon image corresponding to the face image, it performs expression removal processing on the 3D cartoon image. Specifically, the terminal adjusts the expression coefficients in the 3D cartoon image according to a preset neutral expression coefficient, so that the expression coefficients of the 3D cartoon image match the neutral expression coefficients, thus obtaining a basic 3D cartoon image after expression removal, and displays the basic 3D cartoon image after expression removal.
[0189] In one embodiment, the terminal can also adjust the expression coefficients in the 3D cartoon character according to a preset neutral expression coefficient to obtain the adjusted expression coefficients, and then perform deformation processing on the 3D cartoon character according to the adjusted expression coefficients to generate a basic 3D cartoon character with expression removal.
[0190] It's understandable that, since the generated 3D cartoon characters must be neutral and expressionless, an expression removal model can be trained to remove expressions from the 3D cartoon characters. Specifically, principal component analysis (PCA) can be used to reconstruct the expression coefficients of the 3D cartoon characters, thus performing expression removal.
[0191] This process involves pre-generating 3D cartoon base data and 3D cartoon face shape base data. First, a large dataset of 3D cartoon faces is constructed; for example, facial features corresponding to a large number of faces can be extracted from a pre-set database, specifically one million faces. Then, the facial features of these one million faces are clustered to obtain 5000 face categories. These 5000 faces can be considered to encompass most face shapes. Next, the most representative sample face images are selected from each face category. Then, for these 5000 sample face images, 5000 corresponding 3D face mesh models are generated. Then, by deforming and transferring these 5000 3D face mesh models, 5000 corresponding 3D cartoon face models are generated. Since some of the generated 3D cartoon face models may have open mouths, a mesh deformation algorithm is used to close the mouths of all samples with open mouths, thus constructing a complete 3D cartoon face dataset.
[0192] Then, PCA analysis is performed on the vertex data corresponding to all 3D cartoon face models in the 3D cartoon face dataset to generate corresponding 3D cartoon face templates, i.e., average cartoon face models. And the feature vector matrix C corresponding to the average cartoon face model id The generated 3D cartoon face template has a neutral expression coefficient.
[0193] After obtaining the 3D cartoon face template and its corresponding feature vector matrix, it is only necessary to reverse-engineer the expression coefficients corresponding to the 3D cartoon image to deduce the cartoon face image with a neutral expression. The specific expression can be as follows:
[0194] ;
[0195] Among them, C new This indicates the 3D cartoon character that needs to be generated with facial expressions removed; Represents the average face in a 3D cartoon; C id This represents the feature vector matrix corresponding to the 3D cartoon face template.
[0196] The neutral expression coefficient corresponding to a 3D cartoon character can be obtained by minimizing the following formula:
[0197] ;
[0198] Where S represents the generated 3D cartoon character; A 3D cartoon face template; C id α represents the feature vector matrix corresponding to the 3D cartoon face template; α represents the expression coefficient of the 3D cartoon character to be solved; 92 represents the vector dimension retained after principal component analysis of all cartoon model data.
[0199] In this embodiment, since the training data for cartoon faces are all expressionless 3D cartoon face data, the expression coefficients of the 3D cartoon characters can be inferred by using the neutral expression coefficients corresponding to a large amount of 3D cartoon face data. This allows the expression coefficients of the 3D cartoon characters to match the neutral expression coefficients, thus obtaining 3D cartoon characters with neutral expressions.
[0200] In one embodiment, the above-mentioned 3D cartoon image processing method further includes: generating multiple cartoon facial expression bases corresponding to the basic 3D cartoon image. When the target user is within the camera's field of view and the target user has an expression, displaying a 3D cartoon image obtained based on changes to the basic 3D cartoon image includes: acquiring the target user's face image when the target user is within the camera's field of view and the target user has an expression; performing 3D reconstruction processing on the face image to obtain corresponding 3D facial expression features; processing the 3D facial expression features according to the cartoon facial expression bases; mapping the processed 3D facial expression features onto the basic 3D cartoon image to obtain and display a 3D cartoon image that matches the target user's expression.
[0201] Among them, 3D facial expression features can refer to facial expression-related features in a 3D face. Cartoon facial expression base refers to a cartoon face base model with corresponding expressions obtained by transforming the expressions of a basic 3D cartoon character according to a preset strategy; one cartoon facial expression base is one cartoon face base model. However, a 3D cartoon character is obtained by transforming the expressions of a basic 3D cartoon character based on the target user's facial expressions. Therefore, cartoon facial expression base and 3D cartoon character are different terms.
[0202] After acquiring the base 3D cartoon image with removed facial expressions, the terminal also generates multiple cartoon face expression bases corresponding to the base 3D cartoon image. These multiple cartoon face expression bases can be multiple cartoon face expression base models generated by deforming the base 3D cartoon image using different methods.
[0203] Specifically, to enable facial expression changes to a basic 3D cartoon character, such as enlarging the eyes, thickening the mouth, or thinning the face, multiple cartoon face expression bases can be generated for a preset 3D cartoon face template. For example, 50 cartoon face expression base models corresponding to the 3D cartoon face template can be generated. The 3D cartoon face template can be obtained by linearly combining and weighting these 50 cartoon face expression base models. Specifically, these 50 cartoon face expression base models can be solved using a coefficient back-calculation algorithm for 3D vertices. For example, the 50 expression coefficients corresponding to the 50 cartoon face expression bases can be calculated using a least squares algorithm, specifically using the following expression:
[0204] ;
[0205] Where B represents the base data of 50 cartoon facial expressions; S shape Represents the vertex data of a 3D cartoon face template; α id This represents the expression coefficients corresponding to 50 cartoon face expression bases, where α id (id = 1,2..50). This allows for arbitrary manipulation of these 50 cartoon face emoji bases to alter the shape of the user's cartoon face based on the fundamental 3D cartoon image.
[0206] like Figure 14 The diagram illustrates multiple cartoon face expression bases corresponding to a 3D cartoon face template in one embodiment. These 50 cartoon face expression bases can be pre-built models, such as base models that represent relatively clean decoupling, such as those representing only mouth movement, eye movement, and nose movement. By fitting 50 expression coefficients corresponding to these 50 cartoon face expression bases, the original cartoon image can be reconstructed.
[0207] It is understandable that by first removing facial expressions to obtain a basic 3D cartoon image without any expression, and then using the data from these 50 facial expression models to infer 50 inverse coefficients, users can manipulate the facial expression coefficients corresponding to the 50 cartoon face expression bases to deform the expression of the basic 3D cartoon image, thereby driving the expression changes of the basic 3D cartoon image.
[0208] After obtaining the base 3D cartoon image with removed facial expressions, the terminal generates multiple cartoon face expression bases corresponding to the base 3D cartoon image. Specifically, the terminal can generate 51 corresponding cartoon face expression bases for the base 3D cartoon image. Then, it calculates the difference between each cartoon face expression base and the 3D cartoon face template. For example, it can subtract the vertex coordinates of the 3D cartoon face template from each cartoon face expression base to obtain the difference vector between each cartoon face expression base and the 3D cartoon face template.
[0209] Specifically, when the target user is within the camera's field of view and the target user is expressing an expression, the terminal acquires the target user's facial image. The terminal then performs 3D reconstruction processing on the target user's facial image to extract the corresponding 3D facial features. Specifically, the 3D facial expression features may include facial scaling parameters, facial shape rotation matrix, expression basis coefficients, and offset vectors.
[0210] Then, the terminal processes the 3D facial expression features based on the cartoon facial expression base, maps the processed 3D facial expression features onto the basic 3D cartoon image, and thus obtains and displays a 3D cartoon image that matches the target user's expression. Specifically, the 3D facial expression features can be mapped onto the basic 3D cartoon image using the following formula:
[0211] ;
[0212] Where S represents the 3D cartoon character driven by a cartoon facial expression; S0 represents the basic 3D cartoon character; S i S represents the difference vector between each cartoon face expression base and the 3D cartoon face template, where S i (i0,i1,i2….i 50 ); a represents the face scaling parameter; P represents the face shape rotation matrix; w represents the expression basis coefficients; and t represents the offset vector. This allows the generation of a 3D cartoon character that matches the expression in the target user's facial image.
[0213] In this embodiment, multiple cartoon face expression bases corresponding to the basic 3D cartoon image are generated. Then, based on the difference vectors between the multiple cartoon face expression bases and the 3D cartoon face template, the 3D face expression features corresponding to the target user's face image are processed. The processed 3D face expression features are mapped onto the basic 3D cartoon image, thereby transforming the expression image in the basic 3D cartoon image into the target user's expression, thus enabling the accurate generation of a 3D cartoon image that matches the target user's expression.
[0214] In a specific embodiment, such as Figure 15 The diagram illustrates a process for generating a 3D cartoon avatar that matches the expression of a target user, as shown in one embodiment. The terminal first acquires an initial face image, then performs 3D face reconstruction on the initial face image to obtain 3D face features including face shape coefficients, face expression coefficients, and texture coefficients. Next, it acquires a 3D simulated face template pre-constructed based on 3D face principal component vectors and a corresponding 3D cartoon face template, and obtains the face feature difference matrix corresponding to the 3D cartoon face template after deformation and transfer processing; this matrix represents the prior knowledge of the cartoon face. Then, the 3D face features are processed according to the face feature difference matrix, and the processed 3D face features are mapped onto the 3D cartoon face template to obtain a 3D cartoon face model.
[0215] The terminal also performs face detection, gender detection, and hairstyle segmentation on the initial face image, and identifies the image features in the face image. Then, it adds corresponding image features to the 3D cartoon face model and optimizes the facial features, such as adding eyelash models, to generate a 3D cartoon image corresponding to the face image. Next, the terminal reconstructs the expression coefficients of the 3D cartoon image by reverse calculation, adjusting the expression coefficients to match the neutral expression coefficients. This process removes the expression from the 3D cartoon image. Furthermore, the terminal can add corresponding texture color features to the 3D cartoon image based on the texture coefficients corresponding to the face image for texture reconstruction, thus obtaining a basic 3D cartoon image with expression removal.
[0216] The terminal can then collect real-time facial images of the target user. Specifically, when the target user is within the camera's field of view and has an expression, the terminal collects the target user's real-time facial image, then performs 3D facial reconstruction on the real-time facial image to extract the 3D facial expression coefficients corresponding to the real-time facial image, and then maps the 3D facial expression coefficients onto a basic 3D cartoon character. This transforms the expression in the basic 3D cartoon character to match the target user's expression, thereby generating a 3D cartoon character that matches the target user's expression.
[0217] This application also provides an application scenario in which the above-described 3D cartoon character processing method is applied, specifically to social applications, instant messaging applications, and game applications. Specifically, the application of the 3D cartoon character processing method in this scenario is as follows:
[0218] The terminal can deploy social networking applications, instant messaging applications, and games, all of which include avatar customization features. Users can use these features to create a 3D cartoon avatar corresponding to their facial image within these applications, thus customizing a virtual avatar tailored to them.
[0219] In one embodiment, users can also customize their own virtual avatars through a corresponding terminal, that is, generate a virtual avatar that corresponds to the target user. Specifically, users can also obtain a facial image from their local storage, such as a low-cost photo including a face, to generate a virtual avatar similar to the user. The terminal can further customize the virtual avatar to be synchronized with the user's real-time expressions, emotions, facial movements, behaviors, or current appearance based on the user's real-time expressions, emotions, facial movements, behaviors, or current appearance.
[0220] Specifically, the terminal can acquire an initial facial image and then generate a corresponding basic 3D cartoon image with removed facial expressions. The target user can further acquire their current facial image in real time via the corresponding terminal. When the target user is within the camera's field of view and has facial expressions, the terminal acquires the target user's current facial image and, based on the expressions in the current facial image, transforms the basic 3D cartoon image's expression to generate a 3D cartoon image that matches the target user's expression, which is then displayed.
[0221] Furthermore, when the target user's facial expression changes within the camera's field of view, a 3D cartoon image, based on changes in the base 3D cartoon image and synchronized with the target user's expression, is displayed. This allows for effective real-time driving of the base 3D cartoon image after expression removal based on the target user's real-time facial expressions, thereby accurately generating a 3D cartoon image with a high degree of matching to real facial expressions. This enables the precise generation of customized virtual avatars that closely resemble the user's real face.
[0222] It should be understood that, although Figure 2The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps; they can be executed in other orders. Furthermore, Figure 2 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.
[0223] In one embodiment, such as Figure 16 As shown, a 3D cartoon character processing device 1600 is provided. This device can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes: a 3D cartoon character acquisition module 1602 and a 3D cartoon character display module 1604, wherein:
[0224] The 3D cartoon character acquisition module 1602 is used to acquire a basic 3D cartoon character after facial expression removal.
[0225] The 3D cartoon character display module 1604 is used to display a 3D cartoon character of the target user, which is a variation of the basic 3D cartoon character, when the target user is within the camera's field of view and the target user has an expression; the expression of the 3D cartoon character matches the expression of the target user.
[0226] The 3D cartoon image acquisition module is also used to display a 3D cartoon image that is synchronized with the target user's expression when the target user's facial expression changes within the camera's field of view.
[0227] In one embodiment, the aforementioned 3D cartoon character processing device further includes a face scanning module for displaying a face scanning interface; the 3D cartoon character display module is also used to capture the current face image of the target user when the target user is within a first preset acquisition area in the face scanning interface; the first preset acquisition area is an area in the face scanning interface that is within the camera's field of view; if the target user in the current face image has an expression, then the target user's current face image and 3D cartoon character are displayed in the cartoon character display interface; the 3D cartoon character is obtained by controlling the changes of the basic 3D cartoon character based on the target user's expression.
[0228] In one embodiment, the cartoon character display interface includes a second preset acquisition area and a character display area; the current face image is displayed in the second preset acquisition area; the three-dimensional cartoon character is displayed in the character display area; the second preset acquisition area is the area in the cartoon character display interface that is located within the camera's field of view.
[0229] In one embodiment, the 3D cartoon character display module is also used to display a 3D cartoon character in the character display area that is synchronized with the target user's expression, based on the changes in the basic 3D cartoon character, when the target user's expression changes in the second preset acquisition area.
[0230] In one embodiment, the 3D cartoon image acquisition module is further used to acquire the input face image and generate a 3D cartoon image corresponding to the face image; the 3D cartoon image has the face image in the face image; the 3D cartoon image display module is further used to display the basic 3D cartoon image obtained by removing the expression from the 3D cartoon image.
[0231] In one embodiment, the above-mentioned three-dimensional cartoon character processing device further includes a three-dimensional cartoon character generation module, which is used to generate three-dimensional facial features corresponding to a facial image through a trained facial feature prediction network; map the three-dimensional facial features to a three-dimensional cartoon face template to obtain a three-dimensional cartoon face model; add image features from the facial image to the three-dimensional cartoon face model to generate a three-dimensional cartoon character corresponding to the facial image.
[0232] In one embodiment, the face feature prediction network is obtained through a model training step. The aforementioned 3D cartoon image processing device further includes a model training module for acquiring sample data. The sample data includes sample images and corresponding facial key points. Using the face feature prediction network to be trained, based on the preset 3D face data and the 2D face features of the sample images, the 3D face features corresponding to the sample images are predicted to reconstruct the sample 3D face. The sample 3D face is mapped and rendered onto the sample images to obtain a 2D mapped image. A first loss is determined based on the difference between the key points of the sample 3D face and the facial key points. A second loss is determined based on the pixel difference between the 2D mapped image and the sample image. Based on the first loss and the second loss, the parameters of the face feature prediction network are adjusted and training continues until the training stop condition is met, resulting in a trained face feature prediction network.
[0233] In one embodiment, the sample data also includes the face segmentation result corresponding to the sample image; the model training module is further used to determine a third loss based on the difference between the two-dimensional mapping image and the face segmentation result; and to adjust the parameters of the face feature prediction network and continue training based on the first loss, the second loss and the third loss.
[0234] In one embodiment, the 3D cartoon character generation module is further used to obtain a facial feature difference matrix corresponding to the 3D cartoon face template; process the 3D facial features according to the facial feature difference matrix; and map the processed 3D facial features to the 3D cartoon face template to obtain a 3D cartoon face model.
[0235] In one embodiment, the 3D cartoon character generation module is further used to extract facial key points from a face image; based on the facial key points, extract the image regions corresponding to each local image in the face image; identify the image features of each image region through a recognition network corresponding to each local image; and add the image features to the 3D cartoon face model to obtain a 3D cartoon character.
[0236] In one embodiment, the 3D cartoon character generation module is further used to adjust the expression coefficients in the 3D cartoon character so that the expression coefficients of the 3D cartoon character match the neutral expression coefficients, thereby obtaining a basic 3D cartoon character after expression removal; and displaying the basic 3D cartoon character obtained after expression removal.
[0237] In one embodiment, the 3D cartoon character generation module is further used to generate multiple cartoon facial expression bases corresponding to the basic 3D cartoon character; when the target user is within the camera's field of view and the target user has an expression, the module acquires the target user's facial image; performs 3D reconstruction processing on the facial image to obtain the corresponding 3D facial expression features; processes the 3D facial expression features according to the cartoon facial expression bases; and maps the processed 3D facial expression features onto the basic 3D cartoon character to obtain a 3D cartoon character that matches the target user's expression and displays it.
[0238] Specific limitations regarding the 3D cartoon character processing device can be found in the limitations of the 3D cartoon character processing method described above, and will not be repeated here. Each module in the aforementioned 3D cartoon character processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0239] In one embodiment, a computer device is provided, which may specifically be a terminal, and its internal structure diagram may be as follows: Figure 17As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a three-dimensional cartoon character processing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.
[0240] Those skilled in the art will understand that Figure 17 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0241] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0242] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0243] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.
[0244] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0245] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0246] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for processing three-dimensional cartoon characters, characterized in that, The method includes: Obtaining a basic 3D cartoon image after expression removal includes: generating 3D facial features corresponding to a face image using a trained facial feature prediction network; mapping the 3D facial features to a 3D cartoon face template to obtain a 3D cartoon face model; adding image features from the face image to the 3D cartoon face model to generate a 3D cartoon image corresponding to the face image; the 3D cartoon image possesses the face image from the face image; adjusting the expression coefficients in the 3D cartoon image by performing expression coefficient back-construction to make the expression coefficients of the 3D cartoon image match the neutral expression coefficients, thus obtaining a basic 3D cartoon image after expression removal; and displaying the basic 3D cartoon image obtained after expression removal. Expression removal refers to the process of processing the expressions in the 3D cartoon image into neutral expressions. The 3D cartoon image after expression removal is visually expressionless. Generate multiple cartoon face expression bases corresponding to the basic 3D cartoon image; calculate the difference between each cartoon face expression base and the 3D cartoon face template to obtain the difference vector between each cartoon face expression base and the 3D cartoon face template; When the target user is within the camera's field of view and the target user has an expression, a 3D cartoon image of the target user, derived from the base 3D cartoon image, is displayed. This includes: extracting corresponding 3D facial expression features from the acquired face image; processing the 3D facial expression features based on the difference vectors between the multiple cartoon facial expression bases and the 3D cartoon face template; mapping the processed 3D facial expression features onto the base 3D cartoon image; and then deforming the base 3D cartoon image to obtain the transformed 3D cartoon image. The expression of the 3D cartoon image matches the expression of the target user, and the target user may be the same as or different from the user in the face image corresponding to the base 3D cartoon image. When the target user's facial expression changes within the camera's field of view, a 3D cartoon image that changes based on the base 3D cartoon image and is synchronized with the target user's expression is displayed, following the change in the target user's facial expression. This includes: extracting the 3D facial expression features corresponding to the real-time continuously acquired facial images after the expression change; processing the 3D facial expression features based on the difference vectors between the multiple cartoon facial expression bases and the 3D cartoon facial template; and then mapping the processed 3D facial expression features onto the base 3D cartoon image to deform the base 3D cartoon image, thereby obtaining the transformed 3D cartoon image.
2. The method according to claim 1, characterized in that, The method further includes: Display the face scanning interface; The step of displaying a 3D cartoon image of the target user, derived from the base 3D cartoon image, when the target user is within the camera's field of view and has an expression includes: When a target user is within the first preset acquisition area of the face scanning interface, the current face image of the target user is acquired; the first preset acquisition area is the area in the face scanning interface that is within the camera's field of view; If the target user in the current face image has an expression, then the target user's current face image and a 3D cartoon image are displayed on the cartoon image display interface; the 3D cartoon image is obtained by controlling the changes of the base 3D cartoon image based on the target user's expression.
3. The method according to claim 2, characterized in that, The cartoon character display interface includes a second preset acquisition area and a character display area; the current face image is displayed in the second preset acquisition area. The 3D cartoon character is displayed in the character display area; The second preset acquisition area is the area in the cartoon character display interface that is within the camera's field of view.
4. The method according to claim 3, characterized in that, When the target user's facial expression changes within the camera's field of view, the system displays a 3D cartoon image that is synchronized with the target user's facial expression, based on changes in the base 3D cartoon image. This includes: When the target user's facial expression changes within the second preset acquisition area, a 3D cartoon image, synchronized with the target user's facial expression, is displayed in the image display area, based on the changes in the basic 3D cartoon image.
5. The method according to any one of claims 1 to 4, characterized in that, The physical features include at least one of the following: hair, birthmark, mole, eyebrows, eyes, and glasses.
6. The method according to claim 1, characterized in that, The three-dimensional facial features include at least one of facial shape parameters, facial expression coefficients, facial texture coefficients, pose parameters, texture parameters, and lighting parameters.
7. The method according to claim 1, characterized in that, The facial feature prediction network is obtained through a model training step, which includes: Acquire sample data; the sample data includes sample images and facial key points corresponding to the sample images; The face feature prediction network to be trained predicts the three-dimensional face features corresponding to the sample image based on the preset three-dimensional face data and the two-dimensional face features of the sample image in order to reconstruct the three-dimensional face of the sample. The three-dimensional face of the sample is mapped and rendered onto the sample image to obtain a two-dimensional mapped image; The first loss is determined based on the difference between the key points of the sample 3D face and the key points of the face. The second loss is determined based on the pixel difference between the two-dimensional mapped image and the sample image; Based on the first loss and the second loss, the parameters of the face feature prediction network are adjusted and training continues until the training stop condition is met, resulting in a well-trained face feature prediction network.
8. The method according to claim 7, characterized in that, The sample data also includes the face segmentation results corresponding to the sample images; The method further includes: The third loss is determined based on the difference between the two-dimensional mapped image and the face segmentation result; The step of adjusting the parameters of the face feature prediction network based on the first loss and the second loss and continuing training includes: Based on the first loss, the second loss, and the third loss, the parameters of the face feature prediction network are adjusted and training continues.
9. The method according to claim 1, characterized in that, The step of mapping the three-dimensional facial features onto a three-dimensional cartoon facial template to obtain a three-dimensional cartoon facial model includes: Obtain the facial feature difference matrix corresponding to the 3D cartoon face template; The three-dimensional facial features are processed based on the facial feature difference matrix; The processed 3D facial features are mapped onto the 3D cartoon face template to obtain a 3D cartoon face model.
10. The method according to claim 1, characterized in that, The step of adding facial features from the face image to the 3D cartoon face model to generate a 3D cartoon image corresponding to the face image includes: Extract facial key points from the face image; Based on the facial key points, extract the image regions corresponding to each local image in the facial image; The image features of each image region are identified by a recognition network corresponding to each local image. The aforementioned features are added to the 3D cartoon face model to obtain a 3D cartoon image.
11. The method according to claim 1, characterized in that, The method further includes: After constructing the required three-dimensional average face model data, the face shape feature vector, face expression basis vector, and face texture vector are constructed, the face feature prediction network is constructed based on the three-dimensional average face model data, and the face feature prediction network is trained.
12. The method according to claim 9, characterized in that, The facial feature difference matrix includes index key points.
13. A three-dimensional cartoon character processing device, characterized in that, The device includes: A 3D cartoon character acquisition module is used to acquire a basic 3D cartoon character after expression removal. The module includes: generating 3D facial features corresponding to a face image using a trained facial feature prediction network; mapping the 3D facial features to a 3D cartoon face template to obtain a 3D cartoon face model; adding image features from the face image to the 3D cartoon face model to generate a 3D cartoon character corresponding to the face image; the 3D cartoon character possesses the face image from the face image; adjusting the expression coefficients in the 3D cartoon character by performing expression coefficient back-reaming to make the expression coefficients of the 3D cartoon character match the neutral expression coefficients, thus obtaining a basic 3D cartoon character after expression removal; displaying the basic 3D cartoon character obtained after expression removal (expression removal refers to the process of processing the expressions in the 3D cartoon character into neutral expressions; the 3D cartoon character after expression removal is visually expressionless); and generating a character corresponding to the basic 3D cartoon character. The system uses multiple cartoon face expression bases; calculates the difference between each cartoon face expression base and the 3D cartoon face template to obtain the difference vector between each cartoon face expression base and the 3D cartoon face template; and a 3D cartoon image display module, used to display a 3D cartoon image of the target user based on the changes of the base 3D cartoon image when the target user is within the camera's field of view and the target user has an expression, including: extracting corresponding 3D facial expression features from the acquired face image; processing the 3D facial expression features according to the difference vectors between the multiple cartoon face expression bases and the 3D cartoon face template; mapping the processed 3D facial expression features onto the base 3D cartoon image to deform the base 3D cartoon image to obtain the changed 3D cartoon image; the expression image of the 3D cartoon image matches the expression of the target user, and the target user is the same as or different from the user in the face image corresponding to the base 3D cartoon image; The 3D cartoon image acquisition module is also used to display a 3D cartoon image that is synchronized with the target user's expression, based on the changes in the base 3D cartoon image, when the target user's expression changes within the camera's field of view. This includes: extracting the 3D facial expression features corresponding to the real-time continuously acquired facial images after the expression change; processing the 3D facial expression features according to the difference vectors between the multiple cartoon facial expression bases and the 3D cartoon facial template; and then mapping the processed 3D facial expression features onto the base 3D cartoon image to deform the base 3D cartoon image, thereby obtaining the transformed 3D cartoon image.
14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.
15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.
16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.
