A method, electronic device and storage medium for real-time facial expression driving of cartoon digital humans

Abstract

This application provides a method, electronic device, and storage medium for real-time facial expression driving of a cartoon digital human, relating to the field of terminal technology. The method includes: the terminal device first acquires target images containing facial information of a target user through a camera at a preset frequency; then, each target image is input frame by frame into a FaceMesh network for 3D reconstruction of the target user's face to obtain 3D key point information of the target user's face. Next, the 3D key point information of the target user's face is input into a cartoon character facial expression driving network model to predict the cartoon character's facial expression basis coefficients and rotation feature representations; then, using these cartoon character facial expression basis coefficients and rotation feature representations, combined with a preset BlendShape parameter model, and based on computer graphics knowledge, rendering is performed to obtain cartoon character facial expression animation images corresponding to the target images, and these cartoon character facial expression animation images are continuously output to form a real-time facial expression driven animation of the target user's cartoon digital human.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a method for driving real-time facial expressions of a cartoon digital human, an electronic device, and a storage medium. Background Technology

[0002] With the continuous development of technology, mobile terminal devices, represented by smartphones and tablets, are increasingly used in people's lives and work, bringing great convenience. For example, people can use these terminal devices to take photos, record videos, and conduct social communication.

[0003] Currently, when users engage in social communication using mobile devices, they often customize cartoon avatars to enhance the fun of social interaction. However, existing methods for driving cartoon avatar expressions often fail to meet the real-time requirements of mobile devices and have low accuracy, making it difficult to achieve the desired driving effect. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method, electronic device, and storage medium for driving real-time facial expressions in a cartoon digital human. The aim is to enable high real-time performance and high precision in driving facial expressions in a cartoon digital human when users engage in social communication using mobile terminal devices, thereby providing users with a more novel and interesting social experience.

[0005] Firstly, this application provides a method for driving real-time facial expressions in a cartoon digital human. The method includes: upon receiving a cartoon digital human generation command triggered by a user's camera on the terminal device's shooting interface, the terminal device first responds to the command by acquiring a target image containing the target user's facial information via a camera at a preset frequency (e.g., 30fps). Then, each target image is input frame-by-frame into a FaceMesh network for 3D reconstruction of the target user's face, estimating 468 3D facial landmarks in real-time as the target user's 3D facial keypoint information. Next, the target user's 3D facial keypoint information is input into a cartoon character facial expression driving network model to predict the cartoon character's facial basis coefficients and rotation feature representations. Then, using these cartoon character facial basis coefficients and rotation feature representations, combined with a preset BlendShape parameter model (based on a predefined set of 52 facial shapes), rendering is performed according to computer graphics knowledge to obtain a cartoon character facial expression animation image corresponding to the target image. Finally, all obtained cartoon character facial expression animation images are concatenated in chronological order, and the concatenated cartoon character facial expression animation images are continuously output to form a real-time facial expression driven animation of the cartoon digital human corresponding to the target user.

[0006] As can be seen, in the above-mentioned real-time expression driving method for cartoon digital humans, this embodiment pre-constructs a cartoon character expression driving network model that is friendly to mobile terminal devices based on fully connected neural networks and convolutional structures. This not only improves the driving accuracy and speed, but also facilitates the quantitative deployment of various deployment tools on mobile terminal devices, thereby reducing the performance requirements of the mobile terminal device processor without reducing the network computing performance. This enables users to achieve high real-time and high-precision expression driving of cartoon digital humans when using mobile terminal devices such as mobile phones for social communication, thus providing users with a more novel and interesting social experience.

[0007] In one possible implementation, the target image is input into a FaceMesh network for 3D reconstruction of the target user's face image to obtain the target user's 3D facial key point information. This includes: cropping the target user's face region from the target image to obtain a target user face region image; performing rotation and size transformation alignment processing on the target user face region image to obtain an aligned target user face region image; calculating the rotation inverse transformation matrix representation corresponding to the target user face region image; inputting the aligned target user face region image into the FaceMesh network for 3D reconstruction of the target user's face image to obtain 468 facial key point information of the target user; and using the rotation inverse transformation matrix representation corresponding to the target user face region image and the 468 facial key point information of the target user, calculating the target user's 3D facial key point information. In this way, the geometry of the 3D facial surface can be inferred using machine learning with only the input of a single camera, without the need for a dedicated depth sensor, thereby providing real-time performance and meeting the high real-time requirements of mobile terminal devices.

[0008] In one possible implementation, the cartoon character expression-driven network model includes a key information encoder, a cartoon character expression basis coefficient prediction module, and a rotation representation prediction module. The 3D key point information of the target user's face is input into the pre-built cartoon character expression-driven network model to predict the cartoon character expression basis coefficients and rotation feature representations. This includes: inputting the target user's 3D key point information into the key information encoder of the pre-built cartoon character expression-driven network model for feature extraction to obtain a latent space feature representation; inputting the latent space feature representation into the cartoon character expression basis coefficient prediction module of the cartoon character expression-driven network model for mean dimensionality reduction processing to obtain the cartoon character expression basis coefficients; and inputting the latent space feature representation into the rotation representation prediction module of the cartoon character expression-driven network model for size transformation and convolution processing to obtain the rotation feature representation. This improves driving accuracy and speed, facilitates the quantitative deployment of various deployment tools on mobile terminal devices, reduces the performance requirements of mobile terminal device processors, and does not reduce network computing performance.

[0009] In one possible implementation, the key information encoder is an encoder built on a fully connected network structure; the cartoon character expression basis coefficient prediction module is a decoder built on a fully connected network structure; and the rotation representation prediction module is a decoder built on a convolutional structure.

[0010] In one possible implementation, a cartoon character expression animation image corresponding to the target image is rendered by using the cartoon character expression basis coefficients and rotation feature representation, combined with a preset blendShape parameter model. This includes: using the cartoon character expression basis coefficients to perform a weighted summation on the preset blendShape parameter model to predict the cartoon character expression model corresponding to the target image; using the rotation feature representation to perform a rotation transformation on the predicted cartoon character expression model corresponding to the target image to obtain a cartoon character expression model mapped to a 2D plane, and rendering it to obtain a cartoon character expression animation; and aligning the cartoon character expression animation with the target user's face region image to obtain the cartoon character expression animation image corresponding to the target image, thereby improving the driving effect of the cartoon character expression animation image.

[0011] In one possible implementation, animated images of cartoon character expressions are continuously output to form a real-time facial expression-driven animation of a cartoon digital human corresponding to the target user. This includes: sequentially connecting the animated images of cartoon character expressions, and continuously outputting all the connected animated images of cartoon character expressions to form a real-time facial expression-driven animation of a cartoon digital human corresponding to the target user. This can improve the social experience of the target user.

[0012] In one possible implementation, the target loss function includes a first target loss function, a second target loss function, and a third target loss function. The construction method of the cartoon character expression-driven network model is as follows: Sample images containing facial information of sample users are acquired; using preset coefficient priors and rotation priors as constraints, input-output data pairs consisting of 3D keypoint information of the sample user's face, sample cartoon character expression basis coefficients, and sample rotation feature representations are constructed using the sample images, serving as the training image dataset; the initial cartoon character expression-driven network model is trained using the training image dataset, the first target loss function, the second target loss function, and the third target loss function to generate the cartoon character expression-driven network model. In this way, based on an incompletely supervised training strategy, a massive training dataset is generated using a small amount of data, effectively improving the training accuracy of the model.

[0013] In one possible implementation, a first objective loss function is used to constrain the basis coefficients of the cartoon character expressions output by the cartoon character expression-driven network model, making them as close as possible to the basis coefficients of the sample cartoon character expressions in the training image dataset; a second objective loss function is used to constrain the rotation feature representation output by the cartoon character expression-driven network model, making it as close as possible to the sample rotation feature representation in the training image dataset; and a third objective loss function is used to simultaneously constrain the basis coefficients of the cartoon character expressions and the rotation feature representation output by the cartoon character expression-driven network model, making them as close as possible to the basis coefficients of the sample cartoon character expressions and the sample rotation feature representation in the training image dataset, respectively.

[0014] In one possible implementation, the rotation feature is represented as a 6D rotational transformation representation.

[0015] Secondly, this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor is used to call and execute the computer program to implement the real-time expression driving method for cartoon digital humans as described in any of the first aspects above.

[0016] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when run by a processor of an electronic device, is used to implement the real-time expression driving method for cartoon digital humans as described in any one of the first aspects above.

[0017] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to execute the real-time facial expression driving method for cartoon digital humans as described in any of the first aspects. Attached Figure Description

[0018] Figure 1 This is one of the scenario illustrations provided in the embodiments of this application;

[0019] Figure 2 A schematic diagram of a mobile terminal device provided in an embodiment of this application;

[0020] Figure 3 This is a software structure block diagram of a mobile terminal device provided in an embodiment of this application;

[0021] Figure 4 A flowchart of the real-time facial expression driving method for cartoon digital humans provided in the embodiments of this application;

[0022] Figure 5 A schematic diagram illustrating the process of determining 3D key point information of a target user's face using a FaceMesh network, as provided in an embodiment of this application.

[0023] Figure 6A schematic diagram illustrating the process of extracting latent spatial feature representations using a key information encoder, provided in an embodiment of this application;

[0024] Figure 7 The mixer provided in the embodiments of this application m，n A schematic diagram of the network structure of (.);

[0025] Figure 8 This is a schematic diagram of the network structure of the cartoon character facial expression base coefficient prediction module provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of the network structure of the rotation representation prediction module provided in the embodiments of this application;

[0027] Figure 10 This is a schematic diagram illustrating the overall implementation process of real-time facial expression driving for cartoon digital humans provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0031] To enable those skilled in the art to better understand the solution of this application, the application scenario of the technical solution of this application will be described first.

[0032] See Figure 1 The illustration shows a scenario diagram provided by an embodiment of this application.

[0033] In this example scenario, when a user engages in social communication using their mobile phone, such as video chatting with others through social media apps or recording videos and live streaming on short video apps, the user typically captures an image of their face to create a cartoonish digital avatar, enhancing the fun of social interaction. Furthermore, the user can select or design different elements of the cartoon avatar according to personal preferences, such as face shape, eyes, hairstyle, and clothing, and then combine these elements to create a unique cartoonish digital avatar for social communication.

[0034] Traditional methods for driving facial expressions in cartoon digital humans typically begin by acquiring multi-view RGBD images using an image acquisition device. RGB represents the color modes of the red, green, and blue color channels, which can be used to create different color images. D stands for depth map, providing distance information. Then, based on the acquired multi-view RGBD images, a facial point cloud is generated, which is further used to generate a three-dimensional (3D) facial mesh, achieving a 2D to 3D conversion and completing the facial expression driving process for the cartoon digital human. However, this method is cumbersome in terms of image acquisition and places high demands on the image acquisition device, requiring it to have depth acquisition capabilities.

[0035] To address the aforementioned issues with traditional driving methods, the current common approach is to use artificial intelligence (AI) neural network models to drive the facial expressions of cartoon digital humans. However, this method remains costly and technically challenging, and is also limited by the current intelligence level and computing resources of AI, making it difficult for the model to achieve ideal prediction accuracy.

[0036] Furthermore, with the increasing use of mobile devices in people's lives and work, such as live video streaming or video chatting anytime using mobile phones, the implementation of cartoon digital human expression driving on mobile devices also faces challenges in terms of real-time requirements. This is because neither of the two existing methods mentioned above for driving cartoon digital human expressions can meet the real-time requirements of mobile devices. Whether considering device computing power, power consumption, or the simultaneous multitasking of devices, these factors will place a certain burden on the data calculation and processing of mobile devices, thus affecting the implementation of real-time expression driving for cartoon digital humans. Moreover, even if expression driving technology relies on cloud computing power, the uploading and downloading of data will still increase the input and output latency to some extent, which will also affect the real-time performance of cartoon digital human expression driving.

[0037] Therefore, how to achieve near real-time input of human face video frames and output of cartoon digital human facial expressions to achieve a more realistic driving effect is a technical problem that urgently needs to be solved.

[0038] To overcome the above technical problems, this application provides a real-time expression-driving method for cartoon digital humans. A mobile-friendly cartoon character expression-driving network model is pre-constructed based on fully connected neural networks and convolutional structures. This not only improves driving accuracy and speed but also facilitates the quantitative deployment of various tools on mobile devices, thereby reducing the performance requirements of the mobile device processor without compromising network computing performance. This enables users to achieve high real-time and high-precision expression driving for cartoon digital humans when engaging in social communication using mobile devices such as smartphones, thus providing users with a more novel and interesting social experience.

[0039] It should be noted that the real-time expression driving method for cartoon digital humans provided in this application embodiment can be applied to mobile terminal devices such as mobile phones, tablets, personal digital assistants (PDAs), desktop, laptop, and notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices (hereinafter referred to as terminal devices or electronic devices).

[0040] To enable those skilled in the art to better understand the real-time expression driving method for cartoon digital humans provided in this application, the hardware architecture and software system architecture of the electronic device implementing the real-time expression driving method for cartoon digital humans will be described in detail below.

[0041] See Figure 2 The diagram illustrates a mobile terminal device provided in an embodiment of this application.

[0042] like Figure 2 As shown, the mobile terminal device 200 may include a processor 210, a mobile communication module 220, a wireless communication module 230, a sensor module 240, a display screen 250, an internal memory 260, a camera 270, an audio module 280, a speaker 280A, a receiver 280B, a microphone 280C, a headphone jack 280D, an antenna group 1, and an antenna group 2.

[0043] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile terminal device 200. In other embodiments of this application, the mobile terminal device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] Processor 210 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. For example, it can use a camera 270 pre-installed on the mobile terminal device 200 to capture image frames containing human faces at a fixed frame rate. Then, by processing and manipulating the data from each image frame, character expression animation frames can be obtained. After connecting and outputting all the obtained character expression animation frames, real-time cartoon digital human facial expression driving can be achieved.

[0045] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0046] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0047] The sensor module 240 can be used to acquire data signals related to various aspects of the mobile terminal device 200, serving as a basis for implementing corresponding functions. In some embodiments, the sensor module 240 may include, but is not limited to, image sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, temperature sensors, pressure sensors, etc.

[0048] The display screen 250 is used to display images, videos, etc., such as displaying a customized cartoon digital avatar corresponding to a user taking a selfie using the mobile terminal device 200. The display screen 250 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include one or P display screens 250, where P is a positive integer greater than 1.

[0049] Internal memory 260 can be used to store computer executable program code, including instructions. Internal memory 260 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound capture, image capture, etc.). The data storage area may store data created during the use of the mobile terminal device 200 (such as audio data, image data, etc.). Furthermore, internal memory 260 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of the mobile terminal device 200 by running instructions stored in internal memory 260 and / or instructions stored in memory located in the processor.

[0050] In some embodiments, the internal memory 260 stores instructions for executing a real-time expression-driven method for a cartoon digital human. The processor 210 can execute the instructions stored in the internal memory 260 to specifically perform the following functions: First, it pre-trains a cartoon character expression-driven network model based on a fully connected neural network and convolutional structure, using training images and a target loss function, under incomplete supervision. Then, it deploys this model to the mobile terminal device 200. Next, upon receiving a cartoon digital human generation instruction triggered by a user's camera on the mobile terminal device 200's camera interface, it can respond to this instruction by acquiring images (hereinafter referred to as target images) containing the user's (hereinafter referred to as the target user's) facial information via the camera 270 at a preset frequency (e.g., 30fps). Then, it inputs each target image frame-by-frame into a FaceMesh network for 3D reconstruction of the target user's face, estimating 468 3D facial landmarks in real time as the target user's 3D facial keypoint information. Next, the 3D key point information of the target user's face is input into the cartoon character expression-driven network model to predict the cartoon character's expression basis coefficients and rotation feature representations. Then, using these cartoon character expression basis coefficients and rotation feature representations, combined with a preset blendshape parameter model (referring to a predefined set of facial shapes), and based on relevant computer graphics knowledge, rendering is performed to obtain the cartoon character expression animation image corresponding to the target image. All the obtained cartoon character expression animation images are then continuously output in sequence to form a high-real-time, high-precision cartoon digital human real-time facial expression-driven animation corresponding to the target user.

[0051] The camera 270 is used to capture still images or videos. For example, when a user holds the mobile terminal device 200, they can use the camera 270 installed on the mobile terminal device 200 to acquire images containing the user's facial information at a fixed frequency (e.g., 30fps). In some embodiments, the mobile terminal device 200 may include one or K cameras 270, where K is a positive integer greater than 1.

[0052] The mobile terminal device 200 implements display functions through a GPU, a display screen 250, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 250 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0053] The mobile terminal device 200 can implement audio functions through an audio module 280, a speaker 280A, a receiver 280B, a microphone 280C, a headphone jack 280D, and an application processor. Examples include music playback and recording for voice input and output.

[0054] The audio module 280 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 280 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 280 may be located in the processor 210, or some functional modules of the audio module 280 may be located in the processor 210.

[0055] The speaker 280A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The mobile terminal device 200 can listen to music or make hands-free calls through the speaker 280A.

[0056] The receiver 280B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the mobile terminal device 200 answers a phone call or voice message, it can listen to the voice by bringing the receiver 280B close to the user's ear.

[0057] Microphone 280C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 280C, inputting the sound signal into microphone 280C. Mobile terminal device 200 may be equipped with at least one microphone 280C. In some embodiments, mobile terminal device 200 may be equipped with two microphones 280C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 200 may be equipped with three, four, or more microphones 280C, enabling sound signal collection, noise reduction, sound source identification, and directional recording functions, etc.

[0058] The headphone jack 280D is used to connect wired headphones and does not restrict the standard attributes of the jack.

[0059] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the mobile terminal device 200.

[0060] The wireless communication function of the mobile terminal device 200 can be implemented through antenna 1, antenna 2, mobile communication module 220, wireless communication module 230, modem processor, and baseband processor.

[0061] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile terminal device 200 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0062] The mobile communication module 220 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the mobile terminal device 200. The mobile communication module 220 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 220 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 220 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 220 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 220 and at least some modules of the processor 210 may be housed in the same device.

[0063] The wireless communication module 230 can provide solutions for wireless communication applications on the mobile terminal device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 230 can be one or more devices integrating at least one communication processing module. The wireless communication module 230 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 230 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0064] In addition, the mobile terminal device 200 runs an operating system on top of the aforementioned components. Examples include iOS, Android, and Windows operating systems. Applications can be installed and run on this operating system.

[0065] See Figure 3 It shows a schematic diagram of the software structure of the mobile terminal device provided in the embodiments of this application.

[0066] The software system of the mobile terminal device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the mobile terminal device 200.

[0067] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer (APK), the application framework layer (Framework), the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0068] The application layer can include a series of application packages (APPs). For example... Figure 3As shown, the application package may include applications such as camera, call, navigation, WLAN, Bluetooth, and gallery. When the user is taking pictures with the handheld mobile terminal device 200, the camera application can communicate with the camera-related devices in the camera access interface of the frame layer to request camera functions and obtain image data, etc.

[0069] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example... Figure 3 As shown, the application framework layer can include a window manager, notification manager, resource manager, and camera access interface (including but not limited to camera management, camera devices, etc.). The application framework layer can be used to implement interaction between camera services and camera APIs. That is, it provides a unified interface, enabling different camera hardware to interact with different camera applications. The framework layer also handles many common aspects of camera functionality, such as autofocus.

[0070] The window manager is used to manage window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots, among other things.

[0071] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0072] In this embodiment, the Hardware Abstraction Layer (HAL) provides a set of standard interfaces, enabling the framework layer to communicate with camera hardware from various manufacturers without needing to understand the underlying hardware details. The HAL stores the hardware abstraction layer itself and camera algorithm libraries, etc. It should be noted that the real-time expression-driven algorithm for cartoon digital humans provided in this application can be stored in the camera algorithm library of the HAL layer, such as... Figure 3 As shown.

[0073] The real-time facial expression driving algorithm for cartoon digital humans first obtains the user's cartoon digital human generation command through interaction with the user on the shooting interface (such as the interface where the user takes a selfie in a short video application). Then, in response to the command, it uses a camera to capture image frames containing the user's face at a fixed frame rate. The algorithm then processes and manipulates these image frames to obtain character facial animation frames. By connecting and outputting all the obtained character facial animation frames, real-time facial expression driving of the cartoon digital human can be achieved. Specifically, upon receiving the cartoon digital human generation command triggered by the user on the shooting interface of the mobile terminal device 200, the algorithm can respond by acquiring target images containing the user's facial information through the camera 270 at a fixed frequency (e.g., 30fps). Each target image is then input frame by frame into the FaceMesh network for 3D reconstruction processing of the user's face image, estimating 468 3D facial landmarks in real time as the user's 3D facial key point information. Next, the 3D key point information of the user's face is input into a pre-constructed cartoon character expression-driven network model to predict the cartoon character's expression basis coefficients and rotation feature representations. Then, using these cartoon character expression basis coefficients and rotation feature representations, combined with a predefined set of facial shapes (such as 52 BlendShape parameter models), and based on relevant computer graphics knowledge, rendering is performed to obtain the cartoon character expression animation image corresponding to the target image. All the obtained cartoon character expression animation images are then continuously output in sequence to form a high-real-time, high-precision cartoon digital human real-time facial expression-driven animation for the target user.

[0074] The hardware layer may include the hardware components of the aforementioned electronic device. For example, Figure 3 The exhibits included cameras, image signal processors, digital signal processors, and graphics processors.

[0075] Among them, the camera is used to perform focusing, exposure processing, etc., when acquiring images containing facial information at a fixed frequency (such as 30fps).

[0076] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the mobile terminal device 200 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0077] In this way, through the application layer, framework layer, HAL layer, driver layer and hardware layer, interoperability between different camera applications and hardware devices on the Android platform can be achieved, enabling high real-time performance and high-precision expression driving for cartoon digital humans, thereby providing users with a more novel and interesting social experience.

[0078] The technical solutions involved in the following embodiments can all be implemented in electronic devices with the above-described hardware and software architectures.

[0079] Next, the specific implementation process of the cartoon digital human real-time expression driving method provided in this application will be described in detail:

[0080] like Figure 4 As shown, the specific implementation process of this cartoon digital human real-time expression driving method may include the following steps S401-S404:

[0081] S401: The terminal device acquires a target image containing the facial information of the target user through a camera at a preset frequency.

[0082] In this embodiment, when the terminal device (such as a mobile phone) and the user are in a shooting interface (such as the interface of a short video application installed on the mobile phone for taking a selfie, such as...) Figure 1 When interacting with the user and receiving the instruction to generate a cartoon digital human, the device can first use a pre-configured camera (such as the front camera of a mobile phone) on the terminal device (such as a mobile phone) to acquire an image containing the user's facial information at a preset frequency, in order to execute the subsequent step S402.

[0083] It should be noted that, for the purpose of explaining the real-time facial expression driving method for cartoon digital humans proposed in this application, the image containing the user's facial information acquired by the terminal device through the camera at a preset frequency in step S401 is defined as the target image, and the user contained therein is defined as the target user. In this way, by executing the subsequent steps S402-S404, the acquired target images are processed and manipulated to generate a high real-time and high-precision facial expression driving animation for the cartoon digital human corresponding to the target user.

[0084] Furthermore, it should be noted that this application does not limit the preset frequency for the terminal device to acquire the target image. It can be set according to actual conditions and empirical values. For example, the preset frequency can be set to 30fps, meaning 30 frames of the target image can be captured per second. In addition, this embodiment does not limit the type of target image. For example, the target image can be any color image composed of the three primary colors: red (R), green (G), and blue (B), and is defined as I. Furthermore, this embodiment does not limit the size of the target image, i.e., I∈R h*w*3 , where h and w represent the height and width of the target image I, respectively, and their specific values ​​are not limited; 3 indicates that the target image I contains pixel information in three channels: red (R), green (G), and blue (B).

[0085] S402: Input the target image into the FaceMesh network to perform 3D reconstruction of the target user's face image, and obtain the 3D key point information of the target user's face.

[0086] In this embodiment, after the terminal device acquires each frame of target image containing the target user's facial information through the camera at a preset frequency (e.g., 30fps) in step S401, in order to improve the real-time processing of the target image, the target image can be input into a FaceMesh network frame by frame for 3D reconstruction processing of the target user's facial image to obtain the target user's facial 3D key point information, which is then used to execute the subsequent step S403.

[0087] It should be noted that this application employs the FaceMesh network to extract 3D key point information of the target user's face because, compared to traditional methods that require multi-view RGBD images for expression-driven analysis, the FaceMesh network can infer the geometry of the 3D facial surface using machine learning (ML) with only input from a single camera, without the need for a dedicated depth sensor. It utilizes a lightweight model architecture and GPU acceleration throughout the pipeline, providing real-time performance. Furthermore, it can be combined with the FaceTransform module to bridge the gap between facial landmark estimation and practical real-time augmented reality (AR) applications. It also establishes a measurable 3D space and uses the screen positions of facial landmarks to estimate facial transformations within that space, thus meeting the high real-time requirements of mobile devices.

[0088] Specifically, one possible implementation is, as follows: Figure 5 As shown, the specific implementation process of step S402 may include: First, cropping the target user's face region from the target image to obtain a target user face region image. This application does not limit the image cropping method; it can be set according to actual conditions and empirical values. Then, performing rotation and size transformation alignment processing on the target user face region image to obtain an aligned target user face region image, which is then defined as I. i And make I i The size meets the image data size standard that FaceMesh network can process, i.e., I i ∈R 256*256*3 Simultaneously, it can also calculate the rotation inverse transformation matrix representation corresponding to the target user's facial region image, which is different from defining it as M. rotation And M rotation ∈R 3*3 .

[0089] Next, the aligned target user's facial region image I can be... iThe FaceMesh network is used to perform 3D reconstruction of the target user's face image, resulting in 468 facial key point information points for the target user. The specific processing formula is as follows:

[0090] landmarks = F θ (I i (1)

[0091] Among them, F θ (·) represents the image processing function of the FaceMesh network; landmarks represent the aligned target user face region image I of the FaceMesh network. i The above detects and tracks 468 face markers, i.e., landmarks ∈ R. 468*3 These landmarks include various feature points on the target user's face, such as the corners of the eyes, the tip of the nose, and the corners of the mouth, as well as more detailed points related to facial expressions. Each landmark consists of x, y, and z coordinates, where the x and y coordinates are normalized to the range of [0.0, 1.0] of the image width and height, while the z coordinate represents the depth of the landmark, with the depth of the center of the head as the origin. The smaller the value, the closer the landmark is to the camera.

[0092] Then, using the 468 facial landmarks of the target user and the previously calculated inverse rotation transformation matrix corresponding to the target user's facial region image, M is represented. rotation This allows us to calculate the 3D key point information of the target user's face and define it as landmarks. ori The specific processing formula is as follows:

[0093] landmarks ori =landmarks×M rotation (2)

[0094] Among them, landmarks ori ∈R 468*3 .

[0095] S403: Input the 3D key point information of the target user's face into the pre-built cartoon character expression-driven network model to predict the cartoon character expression basis coefficients and rotation feature representations; wherein, the cartoon character expression-driven network model is based on a fully connected neural network and a convolutional structure, and is obtained by incompletely supervised training using training images and a target loss function.

[0096] In this embodiment, the terminal device obtains the 3D key point information (landmarks) of the target user's face through step S402. oriSubsequently, to achieve the desired driving effect, the target user's facial 3D key point information landmarks can be further analyzed. ori The pre-built cartoon character expression-driven network model is input to predict the cartoon character's expression. The predicted expression basis coefficients and rotation feature representations are defined as w and r, respectively, to execute the subsequent step S404, thereby achieving high real-time and high-precision expression driving for the cartoon digital human corresponding to the target user.

[0097] It should be noted that, in order to improve the real-time expression-driven effect of the cartoon digital human, this application pre-uses a training image dataset consisting of a small number of sample images containing facial information of sample users and a large number of unlabeled image data generated based on these sample images. Combined with a target loss function, this dataset performs incomplete supervised training on an initial cartoon character expression-driven network model composed of fully connected neural networks and convolutional structures, thereby constructing a cartoon character expression-driven network model with better prediction performance. Furthermore, this application does not limit the specific network structure of the cartoon character expression-driven network model and can select and set it according to the actual situation. One optional implementation is that the pre-constructed cartoon character expression-driven network model may include, but is not limited to, a key information encoder, a cartoon character expression basis coefficient prediction module, and a rotation representation prediction module. For a detailed description of the specific training process of the cartoon character expression-driven network model in subsequent embodiments, it will not be repeated here.

[0098] Specifically, one optional implementation is that the process of "inputting the 3D key point information of the target user's face into a pre-built cartoon character expression-driven network model to predict the cartoon character expression basis coefficients and rotation feature representation" in step S403 can include the following steps S4031-S4033:

[0099] S4031: Input the 3D key point information of the target user's face into the key information encoder of the pre-built cartoon character expression-driven network model for feature extraction to obtain the latent spatial feature representation.

[0100] In this implementation, the terminal device obtains the 3D key point information (landmarks) of the target user's face in step S402. ori Then, further, the 3D key point information of the target user's face can be used as landmarks. ori The key information encoder of the input cartoon character expression driven network model is used to extract features, obtain the latent space feature representation, and define it as feats for subsequent step S4032.

[0101] In this embodiment, the specific structure of the key information encoder is not limited. To reduce the performance requirements of the terminal device's processor and meet the real-time requirements of the terminal device as much as possible, the key information encoder used in the pre-constructed cartoon character expression-driven network model is an encoder built based on a fully connected network structure. The function of this encoder is to analyze and calculate the 3D key point information landmarks of the target user's face. ori The system extracts features from the data and outputs a feature tensor in the latent space as the feature representation of the latent space, providing prior information for subsequent steps S4032 and S4033 to calculate the basis coefficients and rotation feature representation of cartoon character expressions.

[0102] Specifically, in the feature extraction process, such as Figure 6 As shown, this key information encoder borrows the structure of a multilayer perceptron mixer (MLP-Mixer) network, and now encodes the 3D key point information of the target user's face into a key information encoder. ori (and landmarks) ori ∈R 468*3 The image is divided into 468 one-dimensional patch tensors of length 3, each representing a single keypoint, and defined as landmarks. ori ={l1, l2, ..., l 468}, where l i ∈R 3 Let i = 1, 2, ..., 468. Then, normalize all keypoint information using a single patch tensor as the unit to obtain l′. i =LayerNorm(l i ), then for all l′ i By superimposing and transposing the channels, we obtain l′∈R 3*468 Then, divide l′ into l′={l′1, l′2, l′3} in units of channels. j ∈R 468 j = 1, 2, 3, l′ corresponding to each channel j Input a fully connected network mixer 1，j (.), all mixers 1，j (l′ j The results will be output separately. j ∈R 108108 represents a custom dimension example value, which can also be set to other dimensions below 468 according to actual situation and experience. On the one hand, it can extract key features while achieving dimensionality reduction, and on the other hand, it can facilitate subsequent steps S4032 and S4033 to calculate the cartoon character expression basis coefficients and rotation feature representation more quickly and accurately, ensuring the consistency of the feature matrix dimension ratio.

[0103] Next, l″ j ∈R 108 By superimposing and transposing the data in the form of channels, we obtain l″∈R 108*3 Then, divide l″ into l″ = {l″1, l″2, ..., l″} in units of channels. 108},l″ k ∈R 3 k = 1, 2, ..., 108, for each channel corresponding to l″ k Normalize the tensors respectively to obtain LayerNorm(l″) k The normalized results are then further input into a fully connected network mixer. 2，k (.),like Figure 6 As shown, all mixers 2，k (LayerNorm(l″ k The output will be l″′ respectively. k ∈R 72 Finally, by superimposing them in channel form, we can obtain the feature tensor features of the latent space, where features∈R. 108*72 The ratio of 108 to 72 corresponds to the 6D rotation transformation mentioned in subsequent step S4033, which represents r∈R. 3*2 The ratio is consistent, 3 to 2. It is understandable that this dimension is not a fixed value and can be adjusted adaptively according to the actual situation and experimental conditions.

[0104] It should be noted that the mixer in the key information encoder... m，n (.) represents a simple fully connected residual structural unit. Its function is to extract and analyze features in another dimension of the input tensor while preserving the features of one dimension of the input tensor. Specifically, in mixer... 1，j (.), mixer 2，k (.) respectively extract features from facial key point information under specific spatial information and spatial information under specific facial key point information.

[0105] It should also be noted that this application pertains to the mixer. m，nThe specific composition of (·) is not limited, and can be as follows: Figure 7 As shown. For example, with a mixer 1，j Taking the network structure of (·) as an example, the operation process of this fully connected residual basic unit can be expressed as: input tensor l″1∈R 468 After passing through a linear layer nn.Linear(), a computational tensor l∈R is obtained. n The tensor length n can be customized according to the actual situation. After passing through a Gaussian error linear unit activation layer nn.GeLU() and a linear layer nn.Linear(), the intermediate tensor l' of the fully connected residual unit is obtained. mid ∈R m The length m of this tensor can be customized according to the actual situation, and then the intermediate tensor l′ is... mid With input tensor l′1∈R 468 By splicing, we get l′ mid ∈R m+468 Finally, after passing through a linear layer nn.Linear(), the final output is l″1∈R. 108 .

[0106] S4032: Input the latent spatial feature representation into the cartoon character expression basis coefficient prediction module of the cartoon character expression driven network model for mean dimensionality reduction processing to obtain the cartoon character expression basis coefficients.

[0107] In this implementation, after obtaining the latent spatial feature representations (feats) in step S4031, the target can be further input into the cartoon character expression basis coefficient prediction module of the cartoon character expression-driven network model using the latent spatial feature representations (feats). (This application defines this module as D...) coeff (·)) is subjected to mean dimensionality reduction so that a set of facial expression basis coefficients is output through two simple fully connected layers. This set serves as the basis coefficients for cartoon character facial expressions and is defined as w, where w∈R 52 w = {w i |i=1,2,...,52} is used to execute the subsequent step S404.

[0108] The reason why w is set to 52 dimensions in this embodiment is precisely because it takes into account the 52 preset blend shape parameter models, that is, the 52 predefined facial shapes, including left eye blinking (eyeBlinkLeft), left eye looking down (eyeLookDownLeft), left eye looking at the tip of the nose (eyeLookInLeft), left eye looking to the left (eyeLookOutLeft), left eye looking up (eyeLookUpLeft), left eye squinting (eyeSquintLeft), left eye widening (eyeWideLeft), right eye blinking (eyeBlinkRight), right eye looking down (eyeLookD... (ownRight), right eye looking at the tip of the nose (eyeLookInRight), right eye looking to the left (eyeLookOutRight), right eye looking upward (eyeLookUpRight), right eye squinting (eyeSquintRight), right eye wide open (eyeWideRight), chin forward when pursing (jawForward), chin to the left when pouting (jawLeft), chin to the right when pouting (jawRight), chin down when opening the mouth (jawOpen), mouth closed (mouthClose), mouth slightly open with lips parted (mouthFunnel), mouth pursed (mouthPucker), pouting to the left (m (outhLeft), (mouthRight), (mouthSmileLeft), (mouthSmileRight), (mouthFrownLeft), (mouthFrownRight), (mouthDimpleLeft), (mouthDimpleRight), (mouthStretchLeft), (mouthStretchRight), (mouthRollLower), (mouthRollLower). Lower lip down (mouth shrug lower), upper lip up (mouth shrug upper), lower lip pressed to the left (mouth press left), lower lip pressed to the right (mouth press right), lower lip pressed to the lower left (mouth lower down left), lower lip pressed to the lower right (mouth lower down right), upper lip pressed to the upper left (mouth upper up left), upper lip pressed to the upper right (mouth upper up right), left eyebrow outward (brow down left), right eyebrow outward (brow down right)The five facial features are: frowning (browInnerUp), left eyebrow pointing to the upper left (browOuterUpLeft), right eyebrow pointing to the upper right (browOuterUpRight), cheek puffing outward (cheekPuff), left cheek pointing upward and rotating (cheekSquintLeft), right cheek pointing upward and rotating (cheekSquintRight), nose twitching to the left (noseSneerLeft), nose twitching to the right (noseSneerRight), and tongue sticking out (tongueOut). The 'w' represents 52 sets of feature motion data, used to drive these 52 predefined facial shapes to form facial expression animations corresponding to the target user's 3D face image in the input target image.

[0109] It should be noted that this embodiment does not limit the specific composition and structure of the cartoon character expression basis coefficient prediction module. In order to reduce the performance requirements of the terminal device processor and meet the real-time requirements of the terminal device as much as possible, the cartoon character expression basis coefficient prediction module used in this application when pre-constructing the cartoon character expression driving network model is a decoder built based on a fully connected network structure. The encoder D coeff The function of (·) is to represent the latent space features, such as ∈ R. 108*72 Decoding is performed to obtain the basic coefficients w∈R of the cartoon character's facial expressions. 52 w = {w i |i = 1, 2, ..., 52}. Where w i The specific value is not limited, but the value range can be set between 0 and 1. 0 means that the corresponding facial shape is not activated at all, and 1 means that the corresponding facial shape is fully activated. The closer the value is to 1, the more obvious the facial expression is. For example, when the lower lip is pressed to the left (mouthPressLeft), 0 means not activated and 1 means fully activated.

[0110] like Figure 8 As shown, in this decoding process, for the latent spatial feature representation features∈R 108*72 The process of performing mean-based dimensionality reduction is as follows: First, the latent space feature tensor features ∈ R... 108*72 Represented as Among them, f i ∈R 72 Then for all f i Taking the average of i = 1, 2, ..., 108, we get... and Next, after passing through a linear layer nn.Linear() and a softmax() activation layer, the final output is the prediction result w∈R of the cartoon character's facial expression basis coefficients w. 52 .

[0111] S4033: Input the latent spatial feature representation into the rotation representation prediction module of the cartoon character expression-driven network model for size transformation and convolution processing to obtain the rotation feature representation.

[0112] In this implementation, after obtaining the latent spatial feature representations (feats) in step S4031, the target can be further input into the rotation representation prediction module of the cartoon character expression-driven network model (defined as E in this application) using the latent spatial feature representations (feats). mlp (.)) Perform size transformation and simple convolution to obtain a 6D rotation transformation representation, which is used as a rotation feature representation and defined as r, where r∈R 3*2 This is used to execute the subsequent step S404.

[0113] This application does not limit the specific format of the rotation feature representation. The reason this embodiment uses 6D rotation transformation representation as the rotation feature representation is that: 6D rotation transformation representation can map the rotation changes corresponding to the 3D model of the cartoon character's facial animation to a 2D plane; simultaneously, the 3D rotation matrix can also be identically mapped to the corresponding 3*3 rotation matrix representation through specific mathematical operation rules. Furthermore, the advantage of 6D rotation transformation representation is that this representation method can maintain the continuity of rotation in deep learning (emphasis on continuity), thereby improving the model's learning effect. In deep learning, 6D representation outperforms traditional rotation representation in tasks such as rotation estimation and inverse kinematics. Some traditional rotation representation methods may lead to discontinuities in neural network training, thus affecting model performance. Therefore, this embodiment utilizes the aforementioned advantages of 6D rotation transformation representation, using it as the representation of the cartoon digital human's head rotation action (i.e., rotation feature representation) in deep learning tasks.

[0114] It should be noted that this embodiment does not limit the specific composition and structure of the rotation representation prediction module. In order to reduce the performance requirements of the terminal device processor and meet the real-time requirements of the terminal device as much as possible, the rotation representation prediction module used in this application when pre-constructing the cartoon character expression-driven network model is a decoder based on a convolutional structure. The encoder E mlp The function of (.) is to represent the latent space features, such as features ∈ R. 108*72 Decoding is performed to obtain the rotation feature representation r (e.g., the 6D rotation transformation representation r∈R). 3*3 ).

[0115] like Figure 9 As shown, in this decoding process, for the latent spatial feature representation features∈R 108*72 First, a feature size transformation (resize) operation is performed to obtain the intermediate feature f″∈R. 96*64Among these, the resize operation is a fundamental and important step in computer vision, effectively adjusting the image size to meet the needs of different applications. Specifically, it involves interpolating the feature map by combining the feature values ​​of adjacent pixels to change the feature map size. Then, after two convolution operations (with a kernel size of 3, feature block edge padding of 1, and stride of 2), we obtain f″∈R. 24*16 Finally, after another feature size transformation, the 6D rotation transformation representation r∈R can be obtained. 3*2 .

[0116] S404: Utilizing the basic coefficients and rotation features of cartoon character expressions, combined with the preset BlendShape parameter model, renders the cartoon character expression animation image corresponding to the target image; and continuously outputs the cartoon character expression animation image to form a real-time facial expression driven animation of the cartoon digital human corresponding to the target user.

[0117] In this embodiment, the terminal device uses the cartoon character expression-driven network model in step S403 to predict the basic coefficients w of the cartoon character's expression (e.g., w∈R). 52 ) and rotational feature representation r (i.e., 6D rotational transformation representation r∈R) 3*2 After that, in order to achieve the desired driving effect, the cartoon character's expression basis coefficient w can be used to perform a weighted summation on the preset blend shape parameter model (i.e., the 52 predefined facial shapes mentioned in step S4032 above) to predict the cartoon character's expression model corresponding to the target image. Then, the rotation feature representation r (i.e., the 6D rotation transformation representation r∈R) is used. 3*2 The algorithm performs rotation transformation on the cartoon character expression model corresponding to the predicted target image to obtain a cartoon character expression model mapped to a 2D plane. This model is then rendered to generate a cartoon character expression animation. Next, based on the target user's facial region image, the cartoon character expression animation is aligned with the target user's face to obtain the cartoon character expression animation image corresponding to the target image. Finally, all the obtained cartoon character expression animation images are concatenated in chronological order, and the concatenated cartoon character expression animation images are continuously output to form a real-time facial expression driven animation of the target user's cartoon digital human, thereby improving the target user's social experience.

[0118] Specifically, since each BlendShape parameter model corresponds to a specific facial deformation (such as frowning), complex facial expressions can be generated when multiple BlendShape parameter models are combined with different weights. Technically, the BlendShape parameter model is a dictionary storing motion factors of user facial expression features, containing 52 sets of feature motion data (i.e., weight combination coefficients). Taking the 52 predefined facial shapes (i.e., 52 sets of facial expression feature 3D mesh models) mentioned in step S4032 above as an example, these 52 facial shapes, combined with these motion factors (i.e., cartoon character expression basis coefficients w∈R), 52 This enables the transformation of 2D facial images into 2D or 3D character models. This embodiment utilizes the BlendShapes parametric model as its technical foundation, leveraging the cartoon character's facial expression basis coefficients w∈R. 52 In addition, there are 52 predefined facial shapes (i.e., 52 sets of 3D mesh models of facial expression features). Various expressions of cartoon characters can be realized through a weighted summation method. The specific formula is as follows:

[0119]

[0120] Where b represents the predicted cartoon character facial expression model; b0 represents the baseline shape, typically the facial shape of a neutral expression, and all BlendShapes parametric models are deformed based on this baseline shape. i |i=1,2,…,52} represents 52 predefined facial shapes (i.e., 52 sets of 3D mesh models of facial expressions), each shape representing a specific facial deformity, such as frowning, etc., as detailed in the description of step S4032 above. i |i=1,2,...,52} represents the weight set of 52 predefined facial shapes (i.e., 52 sets of 3D mesh models of facial expression features), which is the basic coefficient w∈R of the cartoon character's facial expression predicted through step S403. 52 The value range is usually between 0 and 1, representing the degree of influence of the corresponding type of facial shape on the final facial expression.

[0121] In this way, for any cartoon character, a mesh model can be created based on 52 predefined facial shapes (i.e., 52 sets of facial expression feature 3D mesh models) and their corresponding baseline shape mesh model {b i |i=0,1,2,...,52}, combined with the cartoon character expression basis coefficients w∈R obtained through step S403 prediction. 52 w = {w i |i=1,2,...,52}, predict the corresponding cartoon character facial expression animation mesh model driven by the 2D face image.

[0122] Furthermore, based on the identity mapping between the 3D rotation 9*9 matrix and the 6D rotation transformation representation, and by combining the 6D rotation transformation representation with the 3D-mesh model of cartoon character facial expression animation, the rotation transformation of the character's head can be predicted simultaneously and mapped onto the 2D plane. Therefore, the cartoon character facial expression driven network model in this application can predict and output the 6D rotation transformation representation r∈R corresponding to the target user's face. 3*2 Then, using this 6D rotational transformation, we can represent r∈R 3*2 By performing a 6D rotation transformation on the predicted cartoon character expression model b, a cartoon character expression model mapped to a 2D plane can be obtained. Further rendering of the cartoon character expression model based on relevant computer graphics knowledge can then yield the cartoon character expression animation.

[0123] Finally, based on the target user's face region cropped from the target image as mentioned in step S402 above, the resulting target user face region image is aligned with the rendered cartoon character's facial animation. In this way, after capturing target image frames containing the target user's facial information in real time through the front-facing camera of the mobile terminal device at a preset frequency (e.g., 30fps), the image data can be processed and processed to obtain character facial animation frames by executing the process of steps S402-S404 above. Then, all character facial animation frames are connected and continuously output to achieve high-precision facial expression driving of the cartoon digital human in real time.

[0124] Next, this embodiment will provide a detailed description of the training process of the cartoon character expression-driven network model. The specific training process may include the following steps: AC:

[0125] Step A: Obtain a sample image containing facial information of the sample user.

[0126] It should be noted that, because this application employs basic neural network operators that are mobile-friendly when constructing the cartoon character expression-driven network model, it significantly improves the inference efficiency of mobile devices by utilizing a small number of parameters and a simple model structure (such as fully connected layers and convolutional structures). Simultaneously, it does not place excessive demands on the device's computational performance and power consumption. This, however, presents a certain challenge in terms of model training accuracy. From an empirical perspective, the inference performance of a model largely depends on the number of model parameters and the quality of the training data. Given a fixed model structure, the volume and quality of the training data greatly influence the computational quality of the model. The real-time expression-driven method for cartoon digital humans proposed in this application uses image frames containing user facial information as raw input data. The FaceMesh network inferences to obtain the user's 3D facial keypoint information, and then uses the cartoon character expression-driven network model to predict and output the cartoon character expression basis coefficients w∈R. 52 and rotational feature representation (i.e., 6D rotational transformation representation) r∈R 3*2 Therefore, obtaining high-precision training data (input data & output label data pairs) is particularly important.

[0127] To address this, this embodiment employs an incompletely supervised training strategy, utilizing a small amount of labeled data and a large amount of unlabeled data to train the cartoon character expression-driven network model, thereby improving the model's performance and generalization ability.

[0128] Specifically, to construct a cartoon character facial expression-driven network model, a small amount of raw data (i.e., a small number of sample images containing facial information of sample users) is first obtained as a foundation. Then, through subsequent step B, this data is processed, manipulated, and calculated to generate a large number of high-precision input-output data pairs, which serve as training data for model training. In this embodiment, the method and quantity of obtaining sample images are not limited. For example, the methods for obtaining a small amount of raw data (i.e., a small number of sample images containing facial information of sample users) can be summarized as follows:

[0129] (1) Baseline state cartoon character mesh created by art engineers;

[0130] (2) The 52 cartoon character facial expressions based on the mesh grid created by the art engineer;

[0131] (3) 8000 original, standard-state real face 3D meshes.

[0132] Step B: Using the preset coefficient prior and rotation prior as constraints, construct input-output data pairs consisting of 3D key point information of sample users' faces, basic coefficients of sample cartoon character expressions, and sample rotation feature representations using sample images, as the training image dataset.

[0133] It should be noted that, in order to achieve incomplete supervised training of the model, after obtaining a small amount of raw data (i.e., a small number of sample images containing facial information of sample users) through step A, a large number of input-output data pairs consisting of 3D key point information of sample users' faces, sample cartoon character expression basis coefficients, and sample rotation feature representations can be generated based on these small amount of raw data. These are defined as landmarksori-coefficients & rotation matrix input-output data pairs, which serve as the training image dataset for subsequent step C.

[0134] Specifically, in order to construct a training image dataset, it is first necessary to determine the constraints that the input and output data pairs that can be used as training images must satisfy. That is, it is necessary to determine the prior label data, including coefficient prior and rotation prior. These two parts serve as constraints, which can provide prior constraints for the generated sample cartoon character expression base coefficient labels and sample rotation feature representation labels in the label data of the subsequent generated input and output data pairs, so as to ensure that the generated input and output data pairs are labels that conform to the actual situation of human facial expressions.

[0135] The role of rotation prior is to limit the actual head rotation angle corresponding to all 6D rotation transformation representations, avoiding excessive rotation angles that could distort the training data. The rotation prior, as a constraint, specifically restricts the following: first, the mesh coordinates corresponding to the rotation center position are determined, typically at the center of the head's neck; then, the possible rotation angle range of the head in the x, y, and z dimensions is limited and defined as [-θ]. x θ x ],[-θ y θ y ],[-θ z θ z The rotation angles in the x-direction represent the angles at which a person tilts their head left or right, the y-direction represents the angles at which a person nods up or down, and the z-direction represents the angles at which a person shakes their head left or right. It should be noted that this application does not limit the range of rotation angles in these three dimensions; they can be set according to actual conditions and empirical values. For example, the rotation angle ranges in the x, y, and z dimensions can be set to [-45, 45], [-45, 45], [-70, 70], etc. However, the range of rotation angles in these three dimensions must satisfy the following condition: in actual situations, random head rotation can satisfy the realization of random facial expressions without distorted rotation.

[0136] Regarding the coefficient prior, different combinations of coefficients, combined with the predefined 52 facial shape mesh expression bases, can form different facial expression shapes through weighted summation. However, randomly selecting coefficient combinations for weighted summation easily leads to distorted facial expressions. This type of data is not helpful for network training and can even have a negative impact. Therefore, to limit the authenticity of the generated coefficient labels, this application proposes the following restrictions: First, the 52 facial shape mesh expression bases are grouped into four regions based on facial areas: left eye, right eye, mouth, and cheek. Within each region, mutually exclusive combinations of all facial shapes are extracted. No two facial shapes in each mutually exclusive combination can appear simultaneously; that is, in each mutually exclusive combination, only 0 or 1 facial shape can have a coefficient that is not zero. For example, "pouting to the left" and "pouting to the right" must exist in the same mutually exclusive combination; they cannot be activated simultaneously, as weighted combination of the two will result in distorted expressions.

[0137] Furthermore, after determining the label data prior, including coefficient prior and rotation prior, the preset coefficient prior and rotation prior can be used as constraints to construct a large number of input-output data pairs (i.e., landmarksori-coefficients & rotation matrix input-output data pairs) using the small number of sample images already acquired. These pairs consist of 3D key point information of sample users' faces, basic coefficients of sample cartoon character expressions, and sample rotation feature representations. These pairs serve as the training image dataset.

[0138] Specifically, firstly, 52 facial shape expression meshes corresponding to each standard state real face mesh can be generated. The deformation transfer technique used is as follows: given the original mesh information S, the deformed mesh information S′ based on the original mesh, and the target mesh information T, the mesh information T′' after the target mesh undergoes the same deformation as the original mesh can be calculated and generated. The two are consistent in shape, and there is a similar deformation relationship between S and S′' and between T and T′', and S and T do not need to have the same number of mesh vertices. Based on this, combined with the standard state mesh information of the cartoon character, the 52 facial expression shape mesh information of the cartoon character obtained in step A, and the mesh information of 8000 real faces in the standard state, 52 mesh information corresponding to the face of each sample user can be generated. In other words, 52 BlendShapes parameter models corresponding to each sample user among these 8000 real people can be generated.

[0139] Then, under the pre-defined constraints of coefficient prior and rotation prior, a large number of random sample cartoon character expression base coefficient combinations and sample 6D rotation transformation representations are randomly generated. A set of random sample cartoon character expression base coefficient combinations is weighted and combined with 52 sets of facial expression shapes corresponding to a random sample user to obtain the real facial expression mesh information corresponding to that sample user. Then, 468 3D facial keypoints are extracted from the corresponding mesh. A 6D rotation transformation representation satisfying the rotation prior is randomly assigned and is identity-mapped into a 3*3 rotation matrix. The extracted 3D keypoint information is rotated to obtain the 3D keypoint information of a specific expression of the corresponding sample user at a certain head rotation angle. This information serves as the input to the initial cartoon character expression-driven network model, and the 52 cartoon character expression base coefficient combinations and 6D rotation transformation representation serve as the output of the label-driven network, resulting in a set of landmarks-coefficients & rotation. By repeatedly sampling random coefficients and 6D rotation transformation representations from all 8000 real human sample users to satisfy the prior labels, a large amount of training dataset with guaranteed accuracy can be obtained.

[0140] Step C: Train the initial cartoon character expression-driven network model using the training image dataset, the first target loss function, the second target loss function, and the third target loss function to generate the cartoon character expression-driven network model.

[0141] After obtaining a large amount of training dataset with guaranteed accuracy through step B, the initial cartoon character expression-driven network model can be trained based on the obtained training dataset, the first objective loss function, the second objective loss function, and the third objective loss function. During the training process, the parameters of the initial cartoon character expression-driven network model can be continuously updated based on the changes in the function values ​​of the first objective loss function, the second objective loss function, and the third objective loss function until the function values ​​of the first objective loss function, the second objective loss function, and the third objective loss function meet the requirements, such as the weighted sum of the three reaching the minimum value and the change range being very small (basically unchanged), or reaching the preset maximum number of iterations (such as 10,000 times). At this point, the update of the model parameters is stopped, the training of the initial cartoon character expression-driven network model is completed, and the cartoon character expression-driven network model is generated.

[0142] The first objective function is used to constrain the cartoon character expression basis coefficient prediction module D in the cartoon character expression driven network model. coeff (.) Output the basic coefficients w∈R of the cartoon character's facial expressions. 52 This is compared with the basis coefficients of cartoon character expressions in the training image dataset (using w). GT ∈R52 (Indicated) to be as close as possible, the specific calculation formula is as follows:

[0143] l coeffs =||ww GT ||2 (4)

[0144] Among them, l coeffs This represents the first objective loss function.

[0145] The second objective function is used to constrain the rotation representation prediction module d in the cartoon character expression-driven network model. rotation The rotational feature representation of the output of (.) is r∈R 3*2 This makes it comparable to the rotation feature representation of samples in the training image dataset (using r). GT ∈R 3*2 (Indicated) to be as close as possible, the specific calculation formula is as follows:

[0146] l rotation =||rr GT ||2 (5)

[0147] Among them, l rotation This represents the second objective loss function.

[0148] Furthermore, the third objective function is used to simultaneously constrain the cartoon character expression basis coefficient prediction module D in the cartoon character expression driven network model. coeff (.) Output the basic coefficients w∈R of the cartoon character's facial expressions. 52 And rotation representation prediction module D rotation The rotational feature of the output of (.) is represented as r∈R 3*2 Let the two be compared with the basis coefficients of cartoon character expressions in the training image dataset (w) respectively. GT ∈R 52 ) and sample rotation feature representation (r GT ∈R 3×2 To make them as close as possible, specifically, first, based on coefficient weighting and combined with different facial shapes of the cartoon character, calculate the predicted mesh state (represented by b) and the labeled mesh state (represented by b) of the cartoon character's facial expression. GT The difference (represented by) is calculated using the following formula:

[0149]

[0150] l recon =||bb GT ||2 (8)

[0151] Among them, l recon This represents the third objective loss function.

[0152] An alternative implementation involves weighted summation of the first, second, and third objective functions to form a total objective loss function, defined as L. This L is used to pre-train the initial cartoon character expression-driven network model. During training, the model parameters are continuously updated based on changes in the value of this total objective loss function L until the value of L meets requirements, such as reaching a minimum and remaining relatively constant, indicating model convergence. At this point, updating the model parameters stops, completing the training of the initial cartoon character expression-driven network model, thus generating the cartoon character expression-driven network model. The specific formula for calculating L is as follows:

[0153] L=λ1*l coeffs +λ2*l rotation +λ3*l rec on (9)

[0154] Wherein, λ1, λ2, and λ3 represent preset coefficients, which can be set according to actual conditions and empirical values. This application does not limit the specific values ​​of the three, for example, λ1, λ2, and λ3 can be set to 0.3, 0.3, and 0.6 respectively.

[0155] Furthermore, regarding the optimizer, this application uses the Adam optimizer, with an initial learning rate set to lr = 1e-5 and eps = 1e-8. Training is performed for a total of 100 epochs, with the learning rate decreasing by 0.1 every 30 epochs. Other common optimizers can also be selected, including but not limited to SGD and RMSprop.

[0156] To facilitate understanding of the real-time expression-driving method for cartoon digital humans proposed in this application, this application also provides a schematic diagram of the overall implementation process of real-time expression-driving for cartoon digital humans, such as... Figure 8As shown, specifically: a video stream containing the facial information of the target user is captured by the front-facing camera of a mobile terminal device (such as a mobile phone) at a preset frequency (e.g., 30fps), and parsed frame by frame into a target image containing the facial information of the target user. This image is then input into the FaceMesh network to estimate 468 3D facial landmarks in real time, serving as the 3D key point information of the target user's face. This information is then input into a cartoon character expression-driven network model. This model is based on an MLP-Mixer architecture and uses a fully connected-convolutional hybrid neural network structure, primarily composed of fully connected neural network basic structural units. It includes a key information encoder, a cartoon character expression basis coefficient prediction module, and a rotation representation prediction module. The key information encoding module extracts features from the 3D key points of the target user's face to obtain the latent spatial feature representation of the target user's face information. Then, it is input into the cartoon character expression basis coefficient prediction module and the rotation representation prediction module to obtain a set of cartoon character expression basis coefficients and rotation feature representations (i.e., 6D rotation transformation representation) calculated based on the target user's facial key point information. Finally, the output of the cartoon character expression driving network model (i.e., the cartoon character expression basis coefficients and rotation feature representations (i.e., 6D rotation transformation representation)) is combined with the corresponding expression basis of the cartoon character (i.e., 52 predefined facial shapes) to render the expression animation image of the cartoon character and the target user's face RGB corresponding to each other.

[0157] As can be seen, this application adopts a neural network infrastructure that is friendly to mobile terminal devices, facilitating the quantitative deployment of various deployment tools. Furthermore, the cartoon character expression-driven network model constructed in this application achieves high-speed inference and edge-side implementation. Moreover, this model is trained using an automatically labeled data expansion training dataset and an incompletely supervised training strategy, ultimately enabling high real-time and high-precision expression driving for digital humans on mobile terminal devices.

[0158] Furthermore, this application also provides an electronic device (i.e., a terminal device). For details regarding the hardware structure and software framework of the electronic device, please refer to [link to relevant documentation]. Figure 2 and Figure 3 The corresponding explanation is as follows: The electronic device includes a memory and a processor. The memory stores a computer program, and the processor calls and executes the computer program to implement the real-time facial expression driving method for cartoon digital humans provided in the above description.

[0159] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by the processor of a terminal device, it is used to implement the real-time facial expression driving method for cartoon digital humans provided in the above description.

[0160] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for driving real-time facial expressions in a cartoon digital human, characterized in that, Applied to a terminal device equipped with a camera, the method includes: The camera acquires a target image containing the facial information of the target user at a preset frequency. The target image is input into a FaceMesh network to perform 3D reconstruction of the target user's face image, thereby obtaining the 3D key point information of the target user's face; The 3D key point information of the target user's face is input into a pre-constructed cartoon character expression-driven network model to predict the cartoon character expression basis coefficients and rotation feature representations. The cartoon character expression-driven network model is based on a fully connected neural network and a convolutional structure, and is obtained through incomplete supervised training using training images and a target loss function. Using the basic coefficients and rotation features of the cartoon character's facial expressions, combined with a preset BlendShape parameter model, the cartoon character's facial expression animation image corresponding to the target image is rendered; and the cartoon character's facial expression animation image is continuously output to form a real-time facial expression driven animation of the cartoon digital human corresponding to the target user. The target loss function includes a first target loss function, a second target loss function, and a third target loss function; the construction method of the cartoon character expression-driven network model is as follows: Obtain sample images containing facial information of sample users; Using preset coefficient priors and rotation priors as constraints, the sample images are used to construct input-output data pairs consisting of 3D facial keypoint information of sample users, basic coefficients of sample cartoon character expressions, and sample rotation feature representations, which serve as the training image dataset. The initial cartoon character expression-driven network model was trained using the training image dataset, the first objective loss function, the second objective loss function, and the third objective loss function to generate a cartoon character expression-driven network model. The first objective loss function is used to constrain the basis coefficients of the cartoon character expressions output by the cartoon character expression-driven network model, making them as close as possible to the basis coefficients of the sample cartoon character expressions in the training image dataset; the second objective loss function is used to constrain the rotation feature representation output by the cartoon character expression-driven network model, making it as close as possible to the rotation feature representation of the samples in the training image dataset; the third objective loss function is used to simultaneously constrain the basis coefficients of the cartoon character expressions and the rotation feature representation output by the cartoon character expression-driven network model, making them as close as possible to the basis coefficients of the sample cartoon character expressions and the rotation feature representation of the samples in the training image dataset, respectively.

2. The method according to claim 1, characterized in that, The step of inputting the target image into a FaceMesh network for 3D reconstruction of the target user's face image to obtain the target user's 3D facial key point information includes: The facial region of the target user is cropped from the target image to obtain the facial region image of the target user; The target user's facial region image is aligned by rotation and size transformation to obtain an aligned target user's facial region image; and the inverse rotation transformation matrix representation of the target user's facial region image is calculated. The aligned target user facial region image is input into the FaceMesh network for 3D reconstruction of the target user's facial image, resulting in 468 facial key point information of the target user. Using the rotation inverse transform matrix representation corresponding to the target user's facial region image and the target user's 468 facial key point information, the target user's 3D facial key point information is calculated.

3. The method according to claim 1, characterized in that, The cartoon character expression-driven network model includes a key information encoder, a cartoon character expression basis coefficient prediction module, and a rotation representation prediction module; the step of inputting the target user's 3D facial key point information into the pre-constructed cartoon character expression-driven network model to predict the cartoon character expression basis coefficients and rotation feature representations includes: The 3D key point information of the target user's face is input into the key information encoder of a pre-built cartoon character expression-driven network model for feature extraction, resulting in a latent spatial feature representation. The latent spatial feature representation is input into the cartoon character expression basis coefficient prediction module of the cartoon character expression driven network model and subjected to mean dimensionality reduction processing to obtain the cartoon character expression basis coefficients. The latent spatial feature representation is input into the rotation representation prediction module of the cartoon character expression-driven network model for size transformation and convolution processing to obtain the rotation feature representation.

4. The method according to claim 3, characterized in that, The key information encoder is an encoder built on a fully connected network structure; the cartoon character expression basis coefficient prediction module is a decoder built on a fully connected network structure; and the rotation representation prediction module is a decoder built on a convolutional structure.

5. The method according to claim 2, characterized in that, The process of rendering the cartoon character's facial expression animation image corresponding to the target image by utilizing the cartoon character's facial expression basis coefficients and rotation features, combined with a preset BlendShape parameter model, includes: The cartoon character's facial expression base coefficients are used to perform a weighted summation on a preset blendShape parameter model to predict the cartoon character's facial expression model corresponding to the target image. Using the rotation feature representation, the predicted cartoon character expression model corresponding to the target image is subjected to rotation transformation processing to obtain a cartoon character expression model mapped to a 2D plane, and then rendered to obtain a cartoon character expression animation. Based on the target user's facial region image, the cartoon character's facial expression animation is aligned with the target user's face to obtain the cartoon character's facial expression animation image corresponding to the target image.

6. The method according to claim 1, characterized in that, The continuous output of the cartoon character's facial expression animation images to form a real-time facial expression-driven animation of the cartoon digital human corresponding to the target user includes: By sequentially connecting the animated images of the cartoon characters' expressions, all the connected animated images of the cartoon characters' expressions will be continuously output to form a real-time facial expression-driven animation of the cartoon digital human corresponding to the target user.

7. The method according to any one of claims 1-6, characterized in that, The rotation feature is represented as a 6D rotation transformation.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor being used to invoke and execute the computer program to implement the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by an electronic device, implements the method of any one of claims 1-7.

Images