Training method and apparatus for face modeling model, modeling method and apparatus, electronic device, storage medium, and program product

Abstract

The present application relates to the field of artificial intelligence, and discloses a training method and apparatus for a face modeling model, a modeling method and apparatus, an electronic device, a computer readable storage medium, and a computer program product. The training method comprises: acquiring sample face images, wherein the sample face images comprise face images of a same object under different visual angles at a same moment; performing feature encoding on the sample face images by means of an encoder of a face modeling model to obtain face region latent codes corresponding to different face regions and face action latent codes corresponding to face actions; performing three-dimensional face reconstruction on the object on the basis of the face region latent codes and the face action latent codes by means of a neural radiance field of the face modeling model to obtain a three-dimensional face image of the object; and acquiring a three-dimensional reconstruction loss between the three-dimensional face image and the sample face images, and training the face modeling model on the basis of the three-dimensional reconstruction loss.

Description

Facial modeling model training method, modeling method, device, electronic device, storage medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of this application are based on and claim the priority of Chinese patent application with application number 202310136271.9 and application date February 9, 2023. The entire contents of the Chinese patent application are hereby introduced into the embodiments of this application as a reference. Technical Field

[0003] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a training method, modeling method, device, electronic device, computer-readable storage medium, and computer program product for a facial modeling model. Background Art

[0004] Generating drivable faces has a wide range of applications in computer vision. In these applications, reconstructing a face requires changing perspective and expression based on input signals, maintaining a natural speaking demeanor, and synchronizing with the driving signal.

[0005] In the related art, most of the facial reconstruction is done by audio-driven and text-driven methods. For the method of facial reconstruction using audio-driven methods, due to the large differences in different people's voices, the trained network may have different generation results, resulting in inaccurate driving, which leads to poor simulation effects when performing facial reconstruction. For the method of facial reconstruction based on text-driven methods, only two-dimensional facial images can be generated, and the simulation effect is poor for three-dimensional facial driving. Based on this, the simulation effects of facial reconstruction in the related art are all poor.

[0006] Summary of the Invention

[0007] The present invention provides a facial modeling model training method, modeling method, apparatus, electronic device, computer-readable storage medium, and computer program product, which achieve three-dimensional facial modeling and improve simulation realism through decoupling control of the face. The technical solution is as follows:

[0008] The present invention provides a method for training a facial modeling model, the method comprising:

[0009] Acquire sample facial images, where the sample facial images include facial images of the same subject at different viewing angles at the same time;

[0010] Performing feature encoding on the sample facial image by an encoder of the facial modeling model to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions;

[0011] Performing three-dimensional facial reconstruction on the object based on the facial region latent code and the facial action latent code using the neural radiation field of the facial modeling model to obtain a three-dimensional facial image of the object;

[0012] A three-dimensional reconstruction loss between the three-dimensional facial image and the sample facial image is obtained, and the facial modeling model is trained based on the three-dimensional reconstruction loss.

[0013] The present invention provides a method for three-dimensional facial modeling, comprising:

[0014] When receiving input text, determining text phonemes corresponding to the input text;

[0015] querying a target region latent code and a target action latent code corresponding to the text phoneme based on a phoneme latent code index, wherein the phoneme latent code index is used to indicate a correspondence between a phoneme and a latent code sequence, wherein the latent code sequence is obtained by feature encoding a facial image corresponding to the phoneme using an encoder of a facial modeling model;

[0016] Performing three-dimensional reconstruction based on the target region latent code and the target action latent code using the neural radiation field of the facial modeling model to obtain a facial action image corresponding to the text phoneme;

[0017] Based on each frame of facial action image, a three-dimensional facial action animation corresponding to the input text is generated.

[0018] The present invention provides a facial modeling model training device, comprising:

[0019] An image acquisition module is configured to acquire sample facial images, wherein the sample facial images include facial images of the same subject at different viewing angles at the same time;

[0020] a feature encoding module configured to perform feature encoding on the sample facial image through an encoder of the facial modeling model to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions;

[0021] a three-dimensional reconstruction module configured to perform three-dimensional facial reconstruction on the object based on the facial region latent code and the facial action latent code using the neural radiation field of the facial modeling model to obtain a three-dimensional facial image of the object;

[0022] The model training module is configured to obtain a three-dimensional reconstruction loss between the three-dimensional facial image and the sample facial image, and train the facial modeling model based on the three-dimensional reconstruction loss.

[0023] The present invention provides a three-dimensional facial modeling device, comprising:

[0024] a phoneme determination module configured to, upon receiving an input text, determine the text phoneme corresponding to the input text;

[0025] a latent code query module configured to query a target region latent code and a target action latent code corresponding to the text phoneme based on a phoneme latent code index, wherein the phoneme latent code index is used to indicate a correspondence between a phoneme and a latent code sequence, wherein the latent code sequence is obtained by feature encoding a facial image corresponding to the phoneme using an encoder of a facial modeling model;

[0026] a three-dimensional reconstruction module configured to perform three-dimensional reconstruction based on the target region latent code and the target action latent code using the neural radiation field of the facial modeling model to obtain a facial action image corresponding to the text phoneme;

[0027] The animation generation module is configured to generate a three-dimensional facial action animation corresponding to the input text based on each frame of facial action image.

[0028] An embodiment of the present application provides an electronic device, including:

[0029] a memory configured to store executable instructions;

[0030] The processor is configured to implement the facial modeling model training method described in the above aspects, or implement the three-dimensional facial modeling method described in the above aspects when executing the computer-executable instructions stored in the memory.

[0031] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the training method of the facial modeling model as described in the above aspects, or implement the three-dimensional facial modeling method as described in the above aspects.

[0032] An embodiment of the present application provides a computer program product, comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the facial modeling model training method provided in the above aspects, or implement the three-dimensional facial modeling method described in the above aspects.

[0033] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0034] In an embodiment of the present application, feature encoding is performed on sample facial images from multiple perspectives to obtain facial region latent codes and facial action latent codes corresponding to the sample facial images. These codes are used to characterize the regional features of different facial regions and facial action features. Subsequently, the facial region latent codes and facial action latent codes are used to perform three-dimensional reconstruction to obtain a three-dimensional facial image. Based on the differences between the three-dimensional facial image and the sample facial image, a facial modeling model capable of decoupling facial control is trained. When used for three-dimensional facial modeling, because the model can decouple and control different facial regions and facial actions, facial images combining different facial regions and different facial actions can be generated. During the three-dimensional facial modeling process, only the latent codes that need to be changed need to be adjusted, enabling long-sequence three-dimensional facial modeling and improving the authenticity of facial simulation during text-driven facial modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 is a schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application;

[0036] FIG2 shows a flow chart of a method for training a facial modeling model provided by an exemplary embodiment of the present application;

[0037] FIG3 shows a flow chart of a method for determining 3D reconstruction loss provided by another exemplary embodiment of the present application;

[0038] FIG4 shows a structural diagram of a motor nerve radiation field provided by an exemplary embodiment of the present application;

[0039] FIG5 shows a structural diagram of a regional neural radiation field provided by an exemplary embodiment of the present application;

[0040] FIG6 shows a structural diagram of a rendered neural radiation field provided by an exemplary embodiment of the present application;

[0041] FIG7 shows a flowchart of a method for training a facial modeling model provided by another exemplary embodiment of the present application;

[0042] FIG8 shows a schematic structural diagram of an encoder provided by an exemplary embodiment of the present application;

[0043] FIG9 shows a schematic structural diagram of a facial modeling model provided by an exemplary embodiment of the present application;

[0044] FIG10 shows a flow chart of a three-dimensional facial modeling method provided by an exemplary embodiment of the present application;

[0045] FIG11 shows a flowchart of generating a phoneme latent index according to an exemplary embodiment of the present application;

[0046] FIG12 shows a structural block diagram of a facial modeling model training device provided by an exemplary embodiment of the present application;

[0047] FIG13 shows a block diagram of a three-dimensional facial modeling apparatus provided by an exemplary embodiment of the present application;

[0048] FIG14 shows a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0050] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0051] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0052] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs graphic processing so that computer processing can become images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image segmentation, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The method involved in the embodiment of the present application is the application of computer vision technology in three-dimensional facial reconstruction.

[0053] Related technologies often use audio-driven facial recognition to achieve three-dimensional facial control. However, during the training process, voices vary significantly between individuals, potentially leading to varying training results. Furthermore, in actual use, recorded audio may contain pronunciation errors, necessitating re-recording, which affects facial control efficiency. Related technologies using text-driven facial recognition can only drive two-dimensional facial images, resulting in poor simulation results.

[0054] In an embodiment of the present application, a training method for a facial modeling model is proposed. This training method can be used to obtain a model for reconstructing a three-dimensional face. This model can achieve decoupled control of different facial attributes, for example, decoupled control of different facial regions and facial movements, facilitating the implementation of long-sequence three-dimensional facial modeling. After the model is trained, the encoder in the model can be used to extract the latent sequences of facial images corresponding to different phonemes. During application, the latent sequences corresponding to text phonemes can be queried and the neural radiation fields in the model can be used to perform long-sequence three-dimensional facial modeling, resulting in three-dimensional facial movement animation corresponding to the text. This allows for text-driven three-dimensional facial modeling and improves the applicability of facial-driven modeling.

[0055] The method provided in the embodiment of the present application can be applied to scenarios where text drives the face. For example, during a video conference, participants can input text through a program. When the computer device receives the input text, it can query the corresponding latent code sequence based on the text phonemes of the input text, thereby performing a three-dimensional reconstruction of the latent code sequence through the neural radiation field, obtaining a three-dimensional facial action animation, realizing simulated speech, and improving the simulation authenticity. Of course, it can also be applied to other scenarios that require driving the face, such as online teaching, virtual anchors, and virtual social scenarios, which are not limited in this embodiment.

[0056] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment includes a terminal 110 and a server 120. Data communication between the terminal 110 and the server 120 is performed via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0057] Terminal 110 is an electronic device with a text-driven facial recognition function. This electronic device can be a mobile terminal such as a smartphone, tablet computer, or laptop computer, or a desktop computer or projection computer, which is not limited in this embodiment of the present application. Furthermore, the text-driven facial recognition function can be provided in the terminal through, for example, conferencing programs, live broadcast programs, and educational applications, which is not limited in this embodiment of the present application.

[0058] Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In the embodiment of the present application, server 120 is a backend server that provides text-driven facial functions in terminal 110. It can generate three-dimensional facial action animations based on the text entered by the user and return them to terminal 110.

[0059] As shown in Figure 1, the terminal 110 can receive the input text entered by the user, and then transmit the input text to the server 120. The server 120 stores a phoneme latent code index 122. The server 120 determines the text phoneme 121 corresponding to the input text, and obtains the corresponding target latent code sequence 123 by querying the phoneme latent code index 122. Then, the target latent code sequence 123 is input into the neural radiation field 124 for three-dimensional reconstruction to obtain the facial action image 125 corresponding to each text phoneme, thereby generating a three-dimensional facial action animation based on the text reading image 125 and feeding it back to the terminal 110.

[0060] The phoneme latent index is established based on the encoding results of different facial images by the encoder in the facial modeling model. The encoder and neural radiance field of the facial modeling model are pre-trained using sample facial images. The training process can be performed by server 120. Subsequently, server 120 uses the trained encoder to establish the phoneme latent index and uses the neural radiance field to perform 3D reconstruction.

[0061] In another possible implementation, the 3D reconstruction process can also be performed by terminal 110. Server 120 sends the trained model to terminal 110, and terminal 110 performs 3D reconstruction locally without the need for server 120. Alternatively, the facial modeling model can be trained on terminal 110, and the 3D reconstruction process can be performed by terminal 110. This embodiment of the present application is not limited to this.

[0062] For the convenience of description, the following embodiments are described by taking the method executed by an electronic device as an example.

[0063] Please refer to Figure 2, which shows a flow chart of a facial modeling model training method provided by an exemplary embodiment of the present application. This embodiment uses the method applied to a computer device as an example to illustrate the method, which includes the following steps.

[0064] Step 201 : Acquire sample facial images, where the sample facial images include facial images of the same subject at different viewing angles at the same time.

[0065] In some embodiments, the electronic device may first obtain a set of sample images from multiple perspectives. In one possible implementation, a multi-perspective camera system may be used to capture facial images of the same subject from different perspectives, thereby obtaining facial images from different perspectives at the same moment. For example, a reader may be given a text in any language. While the reader is reading the text, the multi-perspective camera system may film the reader, obtaining a video of the reader reading the text from multiple perspectives. During the filming process, only the reader's face may be filmed, obtaining a multi-perspective facial video. The electronic device may then process the multi-perspective video to obtain facial images of the reader from different perspectives at the same moment. The facial images at the same moment constitute a set of sample facial images. The electronic device may obtain multiple sets of sample facial images through multi-perspective video processing.

[0066] Illustratively, the electronic device can obtain facial images of the reader from three perspectives, that is, a set of sample facial images includes facial images from the frontal perspective, the left side perspective, and the right side perspective to include comprehensive facial features.

[0067] Step 202 : Feature encoding is performed on the sample facial image by an encoder of the facial modeling model to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions.

[0068] In an embodiment of the present application, a facial modeling model is provided, wherein the model is composed of an encoder-neural radiation field structure. In one possible implementation, the encoder is used to encode facial images from multiple perspectives to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions. That is, the electronic device inputs a set of sample facial images into the encoder, and the encoder of the facial modeling model performs feature encoding on the sample facial images to obtain facial region latent codes and facial action latent codes corresponding to the facial images at the corresponding moment. In other words, the sample facial images are input into the encoder of the facial modeling model for feature encoding to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions.

[0069] The latent code is a feature vector used to represent facial features. The facial region latent code is used to represent the regional features of a local facial region, and the facial action latent code is used to represent facial action features, such as mouth movements, eye movements, and nose movements.

[0070] Different facial regions have different states. In some embodiments, the facial region can be divided into upper and lower regions, i.e., encoding the facial region latent code corresponding to the upper half of the face and the facial region latent code corresponding to the lower half of the face; or it can be divided into left and right regions, i.e., encoding the facial region latent code corresponding to the left half of the face and the facial region latent code corresponding to the right half of the face. This embodiment does not limit the facial division method. Similarly, there is no limit on the number of divided regions; for example, it can be divided into 2, 3, or 4 regions. This embodiment achieves decoupled control of the face through the regional latent codes of different facial regions and the action latent codes of facial movements.

[0071] Illustratively, the frontal face image, left side face image and right side face image of the reader at the same moment can be input into the encoder to obtain the facial region latent code and facial action latent code of the reader's face at that moment.

[0072] Step 203 : Perform three-dimensional facial reconstruction on the object based on the neural radiation field of the facial modeling model, the facial region latent code, and the facial action latent code to obtain a three-dimensional facial image of the object.

[0073] Neural Radiance Field (NeRF) is a model used for 3D reconstruction. It represents a 3D scene as a radiance field approximated by a neural network. The radiance field describes the color and volume density of each point in the scene and each viewing direction. In the embodiment of the present application, the Neural Radiance Field includes multiple Neural Radiance Fields to achieve decoupled control of different parts of the face.

[0074] In one possible embodiment, the neural radiation field may include a neural radiation field for extracting facial area features and a neural radiation field for extracting facial action features. Finally, three-dimensional facial reconstruction is performed through multiple neural radiation fields to obtain the pixel color and pixel density corresponding to each pixel. According to the pixel color and pixel density, a three-dimensional facial image can be rendered.

[0075] In actual implementation, the process of performing three-dimensional facial reconstruction on the object based on the facial region latent code and the facial action latent code through the neural radiation field of the facial modeling model to obtain the three-dimensional facial image of the object is that the facial region latent code and the facial action latent code are input into the neural radiation field of the facial modeling model to perform three-dimensional facial reconstruction to obtain the three-dimensional facial image.

[0076] Step 204 : Obtain a 3D reconstruction loss between the 3D facial image and the sample facial image, and train a facial modeling model based on the 3D reconstruction loss.

[0077] It should be noted that the facial modeling model is trained here based on the three-dimensional reconstruction loss, that is, the encoder and neural radiation field of the facial modeling model are trained based on the three-dimensional reconstruction loss between the three-dimensional facial image and the sample facial image.

[0078] After the three-dimensional facial image is rendered, the three-dimensional reconstruction loss can be determined based on the difference between the three-dimensional facial image and the sample facial image, and the three-dimensional reconstruction loss can be used to train the facial modeling model.

[0079] In one possible implementation, the model can be trained using gradient updates or backpropagation. When the 3D reconstruction loss function reaches convergence, training terminates, resulting in a facial model for 3D facial modeling. This model can achieve decoupled control of the face. During subsequent use, only the facial movement latent code can be modified when facial movements change; and only the facial region latent code corresponding to a local region can be modified when a local region changes, enabling 3D facial modeling over long sequences.

[0080] In an embodiment of the present application, feature encoding is performed on sample facial images from multiple viewpoints to obtain facial region latent codes and facial action latent codes corresponding to the sample facial images. These codes are used to characterize the regional features of different facial regions and facial action features. The facial region latent codes and facial action latent codes are then used to perform three-dimensional reconstruction to obtain a three-dimensional facial image. Based on the differences between the three-dimensional facial image and the sample facial image, a facial model capable of decoupling facial control is trained. When used for three-dimensional facial modeling, the model can decouple and control different facial regions and facial actions, allowing for the generation of facial images combining different facial regions with different facial actions. During the three-dimensional facial modeling process, only the latent codes that need to be changed need to be adjusted, enabling the modeling of long-sequence three-dimensional faces.

[0081] In some embodiments, the neural radiation field includes a regional neural radiation field, an action neural radiation field, and a rendering neural radiation field. Different neural radiation fields are used to achieve decoupled control of different faces and reconstruct a three-dimensional facial image. This will be described below using an exemplary embodiment.

[0082] Please refer to Figure 3, which shows a flow chart of a method for determining 3D reconstruction loss provided by another exemplary embodiment of the present application. This embodiment uses the method applied to an electronic device as an example to illustrate the method, which includes the following steps.

[0083] Step 301: Acquire sample facial images, where the sample facial images include facial images of the same subject at different viewing angles at the same time.

[0084] The implementation of this step can refer to the above-mentioned step 201 and will not be described in detail in this embodiment.

[0085] In step 302, the facial image of the sample is feature-encoded by an encoder of the facial modeling model to obtain an upper region latent code, a lower region latent code, and a facial action latent code. The upper region latent code is used to represent the facial features of the upper half of the face, and the lower region latent code is used to represent the facial features of the lower half of the face.

[0086] In some embodiments, the face is divided into upper and lower parts, that is, multi-view sample facial images are input into an encoder for feature encoding to obtain an upper region latent code corresponding to the upper half of the face, a lower region latent code corresponding to the lower half of the face, and a facial action latent code corresponding to the facial action.

[0087] In one possible implementation, the encoder may first perform feature encoding on sample facial images from multiple perspectives to obtain a facial state latent code used to characterize the overall state of the face, and then decouple the facial state latent code to obtain a facial region latent code and a facial action latent code.

[0088] Schematically, the image size of the sample facial images under three perspectives is 512×374, the latent code size of the facial latent code is 256, and the latent code size of the facial region latent code and the facial action latent code obtained after decoupling is 128.

[0089] Step 303: Input the facial action latent code into the action neural radiation field to obtain the action conversion matrix.

[0090] The neural radiation field based on latent code realizes decoupled control of various parts of the face through multiple neural radiation fields, among which the neural radiation field includes the action neural radiation field, which is used to realize decoupled control of facial movements.

[0091] The action neural radiation field can extract the features of the facial action latent code and obtain the action conversion matrix corresponding to the facial action. The action conversion matrix can be used to realize the transformation of facial action.

[0092] The action neural radiation field is composed of a multi-layer fully connected network. As shown in FIG4 , FIG4 shows a structural diagram of the action neural radiation field provided by an exemplary embodiment of the present application. The action neural radiation field includes a 4-layer fully connected network 401 with a network width of 128. The facial action latent code Z p By inputting the action neural radiation field, the action transformation matrix (R, t) can be obtained.

[0093] Step 304 : Based on the sample facial image, obtain the three-dimensional facial coordinates of the subject, and perform motion transformation on the three-dimensional facial coordinates based on the motion transformation matrix to obtain facial motion coordinates.

[0094] In actual implementation, after the sample facial image is determined, a three-dimensional coordinate system for the sample facial image is constructed, so that the three-dimensional facial coordinates of the object corresponding to the sample facial image are obtained based on the three-dimensional coordinate system.

[0095] After the action conversion matrix is ​​obtained, the three-dimensional coordinates of the face can be transformed by the action conversion matrix to obtain the facial action coordinates corresponding to the facial action.

[0096] The conversion method is as follows: x′=Rx+t Formula (1);

[0097] Among them, x represents the three-dimensional coordinate before transformation, and (R, t) is the transformation matrix.

[0098] After performing motion transformation on the facial three-dimensional coordinates (x, y, z), the facial motion coordinates (x′, y′, z′) can be obtained.

[0099] In another possible implementation, the coordinates (x, y, z) may be divided by the motion transformation matrix (R, t) to obtain the transformed facial motion coordinates.

[0100] Step 305 : input the facial action coordinates and the facial region latent code into the regional neural radiation field to obtain a facial region section. The feature dimension of the facial region section is higher than the feature dimension of the facial region latent code.

[0101] The neural radiation field also includes a regional neural radiation field, which is used to achieve decoupled control of different facial regions. The regional neural radiation field is used for feature mapping, mapping the latent code into a feature vector in a high-dimensional space. That is, a section is a mapping of the latent code in a high-dimensional space. In one possible implementation, a computer device inputs facial action coordinates and facial region latent codes into the regional neural radiation field, which can obtain a facial region section in a high-dimensional space. The feature dimension of the facial region latent code is higher than that of the facial region latent code, thereby learning more facial region features.

[0102] The facial region latent code includes the upper region latent code and the lower region latent code. Correspondingly, the regional neural radiation field may also include the upper neural radiation field for controlling the upper half of the face, and the lower neural radiation field for controlling the lower half of the face.

[0103] In some embodiments, the facial action coordinates and the upper region latent code are input into the upper neural radiation field to obtain the upper region section. f Input to the upper neural radiation field to obtain the upper area section w f .

[0104] In some embodiments, the facial action coordinates and the lower region latent code are input into the lower neural radiation field to obtain the lower region section. m Input to the lower nerve radiation field to obtain the lower area section w m .

[0105] In some embodiments, the regional neural radiation field also uses a multi-layer fully connected network. Schematically, as shown in FIG5 , FIG5 shows a structural diagram of a regional neural radiation field provided by an exemplary embodiment of the present application. The regional neural radiation field includes a 6-layer fully connected network 501 with a network width of 128, which transforms the facial action coordinates (x′, y′, z′) and the facial region latent code (z f / z m ) is encoded into the regional neural radiation field and the facial region section (w f / w m ).

[0106] After the upper region section and the lower region section are obtained, the facial region section is obtained based on the upper region section and the lower region section.

[0107] Step 306 : Input the facial action coordinates and the facial region section into the rendered neural radiation field to obtain the facial color and facial density.

[0108] The neural radiation field also includes a rendered neural radiation field, which is used for 3D facial reconstruction. Once the upper and lower region slices are obtained, the facial motion coordinates and the facial region slices (including the upper and lower region slices) are input into the rendered neural radiation field to obtain facial color and density.

[0109] It should be noted that during the 3D reconstruction process, the view direction and appearance code of the shooting must also be input into the fully connected network that renders the neural radiation field for 3D reconstruction to obtain the pixel color and pixel density of each pixel corresponding to the 3D face.

[0110] In some embodiments, the neural radiation field is also rendered using a multi-layer fully connected network. Schematically, as shown in FIG6 , FIG6 shows a structural diagram of a neural radiation field rendered according to an exemplary embodiment of the present application. The neural radiation field includes a 7-layer fully connected network 601, wherein the network width is 256, and the facial action coordinates (x′, y′, z′) and the facial region section (w f / w m) can be input into the rendered neural radiation field after position encoding, and the view direction (θ, Φ) and appearance code (ψ) can be input into the last layer of the fully connected network for three-dimensional reconstruction to obtain the facial RGB color c and facial density (mask) σ.

[0111] Step 307: Generate a three-dimensional facial image based on the facial color and facial density.

[0112] Step 308 : Determine pixel error loss based on the pixel difference between the 3D facial image and the sample facial image.

[0113] After 3D reconstruction, the difference between the sample facial image and the 3D facial image can be used to determine the 3D reconstruction loss. In some embodiments, the pixel error loss can be determined based on the pixel difference, where the pixel difference can be determined based on the RGB difference of each pixel in the sample facial image and the 3D facial image. In one possible implementation, the pixel error loss can be calculated by calculating the mean squared error (MSE) between pixels based on the pixel differences.

[0114] Step 309 : determining a mask error loss based on the mask difference between the 3D facial image and the sample facial image.

[0115] The mask error loss can also be determined based on the mask differences between the faces. In one possible implementation, the mask corresponding to the sample facial image can be obtained and the cross entropy loss between the mask and the mask of the 3D facial image can be calculated to obtain the mask error loss.

[0116] Step 310 : Determine a 3D reconstruction loss based on the pixel error loss and the mask error loss.

[0117] The 3D reconstruction loss is the sum of the pixel error loss and the mask error loss, as shown in the following formula:

[0118] in, is the sample facial image I and the 3D facial image The pixel error loss between is the facial mask M corresponding to the sample facial image h The mask error loss between the mask (opaque image) α corresponding to the 3D facial image, and λ is a balancing parameter.

[0119] In actual implementation, after determining the three-dimensional reconstruction loss, the facial modeling model is trained based on the three-dimensional reconstruction loss, that is, the encoder and the neural radiation field are trained based on the three-dimensional reconstruction loss. It can be that the encoder and the neural radiation field (including the action neural radiation field, the regional neural radiation field and the rendering neural radiation field) can be trained through the three-dimensional reconstruction loss to obtain the facial modeling model.

[0120] In this embodiment, facial movements and different regions are decoupled and controlled using action neural radiation fields and regional neural radiation fields, respectively. 3D facial images are then reconstructed by rendering the neural radiation fields. This decoupled control allows only the corresponding facial latent codes to be modified, improving the efficiency of 3D facial reconstruction and enabling efficient long-sequence 3D facial modeling.

[0121] To ensure accurate separation of different facial attributes, one possible implementation involves decoupling the upper and lower face by adding regularization. Specifically, the decoupling loss is determined by controlling variables and used to train the encoder to improve the accuracy of decoupling the latent codes corresponding to different attributes. This is illustrated below using an exemplary embodiment.

[0122] Please refer to Figure 7, which shows a flow chart of a method for training a facial modeling model provided by another exemplary embodiment of the present application. This embodiment uses the method applied to an electronic device as an example to illustrate the method, and the method includes the following steps.

[0123] Step 701: Acquire sample facial images, where the sample facial images include facial images of the same subject at different viewing angles at the same time.

[0124] The implementation of this step can refer to the above-mentioned step 201 and will not be described in detail in this embodiment.

[0125] Step 702 : feature encoding is performed on the sample facial image through an encoding network to obtain a facial state latent code. The facial state latent code is used to represent the overall facial state of the reader when reading the text at the corresponding moment of the sample facial image.

[0126] In actual implementation, the sample facial image is feature-encoded through the encoding network to obtain the facial state latent code, that is, the sample facial image is input into the encoding network for feature encoding to obtain the facial state latent code.

[0127] The encoder includes an encoding network and a decoupled neural radiation field, wherein the encoding network is used to encode features of sample facial images from multiple perspectives to obtain a facial state latent code that represents the overall facial state.

[0128] In some embodiments, the encoding network is composed of a convolutional layer and a cascade layer. The convolutional layer is used to perform two-dimensional convolution on the input sample facial image, and the cascade layer is used to cascade the intermediate representations corresponding to each image to obtain a facial state latent code.

[0129] Schematically, as shown in Figure 8, Figure 8 shows a structural diagram of an encoder provided by an exemplary embodiment of the present application. The sample input image includes image 1-image k, and image 1-image k with a size of 512×374 are respectively input into the convolution layer of the encoding network for convolution to obtain an intermediate vector 801 with a size of 4×256. The intermediate vector 801 is then reshaped and cascaded to obtain a facial state latent code Z 802 with a size of 256.

[0130] Step 703: Decouple the facial state latent code by decoupling the neural radiation field to obtain the upper region latent code, the lower region latent code, and the facial action latent code. Based on the upper region latent code and the lower region latent code, determine the facial region latent code; wherein the upper region latent code is used to represent the facial features of the upper half of the face, and the lower region latent code is used to represent the facial features of the lower half of the face.

[0131] In actual implementation, the facial state latent code is decoupled by decoupling the neural radiation field to obtain the upper region latent code, the lower region latent code and the facial action latent code. That is, the facial state latent code is input into the decoupled neural radiation field to decouple the upper region latent code, the lower region latent code and the facial action latent code.

[0132] The encoder also includes a decoupled neural radiation field, which is used to decouple the overall latent code to obtain facial region latent codes (including upper and lower region latent codes) and facial action latent codes.

[0133] In some embodiments, the decoupled neural radiation field is composed of a multi-layer fully connected network. As shown in FIG8 , it is composed of a three-layer fully connected network 803. After the decoupled neural radiation field is input, the facial state latent code Z can be decoupled into the upper region latent code z f , the lower area potential code z m and facial action latent code z p .

[0134] Step 704 : Perform three-dimensional facial reconstruction on the object based on the neural radiation field of the facial modeling model, the facial region latent code, and the facial action latent code to obtain a three-dimensional facial image of the object.

[0135] Step 705 , obtaining a 3D reconstruction loss between the 3D facial image and the sample facial image, and training a facial modeling model based on the 3D reconstruction loss.

[0136] The implementation of steps 704-705 may refer to steps 303-309 in the above embodiment, and will not be repeated in this embodiment.

[0137] Step 706: Determine a decoupling loss based on the upper region latent code, the lower region latent code, and the facial action latent code at different times. The decoupling loss is used to characterize the difference in facial decoupling at different times and is used to train the decoupling neural radiation field.

[0138] In one possible implementation, the training of the decoupled neural radiation field is achieved by controlling variables. When only the latent code of a local area is changed, the rendering results of other areas are not affected. Therefore, the decoupling loss can be determined by the rendering results corresponding to the latent code combination at different times. Among them, this method may include steps 706a-706e (not shown in the figure):

[0139] Step 706a: Obtain a first upper region latent code, a first lower region latent code, and a first facial action latent code corresponding to the sample facial image at the first moment, and obtain a second upper region latent code, a second lower region latent code, and a second facial action latent code corresponding to the sample facial image at the second moment.

[0140] First, the electronic device may encode the sample facial images at two different moments (the first moment t1 and the second moment t2) through an encoder, and obtain the first upper region latent code, the first lower region latent code, and the first facial action latent code (i.e., ), and the second upper region latent code, the second lower region latent code, and the second facial action latent code at the second moment (i.e. ).

[0141] Step 706b: determining an upper decoupling loss based on the first upper region latent code, the first facial action latent code, and the second lower region latent code.

[0142] Changing only the latent code of the lower region at the first moment does not affect the rendering of the upper half of the face. Therefore, in one possible implementation, the latent code of the lower region at the first moment can be changed, and a three-dimensional facial image can be rendered based on the changed latent code combination. The upper decoupling loss can be determined by comparing the difference between the three-dimensional facial image and the upper half of the sample facial image corresponding to the first moment. In one possible implementation, this method can include the following steps:

[0143] Step 1: Input the first upper region latent code, the first facial action latent code, and the second lower region latent code into the neural radiation field to obtain a first facial image.

[0144] The electronic device can change the first lower region latent code at the first moment into the second lower region latent code at the second moment to obtain a first latent code combination Then the first latent code combination is input into the neural radiation field for three-dimensional facial reconstruction to obtain the first facial image

[0145] Step 2: Determine the upper decoupling loss based on the difference between the upper half of the face in the first facial image and the upper half of the face in the sample facial image at the first moment.

[0146] Since only the latent code of the lower half of the face is changed, the rendered upper half face image should be the same as the upper half face image in the sample face image, and the upper decoupling loss can be determined by the difference in the upper half face image.

[0147] In one possible implementation, the upper decoupling loss is as follows:

[0148] Among them, M f Mask the upper face. is a first facial image rendered according to the first latent code combination, is the sample facial image corresponding to the first moment, Refers to the mean square error loss between the two.

[0149] Step 706c: determining a lower decoupling loss based on the first lower region latent code, the first facial action latent code, and the second upper region latent code.

[0150] Similarly, changing only the upper region latent code at the first moment does not affect the rendering of the lower half of the face. Therefore, in one possible implementation, the upper region latent code at the first moment can be changed, and a three-dimensional facial image can be rendered based on the changed latent code combination. The lower decoupling loss can be determined by comparing the difference between the three-dimensional facial image and the lower half of the sample facial image corresponding to the first moment. In one possible implementation, this method can include the following steps:

[0151] Step 1: Input the first lower region latent code, the first facial action latent code, and the second upper region latent code into the neural radiation field to obtain a second facial image.

[0152] The electronic device can change the first upper region latent code at the first moment into the second upper region latent code at the second moment to obtain a second latent code combination. Then the second latent code combination is input into the neural radiation field for three-dimensional facial reconstruction to obtain the second facial image

[0153] Step 2: Determine the lower decoupling loss based on the difference between the lower half of the face in the second facial image and the lower half of the face in the sample facial image at the first moment.

[0154] Since only the latent code of the upper half of the face is changed, the rendered lower half face image should be the same as the lower half face image in the sample facial image corresponding to the first moment, and the lower decoupling loss can be determined by the difference in the lower half face image.

[0155] In one possible implementation, the lower decoupling loss is as follows:

[0156] Among them, M m Mask the lower half of the face, is a second facial image rendered according to the second latent code combination, is the sample facial image corresponding to the first moment, Refers to the mean square error loss between the two.

[0157] Step 706d: determining a motion decoupling loss based on the second facial motion latent code, the first upper region latent code, and the first lower region latent code.

[0158] In order to improve the accuracy of decoupling facial movements, a motion decoupling loss is introduced. When the facial movement is changed, the mask of the rendered image is not affected. Therefore, by changing the facial movement latent code at the first moment, a three-dimensional facial image can be rendered based on the changed latent code combination. The motion decoupling loss is determined by the difference between the three-dimensional facial image and the sample facial image mask corresponding to the first moment. This method may include the following steps:

[0159] Step 1: Input the second facial action latent code, the first upper region latent code, and the first lower region latent code into the neural radiation field to obtain a third facial image.

[0160] In one possible implementation, the first facial action latent code at the first moment can be changed to the second facial action latent code at the second moment to obtain a third latent code combination: Afterwards, the third latent code combination can be input into the neural radiation field for three-dimensional reconstruction to obtain a third facial image.

[0161] Step 2: Determine the motion decoupling loss based on the difference between the mask of the third facial image and the mask of the sample facial image at the first moment.

[0162] When the facial motion changes, the facial projection will not change, that is, the facial mask corresponding to the third facial image should be the same as the facial mask corresponding to the sample facial image at the first moment. The motion decoupling loss can be determined based on the mask difference between the two.

[0163] Among them, the action decoupling loss is as follows:

[0164] in, That is, the facial mask corresponding to the third facial image, M h (p) is the facial mask corresponding to the sample facial image at the first moment.

[0165] It should be noted that this embodiment does not limit the execution sequence of the above-mentioned determination of the upper decoupling loss, the lower decoupling loss and the action decoupling loss (i.e., steps 706b-706d). The three can be executed simultaneously or one after another. This embodiment only provides an example for the implementation method and does not limit the order.

[0166] Step 706e: Determine the decoupling loss based on the upper decoupling loss, the lower decoupling loss, and the motion decoupling loss.

[0167] After determining the upper decoupling loss, lower decoupling loss, and action decoupling loss, the sum of the three can be determined as the decoupling loss, which is used to train the decoupled neural radiation field.

[0168] Step 707: Train the facial modeling model based on the decoupling loss and the 3D reconstruction loss.

[0169] After determining the decoupling loss, the process of training the facial modeling model based on the three-dimensional reconstruction loss may be to train the facial modeling model based on the decoupling loss and the three-dimensional reconstruction loss.

[0170] In the above embodiment, the encoder is also trained using the 3D reconstruction loss, including the training of the decoupled neural field. Therefore, the total loss corresponding to the decoupled neural radiation field is expressed as follows:

[0171] in, is the 3D reconstruction loss, is the upper decoupling loss, is the lower decoupling loss, is the action decoupling loss.

[0172] The decoupled neural radiation field can be trained by the total loss to improve the accuracy of the decoupling of the decoupled neural radiation field.

[0173] In one possible implementation, the model structure of the overall facial modeling model is shown in FIG9 , which shows a schematic diagram of the structure of the facial modeling model provided by an exemplary embodiment of the present application, wherein the model includes an encoder 91 and a neural radiation field 92. The encoder 91 includes an encoding network 910 and a decoupled neural radiation field 911. When a sample facial image under three viewing angles is input, the encoding network 910 can encode the facial state latent code. Afterwards, the decoupled neural radiation field 911 can decode the facial state latent code to obtain the upper region latent code z f , the lower area potential code z m and facial action latent code zp .

[0174] After the encoding is completed, the upper area latent code z f , the lower area potential code z m and facial action latent code z p Input neural radiation field 92, wherein neural radiation field 92 includes action neural radiation field 921, regional neural radiation field 922 and rendering neural radiation field 923. Action neural radiation field 921 is used to generate the facial action latent code z p Generate the action transformation matrix (R, t), then change the facial three-dimensional coordinate x to obtain the facial action coordinate x′, and then the upper area latent code z f , the lower area potential code z m The facial action coordinate x′ is input into the regional neural radiation field 922, and the upper regional section w is generated through the regional neural radiation field 922. f And the lower area section w m , and then cut the upper area w f , lower area section w m And the facial action coordinates x′ are input into the rendered neural radiation field to obtain the facial color c and facial density σ. When generating the facial color c, since the color is affected by the gaze direction and ambient lighting, the gaze direction and appearance code d(θ, Φ, ψ) need to be input into the rendered neural radiation field to complete the three-dimensional facial reconstruction.

[0175] In this embodiment, by combining the latent codes at different times, the decoupling loss is determined in a controlled variable manner, and the decoupling neural radiation field is trained through the decoupling loss, which can improve the decoupling accuracy and thus help improve the accuracy of the decoupling control.

[0176] In the above embodiment, a facial modeling model can be obtained through training, and the trained facial modeling model can be used in the process of text-driven facial expression, which will be described below with an exemplary embodiment.

[0177] Please refer to Figure 10, which shows a flow chart of a three-dimensional face modeling method provided by an exemplary embodiment of the present application. This embodiment uses the method applied to an electronic device as an example to illustrate the method, and the method includes the following steps.

[0178] Step 1001: When an input text is received, determine the text phonemes corresponding to the input text.

[0179] In a possible implementation, a text-driven facial program is running in the electronic device. When receiving input text, the electronic device can determine the text phonemes corresponding to the input text and obtain a text phoneme sequence corresponding to the input text.

[0180] Step 1002: Based on the phoneme latent index, the target area latent code and the target action latent code corresponding to the text phoneme are queried. The phoneme latent index is used to indicate the correspondence between the phoneme and the latent code sequence. The latent code sequence is obtained by feature encoding the facial image corresponding to the phoneme based on the encoder of the facial modeling model.

[0181] After the facial model is obtained through the above training, the encoder in the model can encode the facial image to obtain different latent codes. In one possible implementation, the electronic device can use the encoder to encode the facial image corresponding to different phonemes to obtain latent code sequences corresponding to the different phonemes. The latent code sequence corresponding to each phoneme includes a facial region latent code and a facial action latent code. Based on the encoding results of the facial image by the encoder, a phoneme latent code index can be established to indicate the correspondence between phonemes and latent code sequences.

[0182] In one possible implementation, the electronic device can query the latent code sequence corresponding to each phoneme to obtain the target region latent code and target action latent code corresponding to each phoneme. For example, when the text contains the word "Pu," it also contains the phonemes "p" and "u." The electronic device can query the facial region latent code and facial action latent code corresponding to "p," and the facial region latent code and facial action latent code corresponding to "u," respectively.

[0183] Alternatively, in another possible implementation, a phoneme sequence may be determined first, and then the corresponding latent code sequence may be searched based on the phoneme sequence, that is, the latent code sequence corresponding to "pu" may be searched. In some embodiments, a longest identical subsequence search method may be used to search for the latent code corresponding to the phoneme.

[0184] Among them, the facial image corresponding to the phoneme can be a pre-collected image. For different pronunciation methods of the phoneme, facial images corresponding to the pronunciation methods can be collected respectively. Indicatively, for the phonemes of the same text, facial images corresponding to Mandarin pronunciation, facial images corresponding to dialect pronunciation, and facial images corresponding to singing pronunciation can be collected respectively, so as to encode the latent code sequence corresponding to the phoneme in Mandarin pronunciation (that is, to establish the phoneme latent code index corresponding to Mandarin), the latent code sequence corresponding to the phoneme in dialect pronunciation (the phoneme latent code index corresponding to the dialect), and the latent code sequence corresponding to the phoneme when singing (the phoneme latent code index corresponding to singing), thereby providing users with different modes of facial driving.

[0185] When searching for a latent code sequence corresponding to a phoneme according to a phoneme latent code index, the corresponding phoneme latent code index may be determined according to the pronunciation mode selected by the user, thereby finding the corresponding latent code sequence.

[0186] Step 1003 , three-dimensional reconstruction is performed based on the neural radiation field of the facial modeling model and the target region latent code and the target action latent code to obtain a facial action image corresponding to the text phoneme.

[0187] In actual implementation, the neural radiation field of the facial modeling model is used to perform three-dimensional reconstruction based on the target area latent code and the target action latent code to obtain the facial action image corresponding to the text phoneme. That is, the target area latent code and the target action latent code are input into the neural radiation field of the facial modeling model for three-dimensional reconstruction to obtain the facial action image corresponding to the text phoneme.

[0188] In determining the latent code sequence corresponding to the text, the latent code sequence can be input into the neural radiation field for three-dimensional reconstruction. In one possible implementation, the target action latent code can be input into the action neural radiation field to obtain an action transformation matrix. The pre-set three-dimensional facial coordinates are transformed using the action transformation matrix to obtain facial action coordinates after the action transformation. The facial action coordinates and the upper region latent code are then input into the upper neural radiation field to obtain an upper region section. The facial action coordinates and the lower region latent code are then input into the lower neural radiation field to obtain a lower region section. Finally, the upper region section, the lower region section, and the facial action coordinates are input into the rendering neural radiation field to render a facial action image corresponding to the text phonemes.

[0189] The text phonemes corresponding to the input text contain multiple phonemes. In one possible implementation, the latent code sequence corresponding to each text phoneme is input into the neural radiation field to obtain the facial image corresponding to each text phoneme. When querying the latent code sequence corresponding to the text phoneme, there may be cases where the latent codes corresponding to adjacent phonemes are partially the same. For example, the upper region latent codes corresponding to two adjacent phonemes are the same. In this case, when reconstructing the facial action image corresponding to the second phoneme, only the changed latent code (such as the lower region latent code and the facial action latent code) needs to be input into the neural radiation field for 3D reconstruction, thereby achieving 3D facial modeling of a long sequence.

[0190] Step 1004 : Generate a three-dimensional facial action animation corresponding to the input text based on each frame of facial action image.

[0191] After generating multiple frames of facial action images, the electronic device combines them to create a 3D facial animation corresponding to the input text, ensuring the authenticity of the 3D facial pronunciation simulation. Specifically, when the electronic device receives input text, it generates a 3D facial animation corresponding to the input text. As the text changes, the animation changes accordingly, achieving a text-driven 3D facial animation and enhancing the authenticity of the simulation.

[0192] In this embodiment, an encoder performs feature encoding on facial images corresponding to different phonemes, thereby obtaining the correspondence between different phonemes and latent code sequences and storing the resulting phoneme latent code index. When the electronic device receives text input from a user, it can query the corresponding latent code sequence based on the phoneme latent code index. By inputting the neural radiation field, a 3D facial animation corresponding to the input text can be generated, enabling text-driven 3D facial animation. Users can change the 3D facial animation by changing the input text, improving the applicability of facial driving. Furthermore, because the neural radiation field allows for decoupled control of the face, when only a partial area needs to be changed, only a portion of the latent code needs to be adjusted, facilitating 3D facial modeling over long sequences.

[0193] Among them, the phoneme latent index is determined based on the encoding results of different facial images by the encoder in the facial modeling model. Different phoneme latent indexes can be established for different pronunciation methods. Through different phoneme latent indexes, facial movements corresponding to different pronunciation methods can be simulated to generate corresponding three-dimensional facial movement animations. For example, three-dimensional facial movement animations corresponding to simulated Mandarin, dialects or singing can be obtained to enrich the diversity of text-driven faces. The following is an example of the process of establishing a phoneme latent index in the form of reading text. Please refer to Figure 11, which shows a flowchart of the generation of a phoneme latent index provided by an exemplary embodiment of the present application. The process includes the following steps:

[0194] Step 1101 : Acquire a text reading image, where the text reading image includes facial images of a reader reading a text captured at different viewing angles at the same moment.

[0195] In one possible implementation, a multi-view camera acquisition system can be used to capture facial images of a reader reading different texts, and during this process, audio and video recording can be performed simultaneously to obtain a reading video and audio. The reading video includes the reader's facial image.

[0196] The audio and text are aligned to determine the timeline corresponding to each phoneme. Based on the timeline corresponding to each phoneme, the corresponding text reading image can be determined. Each phoneme's corresponding text reading image includes facial images captured from different viewing angles.

[0197] Step 1102 : Based on the encoder, feature encoding is performed on the text reading image to obtain a facial latent code sequence corresponding to the text reading image. The facial latent code sequence includes a region latent code and an action latent code.

[0198] In actual implementation, based on the encoder, the text reading image is feature encoded to obtain the facial latent code sequence corresponding to the text reading image. The facial latent code sequence includes regional latent codes and action latent codes. That is, the text reading image is input into the encoder for feature encoding to obtain the facial latent code sequence corresponding to the text reading image.

[0199] Electronic devices can use encoders to perform feature encoding on the text reading images corresponding to each phoneme. In this process, the facial state latent code used to represent the overall state is first obtained through encoding network encoding, and then the facial region latent code and facial action latent code are obtained through decoupling of the neural radiation field, thereby obtaining the latent code sequence corresponding to each phoneme.

[0200] Step 1103 : associate and store the facial latent code sequence with the phonemes corresponding to the text reading image to obtain a phoneme latent code index.

[0201] The electronic device can associate and store the facial latent code sequence with the corresponding spoken phonemes, which can be used later in the process of driving the face based on the text.

[0202] Please refer to Figure 12, which shows a structural block diagram of a facial model training device provided by an exemplary embodiment of the present application. The device includes:

[0203] The image acquisition module 1201 is configured to acquire sample facial images, wherein the sample facial images include facial images of the same subject at different viewing angles at the same time;

[0204] a feature encoding module 1202 configured to perform feature encoding on the sample facial image through an encoder of the facial modeling model to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions;

[0205] A three-dimensional reconstruction module 1203 is configured to perform three-dimensional facial reconstruction on the subject based on the facial region latent code and the facial action latent code using the neural radiation field of the facial modeling model to obtain a three-dimensional facial image of the subject;

[0206] The model training module 1204 is configured to obtain a 3D reconstruction loss between the 3D facial image and the sample facial image, and train the facial modeling model based on the 3D reconstruction loss.

[0207] In some embodiments, the neural radiation field includes a regional neural radiation field, an action neural radiation field, and a rendering neural radiation field.

[0208] The three-dimensional reconstruction module 1203 is further configured to:

[0209] Inputting the facial action latent code into the action neural radiation field to obtain an action conversion matrix;

[0210] Based on the sample facial image, obtaining the three-dimensional facial coordinates of the subject, and performing motion transformation on the three-dimensional facial coordinates based on the motion transformation matrix to obtain facial motion coordinates;

[0211] Inputting the facial action coordinates and the facial region latent code into the regional neural radiation field to obtain a facial region section, wherein the feature dimension of the facial region section is higher than the feature dimension of the facial region latent code;

[0212] Inputting the facial action coordinates and the facial region section into the rendered neural radiation field to obtain facial color and facial density;

[0213] The three-dimensional facial image is generated based on the facial color and the facial density.

[0214] The feature encoding module 1202 is further configured to:

[0215] Performing feature encoding on the sample facial image by an encoder of the facial modeling model to obtain an upper region latent code, a lower region latent code, and the facial action latent code, wherein the upper region latent code is used to represent facial features of the upper half of the face, and the lower region latent code is used to represent facial features of the lower half of the face;

[0216] The three-dimensional reconstruction module 1203 is further configured to:

[0217] Inputting the facial action coordinates and the upper region latent code into the upper neural radiation field to obtain an upper region section;

[0218] Inputting the facial action coordinates and the lower region latent code into the lower nerve radiation field to obtain a lower region section;

[0219] The facial region section is obtained based on the upper region section and the lower region section.

[0220] The model training module 1204 is further configured to:

[0221] determining a pixel error loss based on pixel differences between the three-dimensional facial image and the sample facial image;

[0222] determining a mask error loss based on a mask difference between the three-dimensional facial image and the sample facial image;

[0223] The 3D reconstruction loss is determined based on the pixel error loss and the mask error loss.

[0224] In some embodiments, the encoder comprises an encoding network and a decoupled neural radiation field.

[0225] The feature encoding module 1202 is further configured to:

[0226] Performing feature encoding on the sample facial image through the encoding network to obtain a facial state latent code, wherein the facial state latent code is used to represent the overall facial state of the subject corresponding to the sample facial image when reading a text;

[0227] Decoupling the facial state latent code through the decoupled neural radiation field to obtain an upper region latent code, a lower region latent code, and the facial action latent code, and determining the facial region latent code based on the upper region latent code and the lower region latent code;

[0228] The upper region latent code is used to characterize facial features of the upper half of the face, and the lower region latent code is used to characterize facial features of the lower half of the face.

[0229] In some embodiments, the apparatus further comprises:

[0230] a loss determination module configured to determine a decoupling loss based on the upper region latent code, the lower region latent code, and the facial action latent code at different moments, wherein the decoupling loss is used to characterize the difference in facial decoupling at different moments, and is used to train the decoupled neural radiation field;

[0231] The model training module 1204 is further configured to train the facial modeling model based on the decoupling loss and the three-dimensional reconstruction loss.

[0232] In some embodiments, the loss determination module is further configured to:

[0233] Obtaining a first upper region latent code, a first lower region latent code, and a first facial action latent code corresponding to the sample facial image at a first moment, and obtaining a second upper region latent code, a second lower region latent code, and a second facial action latent code corresponding to the sample facial image at a second moment;

[0234] determining an upper decoupling loss based on the first upper region latent code, the first facial action latent code, and the second lower region latent code;

[0235] determining a lower decoupling loss based on the first lower region latent code, the first facial action latent code, and the second upper region latent code;

[0236] determining a motion decoupling loss based on the second facial motion latent code, the first upper region latent code, and the first lower region latent code;

[0237] The decoupling loss is determined based on the upper decoupling loss, the lower decoupling loss, and the motion decoupling loss.

[0238] In some embodiments, the loss determination module is further configured to:

[0239] Inputting the first upper region latent code, the first facial action latent code, and the second lower region latent code into the neural radiation field to obtain a first facial image;

[0240] determining the upper decoupling loss based on a difference between an upper half of the face in the first facial image and an upper half of the face in the sample facial image at the first moment;

[0241] In some embodiments, the loss determination module is further configured to:

[0242] Inputting the first lower region latent code, the first facial action latent code, and the second upper region latent code into the neural radiation field to obtain a second facial image;

[0243] The lower decoupling loss is determined based on a difference between a lower half of the face in the second facial image and a lower half of the face in the sample facial image at the first moment.

[0244] In some embodiments, the loss determination module is further configured to:

[0245] inputting the second facial action latent code, the first upper region latent code, and the first lower region latent code into the neural radiation field to obtain a third facial image;

[0246] The motion decoupling loss is determined based on a difference between the mask of the third facial image and the mask of the sample facial image at the first moment.

[0247] In an embodiment of the present application, feature encoding is performed on sample facial images from multiple viewpoints to obtain facial region latent codes and facial action latent codes corresponding to the sample facial images. These codes are used to characterize the regional features of different facial regions and facial action features. The facial region latent codes and facial action latent codes are then used to perform three-dimensional reconstruction to obtain a three-dimensional facial image. Based on the differences between the three-dimensional facial image and the sample facial image, a facial model capable of decoupling facial control is trained. When used for three-dimensional facial modeling, the model can decouple and control different facial regions and facial actions, allowing for the generation of facial images combining different facial regions with different facial actions. During the three-dimensional facial modeling process, only the latent codes that need to be changed need to be adjusted, enabling the modeling of long-sequence three-dimensional faces.

[0248] Please refer to Figure 13, which shows a block diagram of a three-dimensional facial modeling device provided by an exemplary embodiment of the present application. The device includes:

[0249] The phoneme determination module 1301 is configured to determine the text phonemes corresponding to the input text when receiving the input text;

[0250] a latent code query module 1302 configured to query a target region latent code and a target action latent code corresponding to the text phoneme based on a phoneme latent code index, wherein the phoneme latent code index is configured to indicate a correspondence between a phoneme and a latent code sequence, wherein the latent code sequence is obtained by feature encoding a facial image corresponding to the phoneme using an encoder of a facial modeling model;

[0251] A three-dimensional reconstruction module 1303 is configured to perform three-dimensional reconstruction based on the target region latent code and the target action latent code using the neural radiation field of the facial modeling model to obtain a facial action image corresponding to the text phoneme;

[0252] The animation generation module 1304 is configured to generate a three-dimensional facial action animation corresponding to the input text based on each frame of facial action image.

[0253] In some embodiments, the apparatus further comprises:

[0254] An image acquisition module is configured to acquire a text reading image, wherein the text reading image includes facial images of a reader reading the text acquired at different viewing angles at the same time;

[0255] a feature encoding module configured to perform feature encoding on the text reading image based on the encoder to obtain a facial latent code sequence corresponding to the text reading image, wherein the facial latent code sequence includes a region latent code and an action latent code;

[0256] The storage module is configured to associate and store the facial latent code sequence with the reading phonemes corresponding to the text reading image to obtain the phoneme latent code index.

[0257] In this embodiment, an encoder performs feature encoding on facial images corresponding to different phonemes, thereby obtaining the correspondence between different phonemes and latent code sequences and storing the resulting phoneme latent code index. When the electronic device receives text input from a user, it can query the corresponding latent code sequence based on the phoneme latent code index. By inputting the neural radiation field, a 3D facial animation corresponding to the input text can be generated, enabling text-driven 3D facial animation. Users can change the 3D facial animation by changing the input text, improving the applicability of facial driving. Furthermore, because the neural radiation field allows for decoupled control of the face, when only a partial area needs to be changed, only a portion of the latent code needs to be adjusted, facilitating 3D facial modeling over long sequences.

[0258] Please refer to Figure 14, which shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device can be implemented as a terminal or server in the above-mentioned embodiment. In particular, the electronic device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory 1402 and a read-only memory 1403, and a system bus 1405 connecting the system memory 1404 and the central processing unit 1401. The electronic device 1400 also includes a basic input / output system (I / O system) 1406 that helps transmit information between various components in the computer, and a large-capacity storage device 1407 configured to store an operating system 1413, application programs 1414, and other program modules 1415.

[0259] In some embodiments, the basic input / output system 1406 includes a display 1408 configured to display information and an input device 1409, such as a mouse or keyboard, configured to input information to a user. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include an input / output controller 1410 configured to receive and process input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other type of output device.

[0260] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the electronic device 1400. In other words, the mass storage device 1407 may include a computer-readable medium (not shown) such as a hard disk or drive.

[0261] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1404 and mass storage device 1407 can be collectively referred to as memory.

[0262] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1401. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1401 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0263] According to various embodiments of the present application, the electronic device 1400 may also be connected to a remote computer on a network via a network such as the Internet. That is, the electronic device 1400 may be connected to a network 1412 via a network interface unit 1411 connected to the system bus 1405, or the network interface unit 1411 may be used to connect to other types of networks or remote computer systems (not shown).

[0264] The memory also includes one or more programs, which are stored in the memory and include steps executed by the electronic device in the method provided in the embodiment of the present application.

[0265] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the facial modeling model training method described in any of the above embodiments, or to implement the three-dimensional facial modeling method described in any of the above embodiments.

[0266] Embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the facial modeling model training method provided in the above aspects, or implement the three-dimensional facial modeling method described in any of the above embodiments.

[0267] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be performed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, which can be the computer-readable storage medium included in the memory in the above embodiments, or a separate computer-readable storage medium not incorporated into the terminal. The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the facial modeling model training method described in any of the above method embodiments, or to implement the three-dimensional facial modeling method described in any of the above embodiments.

[0268] Optionally, the computer-readable storage medium may include: ROM, RAM, solid-state drives (SSDs), or optical disks. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments.

[0269] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0270] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. And the "first", "second", etc. mentioned in this article are used to distinguish similar objects, and are not used to limit a specific order or sequence. In addition, the step numbers described in this article only exemplify a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0271] The above description is only an example of some embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for training a facial modeling model, the method being performed by an electronic device, the method comprising: Acquire sample facial images, wherein the sample facial images include facial images of the same object at different viewing angles at the same time; Performing feature encoding on the sample facial image through the encoder of the facial modeling model to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions; Performing three-dimensional facial reconstruction on the object based on the facial region latent code and the facial action latent code through the neural radiation field of the facial modeling model to obtain a three-dimensional facial image of the object; A three-dimensional reconstruction loss between the three-dimensional facial image and the sample facial image is obtained, and the facial modeling model is trained based on the three-dimensional reconstruction loss.

2. The method according to claim 1, wherein: The neural radiation field includes a regional neural radiation field, an action neural radiation field and a rendering neural radiation field; The method of performing three-dimensional facial reconstruction on the object through the neural radiation field of the facial modeling model based on the facial region latent code and the facial action latent code to obtain a three-dimensional facial image of the object includes: Inputting the facial action latent code into the action neural radiation field to obtain an action conversion matrix; Based on the sample facial image, acquiring the three-dimensional facial coordinates of the object, and performing motion transformation on the three-dimensional facial coordinates based on the motion transformation matrix to obtain facial motion coordinates; Inputting the facial action coordinates and the facial region latent code into the regional neural radiation field to obtain a facial region section, wherein the feature dimension of the facial region section is higher than the feature dimension of the facial region latent code; Inputting the facial action coordinates and the facial region section into the rendered neural radiation field to obtain facial color and facial density; The three-dimensional facial image is generated based on the facial color and the facial density.

3. The method according to claim 2, wherein: The encoder of the facial modeling model performs feature encoding on the sample facial image to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions, including: Performing feature encoding on the sample facial image through the encoder of the facial modeling model to obtain an upper region latent code, a lower region latent code, and the facial action latent code, wherein the upper region latent code is used to characterize facial features of the upper half of the face, and the lower region latent code is used to characterize facial features of the lower half of the face; The step of inputting the facial action coordinates and the facial region latent code into the regional neural radiation field to obtain a facial region section includes: Inputting the facial motion coordinates and the upper region latent code into the upper nerve radiation field to obtain an upper region section; Inputting the facial motion coordinates and the lower region latent code into the lower nerve radiation field to obtain a lower region section; The facial region section is obtained based on the upper region section and the lower region section.

4. The method according to any one of claims 1 to 3, wherein: The obtaining of the three-dimensional reconstruction loss between the three-dimensional facial image and the sample facial image comprises: determining a pixel error loss based on a pixel difference between the three-dimensional facial image and the sample facial image; determining a mask error loss based on a mask difference between the three-dimensional facial image and the sample facial image; The three-dimensional reconstruction loss is determined based on the pixel error loss and the mask error loss.

5. The method according to any one of claims 1 to 3, wherein: The encoder comprises an encoding network and a decoupled neural radiation field; The encoder of the facial modeling model performs feature encoding on the sample facial image to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions, including: By means of the encoding network, feature encoding is performed on the sample facial image to obtain a facial state latent code, wherein the facial state latent code is used to characterize the overall facial state of the object corresponding to the sample facial image when reading a text; Decoupling the facial state latent code through the decoupled neural radiation field to obtain an upper region latent code, a lower region latent code and the facial action latent code, and determining the facial region latent code based on the upper region latent code and the lower region latent code; The upper region latent code is used to characterize the facial features of the upper half of the face, and the lower region latent code is used to characterize the facial features of the lower half of the face.

6. The method according to claim 5, wherein: The method further comprises: Determine a decoupling loss based on the upper region latent code, the lower region latent code and the facial action latent code at different times, wherein the decoupling loss is used to characterize the difference in facial decoupling at different times and is used to train the decoupled neural radiation field; The step of training the facial modeling model based on the three-dimensional reconstruction loss comprises: The facial modeling model is trained based on the decoupling loss and the three-dimensional reconstruction loss.

7. The method according to claim 6, wherein: The determining of the decoupling loss based on the upper region latent code, the lower region latent code and the facial action latent code at different moments includes: Acquire a first upper region latent code, a first lower region latent code, and a first facial action latent code corresponding to the sample facial image at a first moment, and acquire a second upper region latent code, a second lower region latent code, and a second facial action latent code corresponding to the sample facial image at a second moment; determining an upper decoupling loss based on the first upper region latent code, the first facial motion latent code, and the second lower region latent code; determining a lower decoupling loss based on the first lower region latent code, the first facial motion latent code, and the second upper region latent code; determining a motion decoupling loss based on the second facial motion latent code, the first upper region latent code, and the first lower region latent code; The decoupling loss is determined based on the upper decoupling loss, the lower decoupling loss, and the motion decoupling loss.

8. The method according to claim 7, wherein: The determining of the upper decoupling loss based on the first upper region latent code, the first facial action latent code and the second lower region latent code comprises: Inputting the first upper region latent code, the first facial action latent code, and the second lower region latent code into the neural radiation field to obtain a first facial image; determining the upper decoupling loss based on a difference between an upper half of the face in the first facial image and an upper half of the face in the sample facial image at the first moment; The determining of the lower decoupling loss based on the first lower region latent code, the first facial action latent code and the second upper region latent code comprises: Inputting the first lower region latent code, the first facial action latent code, and the second upper region latent code into the neural radiation field to obtain a second facial image; The lower decoupling loss is determined based on a difference between a lower half of the face in the second facial image and a lower half of the face in the sample facial image at the first moment.

9. The method according to claim 7, wherein: The determining the motion decoupling loss based on the second facial motion latent code, the first upper region latent code, and the first lower region latent code comprises: Inputting the second facial action latent code, the first upper region latent code, and the first lower region latent code into the neural radiation field to obtain a third facial image; The motion decoupling loss is determined based on a difference between the mask of the third facial image and the mask of the sample facial image at the first moment.

10. A three-dimensional facial modeling method, the method being performed by an electronic device, the method comprising: When receiving input text, determining text phonemes corresponding to the input text; Based on the phoneme latent index, query the target area latent code and the target action latent code corresponding to the text phoneme, wherein the phoneme latent index is used to indicate the correspondence between the phoneme and the latent code sequence, and the latent code sequence is obtained by feature encoding the facial image corresponding to the phoneme based on the encoder of the facial modeling model; By using the neural radiation field of the facial modeling model, three-dimensional reconstruction is performed based on the target area latent code and the target action latent code to obtain a facial action image corresponding to the text phoneme; Based on each frame of facial action image, a three-dimensional facial action animation corresponding to the input text is generated.

11. The method according to claim 10, wherein: The method further comprises: Acquire a text reading image, wherein the text reading image includes facial images of a reader reading the text acquired at different viewing angles at the same time; Based on the encoder, feature encoding is performed on the text reading image to obtain a facial latent code sequence corresponding to the text reading image, wherein the facial latent code sequence includes a region latent code and an action latent code; The facial latent code sequence is associated with the reading phonemes corresponding to the text reading image and stored to obtain the phoneme latent code index.

12. A training device for a facial modeling model, the device comprising: An image acquisition module is configured to acquire sample facial images, wherein the sample facial images include facial images of the same object at different viewing angles at the same time; A feature encoding module configured to perform feature encoding on the sample facial image through an encoder of the facial modeling model to obtain facial region latent codes corresponding to different facial regions and facial action latent codes corresponding to facial actions; A three-dimensional reconstruction module, configured to perform three-dimensional facial reconstruction on the object based on the facial region latent code and the facial action latent code through the neural radiation field of the facial modeling model to obtain a three-dimensional facial image of the object; The model training module is configured to obtain the three-dimensional reconstruction loss between the three-dimensional facial image and the sample facial image, and train the facial modeling model based on the three-dimensional reconstruction loss.

13. A three-dimensional facial modeling device, the device comprising: A phoneme determination module, configured to determine the text phoneme corresponding to the input text when receiving the input text; A latent code query module is configured to query the target area latent code and the target action latent code corresponding to the text phoneme based on a phoneme latent code index, wherein the phoneme latent code index is used to indicate the correspondence between the phoneme and the latent code sequence, and the latent code sequence is obtained by feature encoding the facial image corresponding to the phoneme based on the encoder of the facial modeling model; A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction based on the target area latent code and the target action latent code through the neural radiation field of the facial modeling model to obtain a facial action image corresponding to the text phoneme; The animation generation module is configured to generate a three-dimensional facial action animation corresponding to the input text based on each frame of facial action image.

14. An electronic device comprising: a memory configured to store computer executable instructions; The processor is configured to implement the training method of the facial modeling model as described in any one of claims 1 to 9, or implement the three-dimensional facial modeling method as described in any one of claims 10 to 11 when executing the computer executable instructions stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions for causing a processor to execute the instructions to implement the facial modeling model training method as described in any one of claims 1 to 9, or to implement the three-dimensional facial modeling method as described in any one of claims 10 to 11.

16. A computer program product, comprising a computer program or a computer executable instruction, wherein when the computer program or the computer executable instruction is executed by a processor, the training method of the facial modeling model as claimed in any one of claims 1 to 9 is implemented, or the three-dimensional facial modeling method as claimed in any one of claims 10 to 11 is implemented.

Images