Three-dimensional face generation and expression editing method and system based on text picture driving

Abstract

The invention belongs to the field of computer vision processing, and provides a three-dimensional face generation and expression editing method and system based on text and picture driving, which comprises the following steps: extracting key description information from source face text description, generating a control vector based on common mapping of the key description information and a source face picture; enhancing the source face picture by using the control vector to obtain a source face picture after text enhancement; face model parameters are extracted based on the source face picture after text enhancement, a rough shape is generated with the face model parameters as guidance, and detail enhancement and mapping rendering are carried out on the rough shape to generate a final three-dimensional face model; generating a source face picture after expression migration according to the target face picture and the source face picture, enhancing the source face picture after expression migration by using the target face text description to obtain an expression source face picture after text enhancement, and performing three-dimensional reconstruction on the expression source face picture after text enhancement to obtain an expression source face picture after three-dimensional reconstruction. And obtaining a source three-dimensional face model after expression editing.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision processing technology, and in particular relates to a method and system for generating and editing three-dimensional faces and expressions based on text and image driving. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous development of science and technology, 3D face modeling technology is gradually being applied to face recognition, virtual reality, game development and other fields. A common way to express a face model is to express a face with various expressions as a static face (i.e., a face without expression) and a set of blendshape bases. Each blendshape base models the changes in a face with a specific expression relative to a static face, including changes in geometry and appearance, and has specific semantics. Therefore, these blendshape bases can be linearly combined using blendshape coefficients and applied to a static face to obtain a face with a certain expression. There are three common methods for 3D face modeling: software modeling, instrument acquisition, and image-based modeling. Among them, some improvements have been made to the graphics-based 3D face modeling method. While using graphics, text is also combined to improve the image restoration of the face.

[0004] However, the three-dimensional face models generated by existing methods do not restore real face images to a high degree, lack facial details, and are prone to identity drift and poor expressiveness in extreme expressions when facial expression editing is performed on the three-dimensional face models. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a text-image driven 3D face generation and expression editing method and system. By combining the natural language description of facial details with the real face pictures input by the user, the present invention can generate a realistic and natural 3D face model and expression that meets the user's expectations.

[0006] According to some embodiments, a first solution of the present invention provides a method for generating and editing three-dimensional faces and expressions based on text and image driving, which adopts the following technical solutions: The method of generating and editing 3D faces and expressions based on text and image driving includes: Extract key description information from the source face text description, generate a control vector based on the key description information and the source face image, and enhance the source face image using the control vector to obtain the source face image after text enhancement; Extract facial model parameters based on the source face image after text enhancement, use the facial model parameters as a guide to generate a rough shape, and then perform detail enhancement and texture rendering on the rough shape to generate the final 3D face model; The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after the expression transfer. The source face image after the expression transfer is enhanced using the target face text description to obtain the expression source face image after the text is enhanced. The expression source face image after the text is enhanced is reconstructed in three dimensions to obtain the source three-dimensional face model after the expression is edited.

[0007] Furthermore, the key description information is extracted from the source face text description, a control vector is generated based on the key description information and the source face image, and the source face image is enhanced using the control vector, specifically: Generate a random latent code based on random noise, and then invert the source face image and encode it together with the random latent code to generate a latent code; Encode the source face text description to extract key description information and obtain text embedding; Perform attention decoding based on text embedding and latent encoding to generate control vectors; The control vector and the latent code are input into the generator to obtain the source face image with text enhancement.

[0008] Furthermore, the facial model parameters are extracted from the source face image after text enhancement, and the rough shape is generated using the facial model parameters as a guide, specifically: Use the visual encoder to encode the source face image after text enhancement to obtain the source face encoding features; Decode the source face encoding features to obtain facial shape parameters, posture parameters, expression parameters, albedo parameters, lighting parameters and camera parameters; Facial shape parameters, posture parameters, and expression parameters are input into the facial model to generate a rough shape.

[0009] Furthermore, the rough shape is subjected to detail enhancement and texture rendering to generate the final three-dimensional face model, specifically: Input the albedo parameters, lighting parameters and camera parameters in the facial model parameters into the trained DECA decoder to obtain the displacement map and model surface map; Based on the rough shape, the global and local features of the rough shape are extracted after preprocessing and encoding; Decode global and local features to obtain detailed shapes; Generate a normal map based on the displacement map, perform detail rendering on the normal map and the detailed shape, and then attach the model surface map to the detail rendering result to obtain the final 3D face model.

[0010] Furthermore, the normal map is generated based on the displacement map, the normal map and the detailed shape are rendered in detail, and then the model surface map is attached to the detailed rendering result to obtain the final 3D face model, specifically: Generate normal map based on displacement map; The product of the normal map and the displacement map is concatenated with the detailed shape to obtain a fine and detailed shape; Use the lighting rendering model to render the details of fine and delicate shapes to obtain detailed rendering results; Paste the model surface map onto the detail rendering result to obtain the final 3D face model.

[0011] Furthermore, the target expression is extracted from the target face picture and transferred to the source face picture to generate the source face picture after expression transfer, specifically: Extract the source identity of the source face image using the identity embedder; After extracting features from the source and target face images, a facial feature matrix is generated; Based on the source identity, the noisy target face image at the time step and the time step, the identity feature conditional denoising diffusion probability model is used to generate the source face image after expression transfer.

[0012] According to some embodiments, a second solution of the present invention provides a text-image driven 3D face generation and expression editing system, which adopts the following technical solutions: A text-image driven 3D face generation and expression editing system, including: a text enhancement module configured to extract key description information from the source face text description, generate a control vector based on the mapping of the key description information and the source face image, and enhance the source face image using the control vector to obtain the source face image after text enhancement; a 3D face reconstruction module configured to extract facial model parameters based on the source face image after text enhancement, generate a rough shape using the facial model parameters as a guide, and perform detail enhancement and texture rendering on the rough shape to generate a final 3D face model; The expression editing module is configured to extract the target expression based on the target face image and transfer it to the source face image, generate the source face image after the expression transfer, enhance the source face image after the expression transfer using the target face text description, obtain the expression source face image after the text enhancement, perform three-dimensional reconstruction on the expression source face image after the text enhancement, and obtain the source three-dimensional face model after the expression editing.

[0013] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the text-image-driven three-dimensional face generation and expression editing method as described in the first aspect above.

[0015] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the text-image-driven three-dimensional face generation and expression editing method described in the first aspect above are implemented.

[0017] According to a fifth aspect of the present invention, there is provided a computer program product or computer program according to some embodiments.

[0018] The present invention provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the text-image-driven 3D face generation and expression editing method described in the first aspect.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This method uses natural language input in text to modify attributes in the source facial image and supplement and enhance detailed features in the source facial image. This stage introduces the CLIP text encoder to extract the description of the source face from the text. This is then mapped together with the input source facial image into StyleGAN2 to generate a control vector. The control vector is then used to modify the attributes in the source facial image and enhance the details of unclear features. The resulting source facial image with modified attributes and enhanced facial details is called the text-enhanced source facial image.

[0020] The present invention uses the CLIP visual encoder to encode the source face image after text enhancement, obtains the FLAME model parameters through the decoder, and uses the parameters to guide the generation of a basic face mesh to obtain a rough three-dimensional face shape; the basic face mesh is passed through the CoMA model decoder to generate a high-resolution face mesh supplemented with facial details, that is, a detailed three-dimensional face shape; face displacement mapping and model surface mapping are performed, and rendering is performed to obtain a high-quality source three-dimensional face model with rich texture details.

[0021] In the three-dimensional facial expression editing module of the present invention, the expression of the target face can be transferred to the source face model while keeping the identity feature information of the source face unchanged, thereby realizing expression editing of the source face model. The three-dimensional facial expression editing module consists of three stages: the first stage is the two-dimensional facial expression migration stage, which uses the identity feature conditional denoising diffusion probability model and adopts a facial guided sampling strategy to replace the expression of the source face with the expression of the target face while keeping the identity feature of the source face unchanged, thereby obtaining the source face image after expression migration. The second stage is the text enhancement stage, which introduces the CLIP text encoder to extract the description of the target face in the text, and maps it together with the source face image after expression migration into StyleGAN2, so as to use the text description of the target face to supplement the detailed features in the source face image after expression migration, thereby obtaining a text-enhanced source face image after expression migration. The third stage is 3D face reconstruction. The source face image, after the text-enhanced expression transfer, is fed into the 3D face reconstruction network within the 3D face generation module. This reconstructs a 3D face model corresponding to the source face image after the text-enhanced expression transfer, effectively creating the source 3D face model for expression editing. The fusion of these two modalities, text and facial image data, significantly improves the accuracy of 3D face generation and expression editing.

[0022] The introduction of the CLIP encoder in this invention effectively captures both text and image features, overcoming the limitation of traditional methods that generate 3D faces using only 2D images, which struggle to accurately restore facial features in images. The introduction of the CoMA model as a detail enhancement module effectively addresses the issue of detail loss or distortion in 3D models generated by existing methods. The adoption of a facial-guided sampling strategy addresses the identity drift problem associated with existing expression editing methods. The introduction of the CLIP text encoder effectively supplements facial details with text descriptions of the target face, overcoming the limitation of existing methods that suffer from poor expressiveness when editing and generating extreme expressions on 3D face models. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 It is a framework of a method for generating and editing three-dimensional faces and expressions based on text and image driving in an embodiment of the present invention; Figure 2 This is a framework diagram of the text enhancement stage in an embodiment of the present invention; Figure 3 This is a framework diagram of the rough reconstruction stage in an embodiment of the present invention; Figure 4This is a schematic diagram of a three-dimensional basic face mesh generated by FLAME in an embodiment of the present invention; Figure 5 This is a framework diagram of the detailed reconstruction stage in an embodiment of the present invention; Figure 6 is a schematic diagram of the final three-dimensional face model in an embodiment of the present invention; Figure 7 This is a diagram of a three-dimensional facial expression editing framework according to an embodiment of the present invention; Figure 8 3D facial expression editing effect diagram according to an embodiment of the present invention; Figure 9 This is a comparison diagram of the effects of the method in the embodiment of the present invention and other three-dimensional face generation methods; Figure 10 This is a comparison of the method in the embodiment of the present invention and other expression editing models. DETAILED DESCRIPTION

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0027] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0028] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0029] Glossary: The CLIP model, developed by OpenAI, is a multimodal model capable of processing both text and image data. Its full name stands for Contrastive Language–Image Pretraining. Its core goal is to align text and images in the same semantic space through contrastive learning, enabling cross-modal understanding and generation.

[0030] Diffusion models: Diffusion models are a type of generative model that generate new data by simulating the gradual transformation of data from noise to a target distribution. The core idea of a diffusion model is to transform the data distribution into a simple noise distribution (such as a Gaussian distribution) through a process of gradually adding noise (the forward process). Then, a reverse process is learned to gradually restore the noise distribution back to the data distribution. When generating new data, the model starts with the noise and gradually removes the noise through the reverse process, ultimately generating high-quality data samples.

[0031] StyleGAN2: NVIDIA proposes an improved generative adversarial network (GAN) model, an upgraded version of StyleGAN. It aims to address some of the issues inherent in StyleGAN and further enhance the quality and diversity of generated images. StyleGAN2 has achieved significant success in image generation and has been widely used in tasks such as high-quality image synthesis, artistic creation, and data augmentation.

[0032] Attention Decoder: An attention-based decoder is a decoder architecture widely used in sequence-to-sequence (Seq2Seq) models, primarily for natural language processing (NLP) tasks such as machine translation, text generation, and speech recognition. The introduction of the attention mechanism significantly improves the model's ability to process long sequences and capture contextual information.

[0033] Example 1 The present embodiment provides a method for generating and editing three-dimensional faces and expressions based on text and image drive. The present embodiment uses the method applied to a server as an example. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal, a server, and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in this application. In the present embodiment, the method includes the following steps: Extract key description information from the source face text description, generate a control vector based on the key description information and the source face image, and enhance the source face image using the control vector to obtain the source face image after text enhancement; Extract facial model parameters based on the source face image after text enhancement, use the facial model parameters as a guide to generate a rough shape, and then perform detail enhancement and texture rendering on the rough shape to generate the final 3D face model; The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after the expression transfer. The source face image after the expression transfer is enhanced using the target face text description to obtain the expression source face image after the text is enhanced. The expression source face image after the text is enhanced is reconstructed in three dimensions to obtain the source three-dimensional face model after the expression is edited.

[0034] In view of the problems that the existing methods of reconstructing three-dimensional face models using face pictures have low restoration of real face pictures, missing face details, and prone to identity drift and poor expressiveness in extreme expressions when performing expression editing on three-dimensional face models, this embodiment proposes a three-dimensional face generation and expression editing method and system driven by text-image. The method can generate a real and natural source three-dimensional face model that restores the appearance and details of the source face based on the source face picture input by the user and the natural language text description of the source face details. On this basis, the user can continue to input the target face picture and the text description of the target face. The method can transfer the expression of the target face to the source three-dimensional face model while keeping the identity feature information of the source face unchanged, thereby realizing expression editing of the source three-dimensional face model. The overall framework of the method is as follows: Figure 1 The entire framework is divided into two processes: 3D face generation and 3D facial expression editing.

[0035] 3D face generation consists of two stages: text enhancement and 3D face reconstruction. The text enhancement stage uses the CLIP text encoder to extract the source face description from the text—the source face text description. This is then mapped together with the input source face image into StyleGAN2 to generate a control vector. The control vector is then used to modify the attributes of the source face image and enhance the details of unclear features. The resulting source face image with modified attributes and enhanced facial details is called the text-enhanced source face image. The 3D face reconstruction stage uses the text-enhanced source face image to guide the generation of the final 3D face model. First, the text-enhanced source face image is encoded using the CLIP visual encoder. The decoder generates the FLAME model parameters, which are used to guide the generation of a basic face mesh, resulting in a rough shape. Then, the CoMA model is introduced as a detail enhancement module. The rough shape is passed through the CoMA decoder to generate a high-resolution face mesh, i.e., a refined shape with added facial details. Finally, face displacement mapping and model surface mapping are performed, resulting in a high-quality, textured, and detailed source 3D face model, i.e., the final 3D face model.

[0036] Text enhancement stage, such as Figure 2 As shown in Figure 1, the text enhancement stage takes as input a source face image and a text description of the source face—the source face text description. The CLIP text encoder is introduced, leveraging the source face text description to perform natural and detailed attribute editing of the source face image, such as hairstyle, eye color, skin tone, and expression, to modify the source face image or complete unclear facial features in the source face image. By combining visual and textual information, the text enhancement stage accurately understands the semantics of the text description and maps it into the image space, enabling flexible control of image attributes.

[0037] The specific steps of the text enhancement stage are: first, the random noise The sample is sampled and input into the StyleGAN2 mapping network to obtain random latent codes. At the same time, the source face image is transformed into the latent space of the Gan model through Gan inversion and mixed with random noise. The random latent codes obtained are jointly encoded into latent codes by the Gan encoder .

[0038] Secondly, the source face text description of the source face Through CLIP text encoder The embedding layer converts each word or character into a vector representation. The core of the text encoder is a multi-layer Transformer layer, each of which includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism allows the model to focus on other words in the text when processing each word, thereby capturing the dependencies between words. The feedforward neural network further transforms the representation of each word to extract key facial attribute information.

[0039] The text embedding obtained by CLIP text encoder is denoted as . Again, and Input to the transformer-based attention decoder to generate offset , as the control vector. Finally, Input to the generator G of the StyleGAN2 model to generate the source face image after text enhancement, denoted as , this process is equivalent to according to the offset The original facial image is modified to complete the unclear features of the original face. Through text enhancement, the hair color of the original face is changed, the expression level is increased, and the facial features are made more obvious.

[0040] To ensure the text description of the source face Source face image enhanced with generated text The semantic consistency between them is achieved by using CLIP loss to ensure the alignment of image and text, which is expressed as (1) in, is a pre-set constant. and Represent the visual encoder and text encoder of CLIP respectively, Represents the cosine similarity between the two. Mapping text and image features into the same space achieves feature-level multimodal fusion, enabling the model to be pre-trained on multimodal data and improving model performance.

[0041] The 3D face reconstruction stage consists of two steps: coarse reconstruction and fine reconstruction. The text-enhanced source face image is input. The coarse reconstruction stage first produces a basic 3D face mesh model—the coarse shape. The fine reconstruction stage then generates a mesh model with high-frequency facial details—the fine shape, a facial displacement map, and a surface map. This is then rendered to create a 3D face model corresponding to the text-enhanced source face image.

[0042] The goal of the rough reconstruction stage is to build a basic 3D face mesh model - a rough shape. Given a text-enhanced source face image as input, it is transformed into features by the CLIP visual encoder, and the output features are input into the decoder to decode various parameters. The face-related parameters are then input into the FLAME model to generate a rough shape, i.e., the basic 3D face mesh. The specific process is as follows: Figure 4 shown.

[0043] The CLIP visual encoder is introduced to encode the source face image after text enhancement, and the image is passed through N Transformer layers in the CLIP visual encoder to extract the feature information. The details are as follows: the input source face image after text enhancement is first divided into multiple small blocks in units of pixels. The calculation formula is as follows: (2) in, Indicates the width of the image. Indicates the height of the image. Represents the side length of each small block. Each small block is converted into a one-dimensional vector and combined into a sequence with position encoding. This sequence is input into the CLIP visual encoder, which consists of multiple Transformer encoders. In the Transformer encoder, the relationship between each small block and other small blocks is calculated through a multi-head self-attention mechanism, which enables the model to capture the global feature information in the image, including posture, shape, and expression. Feature enhancement is performed through a feedforward neural network, and the scale difference between features is eliminated through a normalization layer. The parameters output by the encoder are input into the decoder as low-dimensional features. A two-layer fully connected neural network is first defined in the decoder as a regressor. The low-dimensional features are mapped to a higher-dimensional latent space through a fully connected layer and converted into a high-dimensional feature vector. Then, it passes through 4 convolutional layers, where each convolutional layer first improves the spatial resolution of the feature map through an upsampling layer so that the convolutional layer can more easily capture spatial structural information. Let the low-dimensional feature vector input to the decoder be z , then the output of the fully connected layer can be expressed as: (3) in, and are the weights and biases of the fully connected layer, is the activation function. After passing through 4 convolutional layers, the output of each convolutional layer can be expressed as: (4) in, is the activation function, represents the index of the convolutional layer (starting from 2), Indicates the The convolution kernel of the layer, represents the convolution operation, Indicates the The output feature map of the convolutional layer, Indicates the The specific network structure details of the decoder are shown in Table 1.

[0044] Table 1 Decoder structure details

[0045] After decoding, the feature information is converted into facial shape parameters , posture parameters , expression parameters , albedo parameter , lighting parameters , camera parameters . The facial shape parameters , posture parameters With expression parameters Input into the FLAME model, calculate the vertex offset through various parameters, and then add it to the FLAME template grid, and finally output a 5023*3 grid, that is, a three-dimensional basic face grid, such as Figure 5 shown.

[0046] The FLAME model provides a parametric way to represent a 3D face by adjusting its facial shape parameters. , posture parameters With expression parameters , the displacement of the FLAME template mesh vertices can be obtained, thereby flexibly changing the shape, expression and posture of the face. The process is expressed as follows: (5) Among them, the mixed skin function Around the joints Rotation The vertices in , and according to the blending weight Perform linear smooth transformation to adjust the vertex position. This process produces displacement with the facial template mesh, and the identity The function of the joint position is defined as ; Represents the average template, on which mixed shapes are added to add mixed expression shapes ,The FLAME model combines displacement and expression with a neutral face template in ,the latent space to generate a three-dimensional face model that matches the ,two-dimensional face image.

[0047] The CoMA model is introduced as a detail enhancement module in the fine reconstruction stage to upgrade the basic face mesh generated in the rough reconstruction stage to a high-resolution face mesh, thereby obtaining a detailed shape; the face displacement map and model surface map are generated by the DECA decoder to enhance the details of the face model, and the detailed shape of the face is rendered in detail by the differentiable renderer to obtain the final 3D face model. Figure 6 shown.

[0048] The albedo parameter obtained in the rough reconstruction stage , lighting parameters , camera parameters The detail codes are input into the trained DECA decoder, which converts these parameters into displacement maps and model surface maps. The displacement map defines the displacement amount at each position, and the model surface map represents the texture of the surface of the three-dimensional face model. Finally, detail rendering is performed based on the displacement map and the model surface map to create high-frequency details.

[0049] The CoMA model is introduced. It takes the base face mesh (coarse shape) obtained in the rough reconstruction phase as input. It first performs mesh preprocessing, uses an encoder to extract global and local features from the base face mesh, and uses a decoder to generate a high-resolution mesh (fine shape). During the decoding process, local geometric details such as wrinkles and skin texture are gradually added. A convolutional autoencoder is used, defined as: (6) in, For convolutional autodecoder. Details code To fine-tune the specific details of static characters, use expression parameters and jaw posture parameters To capture and show the details of facial wrinkles.

[0050] During detail rendering, an appropriate illumination model is required to more realistically simulate the effects of lighting on the face. In the field of facial reconstruction, illumination models based on spherical harmonics (SH) are one of the most commonly used methods. When assuming that the light source is far from the face and that the reflective properties of the facial surface conform to the Lambertian emission model (i.e., ideal scattering conditions), a facial image with shadow effects can be obtained: (7) in, Indicates the pixel position in UV coordinates. Pixels The final result of lighting rendering at represents the reflectivity, It is a material-related parameter that determines the albedo value. Represents the Hadamard product (i.e., element-wise multiplication). and Respectively The spherical harmonic coefficients and basis coefficients of order, Represents pixels in the UV coordinate system The surface normal at is calculated and spherical harmonics of different orders are summed to simulate lighting effects.

[0051] In the detail rendering process, the displacement map is first processed to generate a normal map to capture the subtle geometric changes on the face surface. The normal map is then combined with the high-resolution face mesh and surface normal data generated by the CoMA model to further generate a face geometry model with finer details, as shown below: (8) in, Representing finer geometric models - finer and more detailed shapes, Represents the geometric model generated by the CoMA model - detailed shape, represents the displacement map, represents its corresponding surface normal, Represents the Hadamard product. Calculate the normal information from the displacement map , rendered with the corresponding normal map To get detailed rendering , which is expressed as follows (9) Represents a rendering process. After the final detail rendering is completed, the model surface map is attached to the detail rendering result to obtain a 3D face model with accurate shape and high-frequency details, such as Figure 7 As shown. The model shows a very high level of realism and detail.

[0052] 3D facial expression editing After generating the source 3D face model, the 3D face expression editor allows the user to continue inputting the target face image and text description about the target face, thereby migrating the target face expression to the source 3D face model and realizing the expression editing of the source 3D face. Figure 8As shown in Figure 1, this module consists of three stages: the first stage is the 2D facial expression transfer stage. This stage utilizes an identity-based denoising diffusion probability model and a facial-guided sampling strategy to replace the source face's expression with that of the target face while preserving its identity, resulting in the transferred source face image. The second stage is the text enhancement stage. This stage introduces the CLIP text encoder to extract the target face's description from the text. This is then mapped together with the transferred source face image into StyleGAN2, where the text description of the target face is used to supplement the detailed features of the transferred source face image, resulting in the text-enhanced source face image. The third stage is the 3D face reconstruction stage. The text-enhanced source face image is input into the 3D face reconstruction network in the 3D face generation module to reconstruct a 3D face model corresponding to the text-enhanced source face image, thus obtaining the expression-edited source 3D face model.

[0053] In the 2D facial expression transfer stage, a denoising diffusion probability model based on identity characteristics is first introduced. The source and target face images provide facial guidance information, including facial features, expression, and posture. This information helps the model maintain the source identity characteristics when generating facial images. During the sampling process, an intermediate facial image is generated—the source face image after expression transfer. This image combines the source identity characteristics and target attributes while retaining the background characteristics of the source face image. This achieves the goal of replacing the source face's expression with that of the target face while preserving its identity characteristics, thus achieving expression transfer between 2D facial images.

[0054] The identity feature conditional denoising diffusion probability model is introduced. By gradually adding noise to the data and controlling the size and step size of the noise, the image is gradually converted into pure noise. Then an inverse process is learned to generate data from the noise. First, the identity information is injected into the identity feature conditional denoising diffusion probability model, which helps the model generate a facial image with the identity according to the identity label during the generation process. Here, the source face image x src Injected into identity embedder D i , get the source identity v id : v id =D i (x src ) (10) Then, the source identity v id and target face image x targ Embedded into the identity feature conditional denoising diffusion probability model ε θ (x t ,t,v id ), where x t is the target face image xtarg The noise version at time step t generates a real image from a standard Gaussian distribution by inverting the cyclic noise process. In the forward process, the data x0~q(x) gradually becomes a Gaussian distribution N, which is defined as follows: Among them, β t is a predefined variance table, I is the identity matrix, indicating the independence between dimensions and the variance is β t The reversal process is as follows: p θ (x t-1 |x t ):=N(x t-1 ;μ θ (x t ,t),σ(x t ,t)I) (12) The inverse cyclic noise process can generate a face image with the input identity label from the noise by iteratively removing the noise.

[0055] The purpose of face-guided sampling is to ensure that the model can accurately capture and fuse the identity information of the source face and the attribute information of the target face, thereby ensuring the generation of a new face image with the source identity and target attributes, that is, the source face image after expression transfer, to achieve a natural and realistic expression transfer effect.

[0056] The face-guided sampling strategy extracts features from the input source and target face images, including facial shape, texture, color, and other information, and encodes them into matrices. The face-guided sampling strategy uses these feature matrices to guide the model's sampling. During the sampling process, it's crucial to preserve the source face's features to prevent identity drift while also fully transferring the target face's expression to the source face. Therefore, the following formula is used to constrain the ID vector of the source face image and the ID vector of the source face image after expression transfer, ensuring that they are closer in the identity embedding space.

[0057] (13) in, The metric value representing identity similarity is used to measure the similarity between the source face and the generated face in terms of identity features during the expression transfer process. The closer the value is to 1, the better the identity features are preserved. Input to Identity Embedder The identity information of the source face and the attribute information of the target face obtained in the expression transfer are combined with the source face image after expression transfer. Find a suitable balance point in the feature space, which enables the generated image to retain the identity characteristics of the source face and incorporate the attribute characteristics of the target face, thereby realizing the migration of expressions on two-dimensional face images.

[0058] The stage of migrating source face images after text enhancement The text description of the target face is used to supplement and enhance the details of the source face image after expression transfer, so as to achieve more refined control of its expression. The process is the same as the previous text enhancement. Specifically, the source face image after expression transfer is input into the GAN inversion module, mapped into the latent space of GAN, and a latent vector is obtained. The latent vector is then input into the Gan decoder to obtain a latent code. At the same time, the text description of the target face entered by the user is passed through the CLIP text encoder to obtain the text embedding recorded as .Will and They are input into the attention decoder together, and the decoded features are generated using the StyleGAN2 model to obtain the expression source face image after text enhancement.

[0059] 3D face reconstruction stage By inputting the text-enhanced source facial image into the 3D face reconstruction network during the 3D face generation process, a 3D face model corresponding to the text-enhanced source facial image is reconstructed—the expression-edited source 3D face model. This 3D face model has the appearance of the source face and the expression of the target face, accurately transferring the expression information from the target facial image to the source 3D face model, thereby enabling expression editing of the source 3D face model. Figure 8 Demonstrates the effect of 3D facial expression editing.

[0060] Experiments on 3D face generation Datasets: Three datasets were used: VGGFace2, BUPT-Balancedface, and NoW. VGGFace2 images were downloaded from Google Image Search and contain 9,131 faces of individuals with significant variations in pose, age, lighting, ethnicity, and occupation, enabling matching of faces across poses and ages. BUPT-Balancedface includes multi-ethnic face images of varying skin tones, ensuring a relatively balanced number of faces of each ethnicity. This helps evaluate the performance consistency of face recognition algorithms across different ethnicities. The NoW (Not Quite in the Wild) dataset contains 2,054 photos of 100 subjects, covering a range of ages, BMIs, and genders (55 females, 45 males).

[0061] Training set: The VGGFACE2 dataset and BUPT-Balancedface dataset were preprocessed, and 68 2D facial landmarks were predicted using FAN (Face Alignment Network). Facial segmentation technology was used to obtain the skin mask, and the model from VGGFace2-pytorch was used to calculate the identity loss. This training set was used to train the 3D face generation module proposed in this paper.

[0062] Test set: The NoW dataset is used as the test set, and the distance from all reference scan vertices to the nearest point on the reconstructed mesh surface is measured.

[0063] Training strategy: The machine configuration and development environment used were: Ubuntu 20.04, GeForce RTX4090, CUDA 12.2, PyTorch 2.1.0, Vscode 1.91.1, and Python 3.8.19. The model was trained for 30 epochs with a batch size of 32.

[0064] Comparative experimental results: The 3D face generation method in this paper is compared with other 3D face generation methods. The generation models of all methods are retrained using the same VGGFace2 and BUPT-Balancedface training sets and environment as the method in this paper. The purpose of this is to ensure that all methods are compared under the same conditions, effectively eliminating the impact of differences in training data or environment on the comparison results, so as to obtain fair results. Figure 9 This figure shows the comparison results of our method with other comparison methods on the NOW evaluation dataset. The first row shows a single source face image as input. The second through fourth rows show the source 3D face models generated using 3DDFA-V2, DECA, and our method, respectively. The last row shows the input text description of the source face, such as a prompt word describing the facial expression. This shows that our method is able to more accurately capture the facial details in the input image, especially the facial expressions, and present them in the generated 3D face models with a high degree of realism and naturalness.

[0065] Table 2 Reconstruction error of our 3D face generation model and other models on the NoW dataset

[0066] Table 2 shows the reconstruction error of our 3D face generation model and other models with single-modal input on the NoW dataset. It can be seen that our model outperforms other models.

[0067] Ablation experiments Ablation experiments were conducted on the CLIP encoder and CoMA detail enhancement module in this 3D face generation module to demonstrate their beneficial effect on 3D face model generation. Table 3 shows the reconstruction performance of the model with each module removed and the complete model on the NoW dataset. As can be seen, the complete model achieves the lowest reconstruction error across all three categories.

[0068] Table 3 Ablation experiment results of 3D face generation module

[0069]

[0070] Experiments on 3D facial expression editing Dataset: We use the FFHQ face dataset. The FFHQ (Flickr-Faces-High-Quality) dataset is a high-quality facial image dataset containing 70,000 high-quality facial images. The images cover a wide range of facial attributes, including age, gender, ethnicity, skin color, expression, face shape, hairstyle, and pose, as well as various facial accessories such as glasses, sunglasses, and hats.

[0071] Training Strategy: The machine configuration and development environment used in the experiment are: Ubuntu 20.04, GeForce RTX 4090, CUDA 12.2, PyTorch 2.1.0, Vscode 1.91.1, and Python 3.8.19. The model was trained for 30 epochs with a batch size of 32.

[0072] Comparative experimental results: The 3D facial expression editing method proposed in this paper is compared with other single-modal 3D facial expression editing methods. All models are retrained in the same environment using the same FFHQ dataset and the same 3D face generation method is used to generate the 3D face models after expression editing. This is done to ensure that all methods are compared under the same conditions to obtain fair results. Figure 10 As shown in the figure, the first column is the source face image, the second column is the target face image, and the third column is the face image after expression transfer. It can be seen that when the single-modal models SimSwap and 3dSwap handle extreme expressions of surprise (open mouth), the transferred face exhibits facial deformation and expression distortion, and identity drift. However, by inputting the text prompt "surprise", this method not only restores the extreme expression, but also maintains the identity characteristics of the original face, making the 3D face model after expression editing more realistic and natural.

[0073] Ablation experiments Ablation experiments were conducted on the CLIP encoder and identity-conditional denoising diffusion probability model (DDPM) in our 3D facial expression editing module to demonstrate their beneficial effect on expression editing of 3D facial models. Table 4 shows the reconstruction performance of the model after removing each module, as well as the full model, on the NoW dataset. As can be seen, the full model achieves the lowest reconstruction error across all three categories.

[0074] Table 4 Ablation experiment results of 3D facial expression editing module

[0075] Example 2 This embodiment provides a text-image driven 3D face generation and expression editing system, including: a text enhancement module configured to extract key description information from the source face text description, generate a control vector based on the mapping of the key description information and the source face image, and enhance the source face image using the control vector to obtain the source face image after text enhancement; a 3D face reconstruction module configured to extract facial model parameters based on the source face image after text enhancement, generate a rough shape using the facial model parameters as a guide, and perform detail enhancement and texture rendering on the rough shape to generate a final 3D face model; The expression editing module is configured to extract the target expression based on the target face image and transfer it to the source face image, generate the source face image after the expression transfer, enhance the source face image after the expression transfer using the target face text description, obtain the expression source face image after the text enhancement, perform three-dimensional reconstruction on the expression source face image after the text enhancement, and obtain the source three-dimensional face model after the expression editing.

[0076] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0077] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] The proposed system can be implemented in other ways. For example, the system embodiment described above is merely illustrative. For example, the above module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0079] Example 3 This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the text-image driven 3D face generation and expression editing method as described in the first embodiment are implemented.

[0080] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the text-image-driven three-dimensional face generation and expression editing method described in the first embodiment are implemented.

[0081] Example 5 This embodiment provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the text-image-driven 3D face generation and expression editing method described in the first embodiment.

[0082] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0086] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0087] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A method for generating and editing three-dimensional faces and expressions based on text and image driving, characterized in that: include: Extract key description information from the source face text description, generate a control vector based on the key description information and the source face image, and enhance the source face image using the control vector to obtain the source face image after text enhancement; Extract facial model parameters based on the source face image after text enhancement, use the facial model parameters as a guide to generate a rough shape, and then perform detail enhancement and texture rendering on the rough shape to generate the final 3D face model; The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after the expression transfer. The source face image after the expression transfer is enhanced using the target face text description to obtain the expression source face image after the text is enhanced. The expression source face image after the text is enhanced is reconstructed in three dimensions to obtain the source three-dimensional face model after the expression is edited.

2. The method for generating and editing three-dimensional faces based on text and image driving according to claim 1, wherein: The key description information is extracted from the source face text description, a control vector is generated based on the key description information and the source face image, and the source face image is enhanced using the control vector, specifically: Generate a random latent code based on random noise, and then invert the source face image and encode it together with the random latent code to generate a latent code; Encode the source face text description to extract key description information and obtain text embedding; Perform attention decoding based on text embedding and latent encoding to generate control vectors; The control vector and the latent code are input into the generator to obtain the source face image with text enhancement.

3. The method for generating and editing three-dimensional faces and expressions based on text and image driving according to claim 1, wherein: The facial model parameters are extracted from the source face image after text enhancement, and the rough shape is generated using the facial model parameters as a guide, specifically: Use the visual encoder to encode the source face image after text enhancement to obtain the source face encoding features; Decode the source face encoding features to obtain facial shape parameters, posture parameters, expression parameters, albedo parameters, lighting parameters and camera parameters; Facial shape parameters, posture parameters, and expression parameters are input into the facial model to generate a rough shape.

4. The method for generating and editing three-dimensional faces based on text and image driving according to claim 1, wherein: The detailed enhancement and texture rendering of the rough shape to generate the final 3D face model is specifically as follows: Input the albedo parameters, lighting parameters and camera parameters in the facial model parameters into the trained DECA decoder to obtain the displacement map and model surface map; Based on the rough shape, the global and local features of the rough shape are extracted after preprocessing and encoding; Decode global and local features to obtain detailed shapes; Generate a normal map based on the displacement map, perform detail rendering on the normal map and the detailed shape, and then attach the model surface map to the detail rendering result to obtain the final 3D face model.

5. The method for generating and editing three-dimensional faces and expressions based on text and image driving according to claim 4, wherein: The method generates a normal map based on the displacement map, performs detail rendering on the normal map and the detailed shape, and then attaches the model surface map to the detail rendering result to obtain the final 3D face model. Specifically: Generate normal map based on displacement map; The product of the normal map and the displacement map is concatenated with the detailed shape to obtain a fine and detailed shape; Use the lighting rendering model to render the details of fine and delicate shapes to obtain detailed rendering results; Paste the model surface map onto the detail rendering result to obtain the final 3D face model.

6. The method for generating and editing three-dimensional faces based on text and image driving according to claim 1, wherein: The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after expression transfer, specifically: Extract the source identity of the source face image using the identity embedder; After extracting features from the source and target face images, a facial feature matrix is generated; Based on the source identity, the noisy target face image at the time step and the time step, the identity feature conditional denoising diffusion probability model is used to generate the source face image after expression transfer.

7. A 3D face generation and expression editing system based on text and image drive, characterized by: include: a text enhancement module configured to extract key description information from the source face text description, generate a control vector based on the mapping of the key description information and the source face image, and enhance the source face image using the control vector to obtain the source face image after text enhancement; a 3D face reconstruction module configured to extract facial model parameters based on the source face image after text enhancement, generate a rough shape using the facial model parameters as a guide, and perform detail enhancement and texture rendering on the rough shape to generate a final 3D face model; The expression editing module is configured to extract the target expression based on the target face image and transfer it to the source face image, generate the source face image after the expression transfer, enhance the source face image after the expression transfer using the target face text description, obtain the expression source face image after the text enhancement, perform three-dimensional reconstruction on the expression source face image after the text enhancement, and obtain the source three-dimensional face model after the expression editing.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the text-image-driven three-dimensional face generation and expression editing method according to any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the text-image-driven three-dimensional face generation and expression editing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the text-image-driven 3D face generation and expression editing method according to any one of claims 1 to 6.

Images