Text image driven three-dimensional face generation and expression editing method and system

Abstract

The application belongs to the field of computer vision processing, and provides a three-dimensional face generation and expression editing method and system based on a text picture driving, key description information is extracted from a source face text description, a control vector is generated based on the key description information and the source face picture, the source face picture is enhanced by using the control vector, and a source face picture after text enhancement is obtained; face model parameters are extracted based on the source face picture after text enhancement, a rough shape is generated by taking the face model parameters as a guide, the rough shape is enhanced in details and is rendered by mapping to generate a final three-dimensional face model; a source face picture after expression migration is generated according to a target face picture and the source face picture, the source face picture after expression migration is enhanced by using a target face text description, a source face picture after expression migration and text enhancement is obtained, and a source three-dimensional face model after expression editing is obtained by performing three-dimensional reconstruction on the source face picture after expression migration and text enhancement.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing technology, specifically relating to a method and system for generating and editing three-dimensional faces and expressions based on text images. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous development of technology, 3D face modeling technology is gradually being applied to fields such as face recognition, virtual reality, and game development. A common method for representing face models is to represent faces with various expressions as static faces (i.e., expressionless faces) and a set of blendshape bases. Each blendshape base models the changes in the face with a specific expression relative to the static face, including geometric and appearance changes, and possesses specific semantics. Therefore, blendshape coefficients can be used to linearly combine these blendshape bases and apply them to the static face to obtain a face with a specific expression. The three main methods of 3D face modeling are software modeling, instrument acquisition, and image-based modeling. Among these, some improvements have been made to image-based 3D face modeling methods, combining text with graphics to improve the accuracy of image-based face reconstruction.

[0004] However, existing methods generate 3D face models that do not accurately reproduce real face images, lack facial details, and are prone to identity drift and poor expressiveness in extreme expressions when editing facial expressions on 3D face models. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a text-image-driven method and system for generating and editing 3D faces and expressions. By combining natural language descriptions of facial details with real facial images input by the user, this invention can generate realistic and natural 3D face models and expressions that meet the user's expectations.

[0006] According to some embodiments, the first solution of the present invention provides a method for generating and editing three-dimensional faces and expressions based on text images, employing the following technical solution:

[0007] A text-image driven 3D face generation and expression editing method includes:

[0008] Key descriptive information is extracted from the source face text description. Based on the key descriptive information and the source face image, a control vector is generated. The control vector is then used to enhance the source face image, resulting in a text-enhanced source face image.

[0009] Facial model parameters are extracted from the text-enhanced source face image. The facial model parameters are used as a guide to generate a rough shape. The rough shape is then enhanced with details and rendered with textures to generate the final 3D face model.

[0010] The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after expression transfer. The source face image after expression transfer is enhanced by the target face text description to obtain the text-enhanced expression source face image. The text-enhanced expression source face image is then reconstructed in 3D to obtain the source 3D face model after expression editing.

[0011] Furthermore, the step of extracting key descriptive information from the source face text description, generating a control vector based on the key descriptive information and the source face image, and using the control vector to enhance the source face image specifically involves:

[0012] Random latent codes are generated based on random noise. At the same time, the source face image is inverted and co-encoded with the random latent codes to generate a latent code.

[0013] The source face text description is encoded to extract key descriptive information, resulting in text embedding;

[0014] Attention decoding is performed based on text embedding and latent encoding to generate control vectors;

[0015] The control vector and latent encoding are input into the generator to obtain the text-enhanced source face image.

[0016] Furthermore, the step of extracting facial model parameters from the text-enhanced source face image and using these parameters as guidance to generate a rough shape specifically involves:

[0017] The source face image after text enhancement is encoded using a visual encoder to obtain the source face encoding features;

[0018] Decode the source face encoding features to obtain facial shape parameters, pose parameters, expression parameters, albedo parameters, illumination parameters, and camera parameters;

[0019] The facial shape parameters, pose parameters, and expression parameters are input into the facial model to generate a rough shape.

[0020] Furthermore, the process of enhancing the details of the rough shape and performing texture rendering to generate the final 3D face model specifically involves:

[0021] The albedo parameters, lighting parameters, and camera parameters in the facial model parameters are input into the trained DECA decoder to obtain the displacement map and the model surface map.

[0022] The rough shape is preprocessed and then encoded to extract global and local features of the rough shape;

[0023] Decoding global and local features yields detailed shapes;

[0024] Normal maps are generated based on displacement maps. The normal maps and detailed shapes are then rendered in detail. Finally, the model surface maps are applied to the detailed rendering results to obtain the final 3D human face model.

[0025] Furthermore, the process of generating a normal map based on a displacement map, rendering the normal map and detailed shape, and then applying the model surface map to the rendered detail result to obtain the final 3D face model is as follows:

[0026] Generating normal maps based on displacement maps;

[0027] The product of the normal map and the displacement map is combined with the detailed shape to obtain a fine, detailed shape;

[0028] The lighting rendering model is used to render the fine and detailed shapes in detail, and the detailed rendering result is obtained.

[0029] The surface texture of the model is applied to the detailed rendering result to obtain the final 3D face model.

[0030] Furthermore, the step of extracting the target expression from the target face image and transferring it to the source face image to generate the expression-transferred source face image specifically involves:

[0031] Extract the source identity of the source face image using an identity embedder;

[0032] After extracting features from the source and target face images, a facial feature matrix is ​​generated.

[0033] Based on the source identity, the noisy target face image at the time step, and the time step, a denoising diffusion probability model with identity features is used to generate the source face image after expression transfer.

[0034] According to some embodiments, a second aspect of the present invention provides a text image-driven 3D face generation and expression editing system, employing the following technical solution:

[0035] A text-image driven 3D face generation and expression editing system includes:

[0036] The text enhancement module is configured to extract key description information from the source face text description, generate a control vector based on the key description information and the source face image, and enhance the source face image using the control vector to obtain the text-enhanced source face image.

[0037] The 3D face reconstruction module is configured to extract facial model parameters based on the text-enhanced source face image, use the facial model parameters as a guide to generate a rough shape, and then perform detail enhancement and texture rendering on the rough shape to generate the final 3D face model.

[0038] The expression editing module is configured to extract the target expression from the target face image and transfer it to the source face image to generate the source face image after expression transfer. The source face image after expression transfer is enhanced by the target face text description to obtain the text-enhanced expression source face image. The text-enhanced expression source face image is then reconstructed in 3D to obtain the source 3D face model after expression editing.

[0039] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the text image-driven 3D face generation and expression editing method described in the first aspect above.

[0041] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0042] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the text image-driven 3D face generation and expression editing method described in the first aspect above.

[0043] According to some embodiments, a fifth aspect of the present invention provides a computer program product or computer program.

[0044] This invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the text-image-driven 3D face generation and expression editing method described in the first aspect above.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] This invention utilizes natural language text input to modify attributes and enhance detailed features in source face images. This stage introduces a CLIP text encoder to extract descriptions of the source face from the text, which are then mapped together with the input source face image into StyleGAN2 to generate control vectors. These control vectors are then used to modify attributes in the source face image and enhance unclear features, resulting in a source face image with altered attributes and enhanced facial details, referred to as the text-enhanced source face image.

[0047] This invention utilizes the CLIP visual encoder to encode the text-enhanced source face image, and then uses the decoder to obtain FLAME model parameters. These parameters are used to guide the generation of a basic face mesh, resulting in a coarse 3D face shape. The basic face mesh is then processed by the CoMA model decoder to generate a high-resolution face mesh with added face details, resulting in a detailed 3D face shape. Face displacement mapping and model surface mapping are then performed to render a high-quality source 3D face model with rich texture details.

[0048] This invention, in its 3D facial expression editing module, can transfer the expression of a target face to a source face model while preserving the source face's identity features, thus enabling expression editing of the source face model. The 3D facial expression editing module consists of three stages: The first stage is a 2D facial expression transfer stage. Utilizing an identity feature-based denoising diffusion probability model and a facial guided sampling strategy, it replaces the source face's expression with the target face's expression while maintaining the source face's identity features, resulting in a source face image with transferred expression. The second stage is a text enhancement stage. This stage introduces a CLIP text encoder to extract descriptions of the target face from the text, and maps these descriptions, along with the source face image with transferred expression, into StyleGAN2. This allows the text descriptions of the target face to supplement the detailed features in the source face image with transferred expression, resulting in a text-enhanced source face image with transferred expression. The third stage is the 3D face reconstruction stage. The source face image, after text-enhanced expression transfer, is input into the 3D face reconstruction network within the 3D face generation module. This reconstructs a 3D face model corresponding to the text-enhanced expression-transferred source face image, thus realizing the source 3D face model for expression editing. The fusion of text and face image data significantly improves the accuracy of 3D face generation and expression editing.

[0049] The CLIP encoder introduced in this invention effectively captures text and image features, overcoming the shortcomings of traditional methods that rely solely on 2D images to generate 3D faces, which struggle to accurately recreate facial features. The introduction of a CoMA model as a detail enhancement module effectively solves the problem of detail loss or distortion in 3D models generated by existing methods. The adoption of a facial guided sampling strategy addresses the issue of identity drift common in existing expression editing methods. The introduction of the CLIP text encoder effectively supplements facial details using textual descriptions of the target face, overcoming the poor expressiveness of existing methods when generating extreme expressions on 3D face models. Attached Figure Description

[0050] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0051] Figure 1 This is a framework for a text-image driven 3D face generation and expression editing method in this embodiment of the invention;

[0052] Figure 2 This is a framework diagram of the text enhancement stage in an embodiment of the present invention;

[0053] Figure 3 This is a framework diagram of the rough reconstruction stage in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the three-dimensional basic face mesh generated by FLAME in an embodiment of the present invention;

[0055] Figure 5 This is a detailed reconstruction stage framework diagram in an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram of the final three-dimensional human face model in an embodiment of the present invention;

[0057] Figure 7 This is a diagram of the three-dimensional facial expression editing framework in an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the three-dimensional facial expression editing effect in an embodiment of the present invention;

[0059] Figure 9 This is a comparison diagram of the effects of the method in this embodiment of the invention with other 3D face generation methods;

[0060] Figure 10 This is a comparison of the method in this embodiment with other facial expression editing models. Detailed Implementation

[0061] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0062] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0063] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0064] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0065] Definitions:

[0066] CLIP model: Developed by OpenAI, this multimodal model can process both text and image data simultaneously. Its full name is Contrastive Language–Image Pretraining, and its core goal is to align text and images in the same semantic space through contrastive learning, thereby achieving cross-modal understanding and generation.

[0067] Diffusion models are a type of generative model that generates new data by simulating the gradual transformation of data from noise to a target distribution. The core idea of ​​diffusion models is to transform the data distribution into a simple noise distribution (such as a Gaussian distribution) through a process of gradually adding noise (forward process), and then learn an inverse process to gradually restore the noise to the data distribution. When generating new data, the model starts with noise, gradually removes noise through the inverse process, and finally generates high-quality data samples.

[0068] StyleGAN2: StyleGAN2 is an improved Generative Adversarial Network (GAN) model proposed by NVIDIA, and is an upgraded version of StyleGAN. It aims to address some of the problems existing in StyleGAN and further improve the quality and diversity of generated images. StyleGAN2 has achieved significant success in the field of image generation and is widely used in tasks such as high-quality image synthesis, artistic creation, and data augmentation.

[0069] Attention-based decoder: An attention-based decoder is a widely used decoder architecture in sequence-to-sequence (Seq2Seq) models, primarily for natural language processing (NLP) tasks such as machine translation, text generation, and speech recognition. The introduction of the attention mechanism significantly improves the model's ability to handle long sequences and capture contextual information.

[0070] Example 1

[0071] This embodiment provides a method for generating and editing 3D faces and expressions based on text-image driven technology. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through interaction between the terminal and the server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0072] Key descriptive information is extracted from the source face text description. Based on the key descriptive information and the source face image, a control vector is generated. The control vector is then used to enhance the source face image, resulting in a text-enhanced source face image.

[0073] Facial model parameters are extracted from the text-enhanced source face image. The facial model parameters are used as a guide to generate a rough shape. The rough shape is then enhanced with details and rendered with textures to generate the final 3D face model.

[0074] The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after expression transfer. The source face image after expression transfer is enhanced by the target face text description to obtain the text-enhanced expression source face image. The text-enhanced expression source face image is then reconstructed in 3D to obtain the source 3D face model after expression editing.

[0075] To address the problems of low fidelity to real-world face images, missing facial details, and identity drift and poor expressiveness in extreme expressions when reconstructing 3D face models from facial images using existing methods, this embodiment proposes a text-image jointly driven method and system for 3D face generation and expression editing. The method generates a realistic and natural source 3D face model that recreates the appearance and details of the source face based on a user-input source face image and a natural language text description of the source face's details. Furthermore, the user can input a target face image and a text description of the target face. The method can then transfer the expression of the target face to the source 3D face model while preserving the identity features of the source face, thus enabling expression editing of the source 3D face model. The overall framework of the method is as follows: Figure 1 As shown, the entire framework consists of two processes: 3D face generation and 3D face expression editing.

[0076] 3D face generation comprises two stages: text enhancement and 3D face reconstruction. The text enhancement stage introduces the CLIP text encoder to extract descriptions of the source face from the text—the source face text description. This description, along with the input source face image, is mapped to StyleGAN2 to generate control vectors. These control vectors are then used to modify attributes in the source face image and enhance details of unclear features, resulting in a source face image with altered attributes and enhanced facial details, known as the text-enhanced source face image. The 3D face reconstruction stage uses the text-enhanced source face image to guide the generation of the final 3D face model. First, the CLIP visual encoder encodes the text-enhanced source face image, which is then decoded to obtain FLAME model parameters. These parameters guide the generation of a basic face mesh, resulting in a rough shape. Next, the CoMA model is introduced as a detail enhancement module. The rough shape is decoded by the CoMA decoder to generate a high-resolution face mesh, thus adding detailed facial features. Finally, face displacement mapping and model surface mapping are performed, rendering a high-quality, textured, and detailed source 3D face model—the final 3D face model.

[0077] Text enhancement stage, such as Figure 2 As shown, the text enhancement stage takes a source face image and a text description of the source face—the source face text description—as input. A CLIP text encoder is introduced to perform natural and detailed attribute editing on the source face image using the text description of the source face, such as hairstyle, eye color, skin tone, and expression, to modify the source face image or complete unclear facial features. By combining visual and textual information, the text enhancement stage can accurately understand the semantics in the text description and map it to the image space, thereby achieving flexible control over image attributes.

[0078] The specific steps of the text enhancement stage are as follows: First, random noise is processed... Sampling is performed and input into the StyleGAN2 mapping network to obtain random latent codes. Simultaneously, the source face image is inverted through the GAN and transformed into the latent space of the GAN model, where it is mixed with random noise. The obtained random latent codes are co-encoded into latent codes by a Gan encoder. .

[0079] Secondly, regarding the source face text description. After CLIP text encoder The embedding layer converts each word or character into a vector representation. The core of the text encoder is a multi-layered Transformer layer, each containing a self-attention mechanism and a feedforward neural network. The self-attention mechanism allows the model to focus on other words in the text while processing each word, thus capturing the dependencies between words. The feedforward neural network further transforms the representation of each word, extracting key facial attribute information.

[0080] The text embedding obtained after CLIP text encoder is denoted as... Again, and The input is fed into a transformer-based attention decoder to generate an offset. , as the control vector. Finally, The generator G, input into the StyleGAN2 model, generates a text-enhanced source face image, denoted as . This process is equivalent to adjusting the offset. This modifies the source face image, thus completing the unclear features of the source face. Text enhancement changes the hair color of the source face, increases the expressiveness of facial expressions, and makes the facial features more prominent.

[0081] To ensure the text description of the source face The source face image enhanced with generated text Semantic consistency between them is ensured using CLIP loss to guarantee image-text alignment, represented as...

[0082] (1)

[0083] in, It is a pre-set constant. and These represent CLIP's visual encoder and text encoder, respectively. This represents the cosine similarity between the two. Mapping text and image features to the same space enables feature-level multimodal fusion, allowing the model to be pre-trained on multimodal data and improving model performance.

[0084] The 3D face reconstruction stage is divided into two steps: coarse reconstruction and fine reconstruction. The input source face image, enhanced with text, first undergoes a coarse reconstruction stage to obtain a basic 3D face mesh model – a rough shape. Then, a fine reconstruction stage generates a mesh model with high-frequency facial details – a fine shape, face displacement maps, and model surface maps, which are then rendered to obtain a 3D face model corresponding to the text-enhanced source face image.

[0085] The goal of the coarse reconstruction stage is to construct a basic 3D face mesh model – a rough shape. Given a text-enhanced source face image as input, it undergoes feature transformation via the CLIP visual encoder. The output features are then fed into a decoder to obtain various parameters. The face-related parameters are then input into the FLAME model to generate the rough shape, i.e., the basic 3D face mesh. The specific process is as follows: Figure 4 As shown.

[0086] A CLIP visual encoder is introduced to encode the text-enhanced source face image. The image is then passed through N Transformer layers in the CLIP visual encoder to extract its feature information. Specifically, the input text-enhanced source face image is first divided into multiple small blocks, pixel by pixel. The calculation formula is as follows:

[0087] (2)

[0088] in, Indicates the width of the image. Indicates the height of the image. This represents the side length of each small block. Each small block is converted into a one-dimensional vector and combined with a positional encoding to form a sequence. This sequence is input into the CLIP visual encoder, which consists of multiple Transformer encoders. In the Transformer encoder, the relationship between each small block and other small blocks is calculated through a multi-head self-attention mechanism, enabling the model to capture global feature information in the image, including pose, shape, and expression. Feature enhancement is performed through a feedforward neural network, and scale differences between features are eliminated through a normalization layer. The parameters output by the encoder are used as low-dimensional features input to the decoder. The decoder first defines a two-layer fully connected neural network as a regressor, which maps the low-dimensional features to a higher-dimensional latent space through the fully connected layers, converting them into high-dimensional feature vectors. Then, they pass through four convolutional layers, where each convolutional layer first uses an upsampling layer to improve the spatial resolution of the feature map so that the convolutional layers can more easily capture spatial structure information. Let the low-dimensional feature vector input to the decoder be... zThe output of the fully connected layer can then be expressed as:

[0089] (3)

[0090] in, and These are the weights and biases of the fully connected layer. It is the activation function. Then, through four convolutional layers, the output of each convolutional layer can be represented as:

[0091] (4)

[0092] in, For activation function, Indicates the index of the convolutional layer (starting from 2). Indicates the first The convolution kernel of the layer, This represents the convolution operation. Indicates the first Output feature maps of convolutional layers Indicates the first Layer bias. See Table 1 for details of the decoder's specific network structure.

[0093] Table 1 Decoder Structure Details

[0094]

[0095] The decoded feature information is then converted into facial shape parameters. Posture parameters Expression parameters albedo parameter Lighting parameters Camera parameters . Facial shape parameters Posture parameters With facial expression parameters The input is fed into the FLAME model, vertex offsets are calculated using various parameters, and then added to the FLAME template mesh. The final output is a 5023*3 mesh, which is the basic 3D face mesh, as shown below. Figure 5 As shown.

[0096] The FLAME model provides a parametric way to represent 3D faces by adjusting their facial shape parameters. Posture parameters With facial expression parameters This allows us to obtain the displacement of the vertices of the FLAME template mesh, thereby flexibly changing the shape, expression, and pose of the face. The process is represented as follows:

[0097] (5)

[0098] Among them, the hybrid skin function Around the joint Rotation The vertices in the data, and based on the mixed weights A linear smooth transformation is performed to adjust the vertex positions. This process causes displacement of the facial template mesh, which will affect the identity. The function is defined as the joint position. ; This represents the average template, upon which blended shapes are added, including blended facial expression shapes. The FLAME model combines displacement and expression with a neutral face template in the latent space to generate a 3D face model that matches a 2D face image.

[0099] In the detailed reconstruction stage, the CoMA model is introduced as a detail enhancement module, upgrading the basic face mesh generated in the coarse reconstruction stage to a high-resolution face mesh, thus obtaining a detailed shape. The DECA decoder generates face displacement maps and model surface maps to enhance the details of the face model. These detailed face shapes are then rendered using a differentiable renderer to obtain the final 3D face model. The workflow is as follows: Figure 6 As shown.

[0100] albedo parameters obtained during the coarse reconstruction stage Lighting parameters Camera parameters The detailed code is input into the trained DECA decoder, which converts these parameters into displacement maps and model surface maps. The displacement map defines the displacement at each position, and the model surface map represents the texture of the 3D face model surface. Finally, high-frequency details are created by rendering details based on the displacement map and model surface map.

[0101] A CoMA model is introduced, using the basic face mesh (rough shape) obtained in the coarse reconstruction stage as input. Mesh preprocessing is performed first, then an encoder extracts global and local features of the basic face mesh, and a decoder generates a high-resolution mesh (detailed shape). During decoding, local geometric details, such as wrinkles and skin texture, are gradually added. A convolutional autoencoder is used, defined as:

[0102] (6)

[0103] in, This is an automatic convolutional decoder. See the detailed code. To finely control specific details of static figures, using facial expression parameters and jaw posture parameters To capture and showcase the details of facial wrinkles.

[0104] In the detailed rendering process, an appropriate lighting model is needed to more realistically simulate the effect of lighting on a face. In the field of facial reconstruction technology, lighting models based on spherical harmonics (SH) are one of the most commonly used methods. When it is assumed that the light source is far from the face and that the reflectivity of the face surface conforms to the Lambertian emission model (i.e., ideal scattering conditions), a face image with shadow effects can be obtained.

[0105] (7)

[0106] in, This indicates the position of the pixel in the UV coordinate system. For pixels The final result of the lighting rendering at that location. Indicates reflectivity, The albedo value is determined by parameters related to the material. This represents the Hadamard product (i.e., element-wise multiplication). and The first The coefficients and basis coefficients of the spherical harmonic function of order 1. Represents pixels in the UV coordinate system The surface normal at a given location is summed over spherical harmonic function terms of different orders to simulate lighting effects.

[0107] In the detailed rendering process, a normal map is first generated by processing the displacement map to capture subtle geometric changes on the face surface. Then, the normal map is fused with the high-resolution face mesh generated by the CoMA model and the surface normal data to further generate a face geometry model containing more refined details, as shown below.

[0108] (8)

[0109] in, Represents a more refined geometric model - a finely detailed shape. This represents the detailed shape of the geometric model generated by the CoMA model. This indicates a displacement map. This represents the corresponding surface normal. This represents the Hadamard product. Normal information is calculated from the displacement map. Render using the corresponding normal map To obtain detailed rendering , means as follows

[0110] (9)

[0111] This represents one rendering process. After the final detail rendering is complete, the model's surface texture is applied to the detail rendering result, resulting in a 3D face model with accurate shape and high-frequency details, such as... Figure 7 As shown, this model exhibits an extremely high degree of realism and richness of detail.

[0112] 3D facial expression editing

[0113] After generating the source 3D face model, the 3D face expression editor allows users to input a target face image and a text description of the target face, thereby transferring the expression of the target face to the source 3D face model and enabling expression editing of the source 3D face. For example... Figure 8 As shown, this module consists of three stages: The first stage is the 2D facial expression transfer stage, which utilizes an identity feature-based denoising diffusion probability model and a face-guided sampling strategy to replace the expression of the target face with the expression of the source face while maintaining the identity features of the source face, thus obtaining the source face image after expression transfer. The second stage is the text enhancement stage, which introduces the CLIP text encoder to extract the description of the target face from the text and maps it together with the source face image after expression transfer into StyleGAN2. This allows the text description of the target face to supplement the detailed features in the source face image after expression transfer, resulting in a text-enhanced expression source face image. The third stage is the 3D face reconstruction stage, where the text-enhanced expression source face image is input into the 3D face reconstruction network in the 3D face generation module, reconstructing a 3D face model corresponding to the text-enhanced expression source face image, thus obtaining the source 3D face model after expression editing.

[0114] In the 2D facial expression transfer stage, firstly, an identity-feature-based conditional denoising diffusion probability model is introduced. This model uses source and target face images to provide facial guidance information, including facial features, expressions, and poses. This information helps the model maintain the source identity features when generating facial images. During sampling, an intermediate facial image—the expression-transferred source face image—is generated. This image integrates the source identity features and target attributes while retaining the background features of the source face image. This achieves expression transfer between 2D facial images by replacing the expression of the target face with the expression of the source face while preserving its identity features.

[0115] An identity-feature-conditional denoising diffusion probability model is introduced. By progressively adding noise to the data and controlling the magnitude and step size of the noise, the image is gradually converted into pure noise. Then, an inverse process is learned to generate data from the noise. First, identity information is injected into the identity-feature-conditional denoising diffusion probability model. This helps the model generate facial images with the corresponding identity based on the identity label during the generation process. Here, the source face image is used... Injected into identity embedder Obtain the source identity :

[0116] (10)

[0117] Then, the source identity and target face image Embedded into the identity feature-conditional denoising diffusion probability model Among them It is a picture of the target face. time step The noisy version at that point generates a realistic image from a standard Gaussian distribution by reversing the cyclic noise process. Data during the forward process... Gradually transforms into a Gaussian distribution The definition is as follows:

[0118] (11)

[0119] in, It is a predefined variance table. It is an identity matrix, representing the independence between its dimensions and having a variance of . The reversal process is as follows:

[0120] (12)

[0121] The reverse cyclic noise process iteratively removes noise and can generate a facial image with that identity from the noise based on the input identity label.

[0122] The purpose of guided facial sampling is to ensure that the model can accurately capture and fuse the identity information of the source face and the attribute information of the target face, thereby ensuring the generation of a new face image with the source identity and target attributes, i.e., the source face image after expression transfer, achieving a natural and realistic expression transfer effect.

[0123] Feature extraction is performed on the input source and target face images, including facial shape, texture, and color information, and encoded into matrix form. The face-guided sampling strategy uses these feature matrices to guide the model in sampling. During the sampling process, it is necessary to preserve the source face features to prevent identity drift, while also completely transferring the expression of the target face to the source face. Therefore, the following formula is used to constrain the ID vector of the source face image and the ID vector of the expression-transferred source face image, making them closer in the identity embedding space.

[0124] (13)

[0125] in, This metric represents identity similarity, used to measure the similarity in identity features between the source and generated faces during expression transfer. A value closer to 1 indicates better preservation of identity features. (Based on the source face image...) Input to identity embedder The identity information of the source face and the attribute information of the target face obtained from the image are compared with the source face image after expression transfer. The goal is to find a suitable balance point in the feature space that allows the generated image to retain the identity features of the source face while incorporating the attribute features of the target face, thus achieving the transfer of facial expressions from two-dimensional face images.

[0126] Text-enhanced migration source face image stage

[0127] Textual descriptions of the target face are used to supplement and enhance details in the source face image after expression transfer, enabling finer control over its expression. This process is similar to the previous text enhancement; specifically, the source face image after expression transfer is input into the GAN inversion module, mapping it into the GAN's latent space to obtain a latent vector. This latent vector is then input into the GAN decoder to obtain a latent code. Simultaneously, the text description of the target face input by the user is processed by the CLIP text encoder to obtain the text embedding, denoted as... .Will and The features are fed into the attention decoder and the StyleGAN2 model is used to generate text-enhanced facial images of the expression source.

[0128] 3D face reconstruction stage

[0129] By inputting the text-enhanced image of the source face with the desired expression into the 3D face reconstruction network during the 3D face generation process, a 3D face model corresponding to the text-enhanced image of the source face with the desired expression—the source 3D face model with edited expression—can be reconstructed. This 3D face model possesses both the appearance of the source face and the expression of the target face, enabling precise transfer of expression information from the target face image to the source 3D face model, thereby achieving expression editing of the source 3D face model. Figure 8 It demonstrates the effects of 3D facial expression editing.

[0130] Experiments related to 3D face generation

[0131] Datasets: The datasets used include VGGFace2, BUPT-Balancedface, and NoW. VGGFace2 contains images downloaded from Google Image Search, representing 9131 individuals with significant differences in pose, age, lighting, ethnicity, and occupation, supporting face matching across different poses and ages. BUPT-Balancedface includes multi-ethnic face images with varying skin tones, ensuring a relatively balanced number of images for each ethnicity. This helps evaluate the performance consistency of the face matching algorithm across different ethnic groups. The NoW (Not quite in-the-Wild) dataset contains 2054 photos of 100 participants, covering different age, BMI, and gender distributions (55 women, 45 men).

[0132] Training set: The VGGFACE2 dataset and BUPT-Balancedface dataset were preprocessed, and FAN (Face Alignment Network) was used to predict 68 2D facial landmarks. Facial segmentation techniques were used to obtain skin masks, and a model from VGGFace2-pytorch was used to calculate identity loss. This training set was used to train the proposed 3D face generation module.

[0133] Test set: The NoW dataset is used as the test set to measure the distance from all reference scan vertices to the nearest point on the reconstructed mesh surface.

[0134] Training strategy: The machine configuration and development environment used were: Ubuntu 20.04; GeForce RTX4090; CUDA 12.2; PyTorch 2.1.0; VS Code 1.91.1; Python 3.8.19. The model was trained for 30 epochs with a batch size of 32.

[0135] Comparative Experiment Results: The 3D face generation method in this paper is compared with other 3D face generation methods. The generative models of all methods are retrained using the same VGGFace2 and BUPT-Balancedface training sets and environment as the method in this paper. The purpose of this is to ensure that all methods are compared under the same conditions, effectively eliminating the influence that differences in training data or environment may have on the comparison results, so as to obtain fair results. Figure 9This paper presents the comparison results of our proposed method with other methods on the NOW evaluation dataset. The first row shows the input single source face image. The second to fourth rows show the source 3D face models generated using 3DDFA-V2, DECA, and our proposed method, respectively. The last row is the input text description of the source face, such as prompts about facial expressions. It is evident that our proposed method can more accurately capture the facial details in the input image, especially in reproducing facial expressions, and presents them with high realism and naturalness in the generated 3D face model.

[0136] Table 2. Reconstruction errors of the 3D face generation model in this paper and other models on the NoW dataset.

[0137]

[0138] Table 2 shows the reconstruction error of our proposed 3D face generation model compared to other single-modal input models on the NoW dataset. It can be seen that our proposed model outperforms the other models.

[0139] ablation experiment

[0140] Ablation experiments were conducted on the CLIP encoder and CoMA detail enhancement module in the 3D face generation module of this paper to demonstrate that these two components have a gain effect on the generation of the 3D face model. Table 3 shows the reconstruction results of the model after removing each module and the complete model on the NoW dataset. It can be seen that the complete model has the smallest reconstruction errors in all three categories.

[0141] Table 3 Ablation Experiment Results of the 3D Face Generation Module

[0142]

[0143]

[0144] Experiments related to 3D facial expression editing

[0145] Dataset: The FFHQ face dataset is used. The FFHQ (Flickr-Faces-High-Quality) dataset is a high-quality face image dataset containing 70,000 high-quality face images, covering a variety of facial attributes such as age, gender, race, skin color, expression, face shape, hairstyle, and facial pose, as well as various facial accessories such as glasses, sunglasses, and hats.

[0146] Training Strategy: The machine configuration and development environment used in the experiment were: Ubuntu 20.04; GeForce RTX 4090; CUDA 12.2; PyTorch 2.1.0; VS Code 1.91.1; Python 3.8.19. The model was trained for 30 epochs with a batch size of 32.

[0147] Comparative Experiment Results: The proposed 3D facial expression editing method was compared with other unimodal 3D facial expression editing methods. Using the same FFHQ dataset, all models were retrained under the same conditions, and the same 3D face generation method was used to generate the edited 3D face models. This was done to ensure that all methods were compared under the same conditions to obtain fair results. Figure 10 As shown, the first column is the source face image, the second column is the target face image, and the third column is the face image after expression transfer. It can be seen that for a wide-mouthed surprised expression, the unimodal models SimSwap and 3dSwap exhibit facial deformation and expression distortion after expression transfer when processing such extreme expressions, and identity drift also occurs. However, the method presented in this paper, by inputting the text prompt "surprised," not only restores the extreme expression but also maintains the original face's identity features, making the 3D face model after expression editing more realistic and natural.

[0148] ablation experiment

[0149] Ablation experiments were conducted on the CLIP encoder and the identity feature conditional denoising diffusion probability model (DDPM) in the 3D face expression editing module of this paper to demonstrate that these two components have a beneficial effect on the expression editing of the 3D face model. Table 4 shows the model after removing each module, as well as the reconstruction effect of the complete model on the NoW dataset. It can be seen that the complete model has the smallest reconstruction errors in all three categories.

[0150] Table 4 Ablation Experiment Results of the 3D Facial Expression Editing Module

[0151]

[0152] Example 2

[0153] This embodiment provides a text-image driven 3D face generation and expression editing system, including:

[0154] The text enhancement module is configured to extract key description information from the source face text description, generate a control vector based on the key description information and the source face image, and enhance the source face image using the control vector to obtain the text-enhanced source face image.

[0155] The 3D face reconstruction module is configured to extract facial model parameters based on the text-enhanced source face image, use the facial model parameters as a guide to generate a rough shape, and then perform detail enhancement and texture rendering on the rough shape to generate the final 3D face model.

[0156] The expression editing module is configured to extract the target expression from the target face image and transfer it to the source face image to generate the source face image after expression transfer. The source face image after expression transfer is enhanced by the target face text description to obtain the text-enhanced expression source face image. The text-enhanced expression source face image is then reconstructed in 3D to obtain the source 3D face model after expression editing.

[0157] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0158] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0159] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0160] Example 3

[0161] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the text image-driven 3D face generation and expression editing method described in Embodiment 1 above.

[0162] Example 4

[0163] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the text image-driven 3D face generation and expression editing method described in Embodiment 1 above.

[0164] Example 5

[0165] This embodiment provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the text-image-driven 3D face generation and expression editing method described in Embodiment 1.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0171] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A text-image driven 3D face generation and expression editing method, characterized in that, include: Key descriptive information is extracted from the source face text description. Based on the key descriptive information and the source face image, a control vector is generated. The control vector is then used to enhance the source face image, resulting in a text-enhanced source face image. The step of extracting key descriptive information from the source face text description, generating a control vector based on the key descriptive information and the source face image, and enhancing the source face image using the control vector includes: Random latent codes are generated based on random noise. At the same time, the source face image is inverted and co-encoded with the random latent codes to generate a latent code. The source face text description is encoded to extract key descriptive information, resulting in text embedding; Attention decoding is performed based on text embedding and latent encoding to generate control vectors; The control vector and latent encoding are input into the generator to obtain the text-enhanced source face image; Facial model parameters are extracted from the text-enhanced source face image. The facial model parameters are used as a guide to generate a rough shape. The rough shape is then enhanced with details and rendered with textures to generate the final 3D face model. The target expression is extracted from the target face image and transferred to the source face image to generate the source face image after expression transfer. The source face image after expression transfer is enhanced by the target face text description to obtain the text-enhanced expression source face image. This process is the same as the previous text enhancement. The text-enhanced expression source face image is then reconstructed in 3D to obtain the source 3D face model after expression editing. The step of extracting the target expression from the target face image and transferring it to the source face image to generate the expression-transferred source face image includes: Extract the source identity of the source face image using an identity embedder; After extracting features from the source and target face images, a facial feature matrix is ​​generated. Based on the source identity, the noisy target face image at the time step, and the time step, a denoising diffusion probability model with identity features is used to generate the source face image after expression transfer.

2. The text-image-driven 3D face generation and expression editing method as described in claim 1, characterized in that, The process involves extracting facial model parameters from the text-enhanced source face image and using these parameters as a guide to generate a rough shape. Specifically: The source face image after text enhancement is encoded using a visual encoder to obtain the source face encoding features; Decode the source face encoding features to obtain facial shape parameters, pose parameters, expression parameters, albedo parameters, illumination parameters, and camera parameters; The facial shape parameters, pose parameters, and expression parameters are input into the facial model to generate a rough shape.

3. The text-image-driven 3D face generation and expression editing method as described in claim 1, characterized in that, The process of enhancing the details of the rough shape and rendering textures to generate the final 3D face model specifically involves: The albedo parameters, lighting parameters, and camera parameters in the facial model parameters are input into the trained DECA decoder to obtain the displacement map and the model surface map. The rough shape is preprocessed and then encoded to extract global and local features of the rough shape; Decoding global and local features yields detailed shapes; Normal maps are generated based on displacement maps. The normal maps and detailed shapes are then rendered in detail. Finally, the model surface maps are applied to the detailed rendering results to obtain the final 3D human face model.

4. The text-image-driven 3D face generation and expression editing method as described in claim 3, characterized in that, The process involves generating a normal map based on a displacement map, rendering the normal map and detailed shape, and then applying the model surface map onto the rendered detail to obtain the final 3D face model. Specifically: Generating normal maps based on displacement maps; The product of the normal map and the displacement map is combined with the detailed shape to obtain a fine and detailed shape; The lighting rendering model is used to render the fine and detailed shapes in detail, and the detailed rendering result is obtained. The surface texture of the model is applied to the detailed rendering result to obtain the final 3D face model.

5. A text-image driven 3D face generation and expression editing system, characterized in that, The method for generating and editing 3D faces based on text images as described in any one of claims 1-4 includes: The text enhancement module is configured to extract key description information from the source face text description, generate a control vector based on the key description information and the source face image, and enhance the source face image using the control vector to obtain the text-enhanced source face image. The 3D face reconstruction module is configured to extract facial model parameters based on the text-enhanced source face image, use the facial model parameters as a guide to generate a rough shape, and then perform detail enhancement and texture rendering on the rough shape to generate the final 3D face model. The expression editing module is configured to extract the target expression from the target face image and transfer it to the source face image to generate the source face image after expression transfer. The source face image after expression transfer is enhanced by the target face text description to obtain the text-enhanced expression source face image. The text-enhanced expression source face image is then reconstructed in 3D to obtain the source 3D face model after expression editing.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the text image-driven 3D face generation and expression editing method as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the text image-driven 3D face generation and expression editing method as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps in the text-image-driven 3D face generation and expression editing method as described in any one of claims 1-4.

Images