A Text-Guided Optimized Method for Refined Expression Editing of 3D Avatars

Abstract

This invention provides a text-guided optimization method for refined facial expression editing of 3D avatars, comprising three core modules: a basic synthesis network module, a dual mapper module, and a text-guided optimization module. The dual mapper module, used for refined facial expression editing of 3D avatars, is essentially composed of a texture mapper and an emotion mapper, achieving collaborative and interactive optimization of the texture latent space and geometric expression space through a cross-attention mechanism. The text-guided optimization module utilizes a pre-trained CLIP model, using natural language text as a supervision signal, thereby achieving effective training of the dual mapper module without requiring a large-scale, finely labeled dataset. This method offers advantages such as improving the precision of refined facial expression control, addressing data scarcity while ensuring correct editing direction, and decoupling identity and expression while maintaining identity fidelity.

Description

Technical Field

[0001] This invention relates to avatar expression editing technology, and in particular to a text-guided optimization method for refined expression editing of 3D perceived avatars, belonging to the field of 3D face reconstruction and editing. Background Technology

[0002] Facial expression editing technology aims to manipulate the facial expressions of a given face image to match a target expression while preserving the person's identity information. This technology has broad application prospects in fields such as virtual avatars, human-computer interaction, psychological analysis, and augmented reality.

[0003] With the development of deep learning, generative techniques such as Generative Adversarial Networks (GANs) and diffusion models have greatly promoted the development of facial attribute editing. Early methods mainly relied on two-dimensional image transformation techniques to achieve expression editing by learning the mapping between different expression domains or manipulating conditional labels in the latent space. However, these methods ignored the three-dimensional geometric characteristics of the human face and struggled to handle pose changes while ensuring three-dimensional consistency.

[0004] To overcome these limitations, subsequent research began combining 3D deformable face models (3DMMs) with generative models, thereby achieving more precise control over 3D factors such as expressions and poses. Despite these advancements, ensuring visual accuracy remains a challenge. Figure 1 The ability to express emotions and handle subtle changes in facial expressions remains an unresolved problem.

[0005] In recent years, 3D perception GAN methods, such as AniFaceGAN and Next3D, have successfully generated high-quality visual images by combining neural radiation fields (NeRF) and mesh-guided deformation techniques. Figure 1 These methods produce RGB images. However, the 3DMM models and deformation fields upon which these methods rely suffer from insufficient constraints, leading to challenges in fine-grained control and resulting in low-quality output. While advanced models such as Next3D can separate dynamic and static head components, their facial expression control capabilities are limited by the inherent limitations of estimation methods such as FLAME and DECA.

[0006] To address the scarcity of detailed facial expression data, researchers have begun leveraging the multimodal capabilities of large-scale image-text pre-trained models (such as CLIP). By using natural language descriptions (e.g., "a person raising an eyebrow") as supervision signals, the dependence on large labeled datasets can be bypassed, guiding the model to generate high-quality 3D avatars with accurate expressions. This invention, against this backdrop, aims to address the shortcomings of existing 3D perception models in detailed facial expression control by proposing a novel optimization framework. Summary of the Invention

[0007] This invention discloses a text-guided optimization method for refined facial expression editing of 3D avatars, comprising four core aspects. First, a dual mappers module for refined facial expression editing of 3D avatars, its core comprising a texture mapper and an emotion mapper, achieving collaborative and interactive optimization of the texture latent space and geometric expression space through a cross-attention mechanism. Second, a text-guided optimization training method utilizing a pre-trained CLIP model with natural language text as a supervision signal, thereby achieving effective training of the dual mappers module without requiring a large-scale refined annotation dataset. Third, a subspace projection mechanism for improving editing accuracy and identity preservation, constraining text-guided editing within a predefined expression feature subspace, effectively avoiding unintended modifications to irrelevant attributes such as identity, and achieving decoupling of expression from identity. Fourth, a strategy combining an EMOCA model for initial facial expression parameter extraction, which, compared to commonly used models such as DECA in existing technologies, can more accurately capture fine-grained facial expression features in the input image, providing higher-quality initial conditions for subsequent refined editing. This is to improve the precision of fine-grained facial expression control, solve the problem of data scarcity and ensure the correct editing direction, and decouple identity from facial expressions while ensuring identity fidelity.

[0008] This invention discloses a text-guided optimization-based method for refining facial expressions in 3D avatars, comprising three core modules: a basic synthesis network module, a dual mappers module, and a text-guided optimization module.

[0009] Basic Synthesis Network module:

[0010] The image synthesis network uses a pre-trained Next3D model as its foundation. The core process of this network is as follows: First, a mapping network maps randomly sampled noise z to intermediate hidden codes w. Then, a StyleGAN2-type generator G... uv Generate UV texture feature map F using w uv Meanwhile, this invention employs the EMOCA model (instead of Next3D's original DECA) to extract more accurate FLAME expression parameters α, identity parameters θ, and pose parameters β from the input reference image, generating a 3D face mesh M. Using rasterization technology, the front, side, and top views of the 3D mesh M are rendered onto the UV texture feature map F. uvThe process involves generating neural texture tri-planes. Finally, volume rendering and a super-resolution module are used to generate the final high-resolution RGB image.

[0011] Dual Mappers module:

[0012] To achieve fine-grained editing of facial expressions, a dual mapper module was designed. Its goal is to collaboratively predict the variations in texture latent coding *w* and the facial expression parameter *α*. This module contains a texture mapper (M... T ) and an emotion mapper (M E It interacts using a cross attention mechanism.

[0013] Specifically, in order to predict the transform amount Δw of texture encoding, the model uses the expression parameter α R As a query, the original texture implicit code w is used as the key and value. The update information is obtained through cross-attention calculation, and then passed through the texture mapper M. T We obtain Δw. Conversely, using w as the query, we can similarly obtain the change Δα of the facial expression parameters. This interactive mechanism allows the textural and geometric editing processes to be mutually aware and guided. To ensure training stability, this module does not directly predict the edited parameters, but rather predicts their changes (Δw, Δα). The final parameters are obtained by adding the original values ​​to the changes (w' = w + Δw). Furthermore, an L2 loss function is used... This constrains the magnitude of changes, preventing edited parameters from deviating too far from their initial values, thereby avoiding the introduction of artifacts or changes to other irrelevant properties.

[0014] Text-Guided Optimization module:

[0015] To accurately guide the editing process using natural language, this invention designs a text-guided optimization module. This module utilizes a pre-trained CLIP model to process the edited image I... E The target text description is encoded into image features e. I ′ and text features e T To address the attribute entanglement problem that may result from directly comparing image and text features, this invention introduces a subspace projection mechanism.

[0016] The mechanism first constructs an "emoji subspace" defined by multiple basic emoji texts, and then projects the features of the initial image into this subspace, so that subsequent editing operations are mainly focused on the dimensions related to the emoji, while features unrelated to identity are retained in the residual vector.

[0017] Specifically, since our primary target attribute for editing is facial expression, we select some text descriptions related to facial expressions as basis vectors b. k k = 1, ..., N are used to construct subspaces. Then, we embed the initial image into e I Projected to Above, while retaining the residual vector r, its form is:

[0018]

[0019] in Indicates to The projection operation, e P This is the projected embedding vector. Next, we enhance the text cue t on the projected embedding e. P The influence of this is then considered, and the residual vector r is subsequently reintroduced to form the final embedding:

[0020]

[0021] in Indicates in e T Under guidance, for e P The enhancement operation performed, where γ is the enhancement intensity, satisfies... Its principle is to weaken e P In and e T Irrelevant attributes, thus enhancing the effect of e T The attributes they represent have an impact. Specifically, Defined as:

[0022]

[0023] Where c k =e P T b k ,d k =e T T b k And b k It is a set of basis vectors, each vector describing an expression.

[0024] In conclusion, in placing e I Projected onto the emoji subspace Subsequently, text-guided facial editing focuses only on attributes related to facial expressions, thus ensuring that other attributes remain unchanged. This is achieved by optimizing the final embedding. T ′ with edited image embedding e I By using the cosine similarity between ' and ', fine-grained text-based emoji editing can be achieved.

[0025] (1) CLIP loss Calculate the cosine similarity between the edited image features and the target text features after subspace projection and enhancement, so that the generated emoji is aligned with the text description.

[0026]

[0027] (2) Loss of identity A pre-trained face recognition network (ArcFace) is used to calculate the similarity of identity features between the original image and the edited image, ensuring that the identity of the person remains unchanged.

[0028]

[0029] (3) Mapper loss This refers to the aforementioned L2 constraint loss.

[0030] The final total loss function is the weighted sum of these three terms:

[0031]

[0032] This invention discloses a text-guided optimization method for refining facial expressions in 3D avatars, specifically comprising the following steps:

[0033] S1: Data input, input a reference face image I R A text description of the target expression.

[0034] S2: Initial parameter extraction, using a frozen EMOCA encoder from reference image I R Extracting the facial expression parameter α from the FLAME model R And the identity / pose parameters (θ, β). Simultaneously, an initial hidden code w is randomly sampled from the latent space of StyleGAN.

[0035] S3: Parameter co-optimization, adjusting the initial facial expression parameter α R The expression parameters and texture latent code w are fed into the dual mapper module. Through cross-attention and their respective mapper networks, the changes in expression parameters Δα and texture encoding Δw are calculated, yielding the optimized parameter α. R '=α R +Δα and w'=w+Δw.

[0036] S4: Image synthesis, using the optimized parameter α R 'and w', along with fixed identity and pose parameters, are input into a frozen synthesis network to generate an image I with edited facial expressions. E .

[0037] S5: Multi-target loss calculation, generating image I E Initial Image I I (based on initial parameters w and α) R The generated text and the target text are fed into the text guidance optimization module. CLIP loss, identity loss, and mapper L2 loss are calculated respectively.

[0038] S6: Model optimization, based on the calculated total loss. The backpropagation algorithm is used to update only the parameters of the dual mapper module. Throughout the process, the pre-trained modules such as the synthesis network, EMOCA, and CLIP encoder remain frozen.

[0039] S7: Iterative training, repeating steps S3 to S6, training each expression category separately until the model converges, obtaining a dual mapper capable of precise expression editing based on text prompts.

[0040] Through the above technical solutions, this invention has the following advantages compared with existing three-dimensional perceptual facial editing technologies:

[0041] (1) Higher editing precision and realism: Through an innovative dual mapper module and EMOCA parameter extractor, this invention can capture and generate extremely subtle facial expression changes, such as the subtle twitching of the muscles at the corners of the mouth and the fine movements of the eyebrows. Compared with methods such as Next3D and Diffusion-rig, the expressions generated by this method are not only topologically correct (e.g., successfully closing the eyes), but also closer in detail to the reference image and target text description, greatly improving the realism and accuracy of editing.

[0042] (2) Enhanced attribute decoupling and identity preservation capabilities: The constructed subspace projection mechanism effectively solves the "attribute entanglement" problem that is common in text-guided editing. It ensures that editing operations are applied precisely to expressions, without accidentally changing the character's clothing or background as in methods such as StyleCLIP, or causing a shift in identity features as in some baseline methods, thus achieving high-quality identity preservation.

[0043] (3) Excellent 3D consistency: As a framework based entirely on 3D perception, the facial images generated by this invention can maintain good 3D consistency under any new viewpoint. This is something that all 2D editing methods (such as DeltaEdit and StyleCLIP) cannot achieve, and it is also superior to 3D methods that suffer from view inconsistency due to a lack of multi-view training.

[0044] (4) Superior data efficiency and robustness: This invention cleverly utilizes the knowledge of large pre-trained models through text-guided optimization, without relying on any training dataset with finely labeled facial expressions, greatly reducing the application threshold. At the same time, this method is robust when dealing with challenging facial expressions with large topological changes, such as "closed eyes," demonstrating stronger generalization ability than methods such as DiscoFaceGAN and AniFaceGAN. Attached Figure Description

[0045] Figure 1 Overall architecture diagram of a text-guided optimization method for refined facial expression editing of 3D-perceptive avatars.

[0046] Figure 2 Structure diagram of the basic synthesis network module.

[0047] Figure 3 A flowchart of a text-guided optimization method for refining facial expressions in 3D-perceptive avatars. Detailed Implementation

[0048] The invention will now be described in more detail with reference to the accompanying drawings.

[0049] This invention discloses a text-guided optimization-based method for refined facial expression editing in 3D avatar perception, comprising three core modules: a basic synthesis network module, a dual mappers module, and a text-guided optimization module. Please refer to... Figure 1 This is an overall architecture diagram of the method of the present invention, wherein... Figure 1 (a) is the Dual Mappers module. Figure 1 (b) is the Text-Guided Optimization module. Please refer to [link / reference]. Figure 2 It is the basic synthesis network module.

[0050] Please refer to Figure 2The Synthesis Network module uses a pre-trained Next3D model as the base image synthesis network. The core process of this network is as follows: First, a mapping network maps randomly sampled noise z to intermediate hidden codes w. Then, a StyleGAN2-type generator G... uv Generate UV texture feature map F using w uv Meanwhile, this invention employs the EMOCA model (instead of Next3D's original DECA) to extract more accurate FLAME expression parameters α, identity parameters θ, and pose parameters β from the input reference image, generating a 3D face mesh M. Using rasterization technology, the front, side, and top views of the 3D mesh M are rendered onto the UV texture feature map F. uv The process involves generating neural texture tri-planes. Finally, volume rendering and a super-resolution module are used to generate the final high-resolution RGB image.

[0051] Please refer to Figure 1 The Dual Mappers module (a) is designed to enable fine-grained editing of facial expressions. Its goal is to collaboratively predict the changes in texture latent coding (w) and facial expression parameters (α). This module contains a texture mapper (M... T ) and an emotion mapper (M E It interacts using the CrossAttention mechanism.

[0052] Specifically, in order to predict the transform amount Δw of texture encoding, the model uses the expression parameter α R As a query, the original texture implicit code w is used as the key and value. The update information is obtained through cross-attention calculation, and then passed through the texture mapper M. T We obtain Δw. Conversely, using w as the query, we can similarly obtain the change Δα of the facial expression parameters. This interactive mechanism allows the textural and geometric editing processes to be mutually aware and guided. To ensure training stability, this module does not directly predict the edited parameters, but rather predicts their changes (Δw, Δα). The final parameters are obtained by adding the original values ​​to the changes (w' = w + Δw). Furthermore, an L2 loss function is used... This constrains the magnitude of changes, preventing edited parameters from deviating too far from their initial values, thereby avoiding the introduction of artifacts or changes to other irrelevant properties.

[0053] Please refer to Figure 1(b) shows the Text-Guided Optimization module, which utilizes a pre-trained CLIP model to optimize the edited image I. E The target text description is encoded into image features e. I ′ and text features e T To address the attribute entanglement problem that may result from directly comparing image and text features, this invention introduces a subspace projection mechanism.

[0054] The mechanism first constructs an "expression subspace" defined by multiple basic expression texts (such as "happy", "sad", etc.), and then projects the features of the initial image into this subspace, so that subsequent editing operations are mainly focused on the expression-related dimensions, while features unrelated to identity are retained in the residual vector.

[0055] Specifically, since our primary target attribute for editing is facial expression, we select some textual descriptions related to expression (e.g., "a person raising their eyebrows") as the basis vector b. k k = 1, ..., N are used to construct subspaces. Then, we embed the initial image into e I Projected to Above, while retaining the residual vector r, its form is:

[0056]

[0057] in Indicates to The projection operation, e P This is the projected embedding vector. Next, we enhance the text cue t on the projected embedding e. P The influence of this is then considered, and the residual vector r is subsequently reintroduced to form the final embedding:

[0058]

[0059] in Indicates in e T Under guidance, for e P The enhancement operation performed, where γ is the enhancement intensity, satisfies... Its principle is to weaken e P In and e T Irrelevant attributes, thus enhancing the effect of e T The attributes they represent have an impact. Specifically, Defined as:

[0060]

[0061] Where c k =e P T b k ,d k =e T T b k And b k It is a set of basis vectors, each vector describing an expression.

[0062] In conclusion, in placing e I Projected onto the emoji subspace Subsequently, text-guided facial editing focuses only on attributes related to facial expressions, thus ensuring that other attributes remain unchanged. This is achieved by optimizing the final embedding. T ′ with edited image embedding e I By using the cosine similarity between ' and ', fine-grained text-based emoji editing can be achieved.

[0063] (1) CLIP loss Calculate the cosine similarity between the edited image features and the target text features after subspace projection and enhancement, so that the generated emoji is aligned with the text description.

[0064]

[0065] (2) Loss of identity A pre-trained face recognition network (ArcFace) is used to calculate the similarity of identity features between the original image and the edited image, ensuring that the identity of the person remains unchanged.

[0066]

[0067] (3) Mapper loss This refers to the aforementioned L2 constraint loss.

[0068] The final total loss function is the weighted sum of these three terms:

[0069]

[0070] Please refer to Figure 3 This is a flowchart of a text-guided optimization method for refining facial expressions in 3D-perceptive avatars. The method of this invention specifically includes the following steps:

[0071] S1: Data input, input a reference face image I R And a text description of the target expression (e.g., "an angry person").

[0072] S2: Initial parameter extraction, using a frozen EMOCA encoder from reference image I RExtracting the facial expression parameter α from the FLAME model R And the identity / pose parameters (θ, β). Simultaneously, an initial hidden code w is randomly sampled from the latent space of StyleGAN.

[0073] S3: Parameter co-optimization, adjusting the initial facial expression parameter α R The expression parameters and texture latent code w are fed into the dual mapper module. Through cross-attention and their respective mapper networks, the changes in expression parameters Δα and texture encoding Δw are calculated, yielding the optimized parameter α. R '=α R +Δα and w'=w+Δw.

[0074] S4: Image synthesis, using the optimized parameter α R 'and w', along with fixed identity and pose parameters, are input into a frozen synthesis network to generate an image I with edited facial expressions. E .

[0075] S5: Multi-target loss calculation, generating image I E Initial Image I I (based on initial parameters w and α) R The generated text and the target text are fed into the text guidance optimization module. CLIP loss, identity loss, and mapper L2 loss are calculated respectively.

[0076] S6: Model optimization, based on the calculated total loss. The backpropagation algorithm is used to update only the parameters of the dual mapper module. Throughout the process, the pre-trained modules such as the synthesis network, EMOCA, and CLIP encoder remain frozen.

[0077] S7: Iterative training, repeating steps S3 to S6, training each expression category separately until the model converges, obtaining a dual mapper capable of precise expression editing based on text prompts.

[0078] To illustrate the technical advancement of the method of this invention, the following four core evaluation indicators are used to compare the quantitative performance of the facial expression editing task:

[0079] AUAcc (Action Unit Accuracy)↑: Facial Action Units (AUs) are the basic muscle movements that constitute human facial expressions (such as raising the corners of the mouth, frowning, etc.). This metric measures the accuracy of the generated expression by detecting AUs in the generated image and comparing them with the AUs that the target expression should possess. The higher the score, the more closely the generated expression matches the muscle movement characteristics of the target expression, i.e., the more accurate the expression.

[0080] CLIP Score↑: This is a metric that measures the semantic similarity between an image and its text description. It uses the CLIP model to calculate how well the generated image matches the given text prompt (such as "a smiling person"). The higher the score, the more consistent the generated image content is with the text description.

[0081] FID (Fréchet Inception Distance): This is one of the standards for evaluating the image quality of generative models. It measures the realism (fidelity) and diversity of generated images by comparing the distribution distance between the generated image set and the real image set in the feature space. The lower the score, the higher the quality of the generated images and the more difficult it is to distinguish them from real images.

[0082] KID (Kernel Inception Distance): Similar to FID, KID is also a metric that measures the difference in distribution between generated and real images, but it is usually more stable on small sample data. A lower score generally indicates higher image quality.

[0083] Table 1. Quantitative performance comparison of the emoji editing task

[0084] index DiscoFaceGAN AniFaceGAN Next3D Diffusion-rig DeltaEdit StyleCLIP Ours (This Invention) AUAcc↑ 0.45 0.49 0.47 0.55 N / A N / A 0.6 CLIPScore↑ 23.9 24.69 24.62 25.15 23.73 24.64 25.28

[0085] As shown in Table 1, the quantitative evaluation results demonstrate that the method of this invention exhibits significant performance advantages in the facial expression editing task. Regarding facial expression editing accuracy, measured by the AUAcc metric, this method achieved the highest score of 0.60, surpassing other current state-of-the-art 3D baseline models. This result validates the effectiveness of the model architecture's use of EMOCA for initial parameter extraction and the refined 3D mesh geometry control through the "Emotion Mapper." In terms of semantic consistency, evaluated using the CLIP Score, this method scored 25.28, outperforming not only similar 3D methods but also specialized 2D text-guided editing models. This performance improvement is primarily attributed to the novel "text-guided optimization" framework, particularly the "subspace projection" mechanism, which effectively constrains editing within the expression-related semantic subspace, thereby avoiding unintended perturbations to irrelevant attributes. In summary, this method, with its "dual-mapper" architecture, achieves synergistic optimization of 3D geometry (accuracy) and texture appearance (semantics), ensuring high-fidelity physical simulation while achieving a high degree of semantic alignment with text instructions.

[0086] Table 2 Visual quality assessment of generated images

[0087] index StyleCLIP DeltaEdit DiscoFaceGAN AniFaceGAN Next3D Diffusion-rig Ours (This Invention) FID↓ 35.53 45.78 58.95 51 32.95 31.26 30.71 <![CDATA[KID(×10 -3 )↓]]> 21 24 40 35 19 24 18

[0088] As shown in Table 2, in the quantitative evaluation of image visual quality, the method of this invention achieved the best scores in both FID (score 30.71) and KID (score 0.018), indicating that its generated results outperform other state-of-the-art baseline models in terms of realism and diversity. This demonstrates that the proposed facial expression editing framework can integrate complex control functions without sacrificing the high fidelity of the underlying generative model (Next3D). This invention integrates multiple regularization constraints, including L2 loss to limit the offset of editing parameters and Identity Loss to maintain the consistency of character features. These mechanisms work together to effectively prevent the introduction of visual artifacts or identity shifts during the editing process, thereby ensuring the high quality and high fidelity of the final generated image.

Claims

1. A three-dimensional perception avatar fine expression editing method based on text-guided optimization, characterized in that, Comprising the following steps: S1: data input, input a reference face image and a text description of a target expression; S2: Initial parameter extraction, extract initial expression parameters of FLAME model from the reference image using the frozen EMOCA encoder ;​​​​ S3: parameter collaborative optimization, send initial expression parameters and initial texture latent encoding into a double mapper module, which contains a texture mapper and an emotion mapper , in order to predict the transformation amount of texture encoding , the model takes initial expression parameters as the query, initial texture latent encoding as the key and value, obtains updated information through cross attention calculation, and then obtains through the texture mapper , conversely, taking as the query, the transformation amount of the expression parameters can be obtained in the same way , and then the optimized parameters are obtained​ S4: image synthesis, the optimized parameters and together with fixed identity and pose parameters, are input into the frozen synthesis network to generate an expression edited image ; S5: multi-objective loss calculation, the generated image , the initial image and the target text are input into the text-guided optimization module, and the CLIP loss, the identity loss and the mapper loss are calculated respectively, wherein the initial image is generated by the initial parameters and ; S6: model optimization, according to the calculated total loss Through the back propagation algorithm, only the parameters of the double mapper module are updated, and the entire process keeps the synthetic network, EMOCA and CLIP encoder pre-training modules in a frozen state. S7: Iterative training, repeating steps S3 to S6, training each expression category respectively, until the model converges, obtaining a double mapper capable of accurate expression editing according to the text prompt.

2. The method of claim 1, wherein, The total loss in the step S6 is the CLIP loss , the identity loss and the mapper loss The weighted sum of the three losses: 。

Images