A training method for expression editing model based on deep learning

Abstract

The present invention discloses a training method for an expression editing model based on deep learning, and relates to the technical field of expression editing. The present invention aims to provide a convenient and fast expression editing method, wherein a user only needs to upload a source image, i.e., a photo of a person whose expression needs to be edited, and provide a target expression image or select from preset expression templates. The efficiency and convenience of the present invention significantly improve the editing speed and reduce labor costs, so that even if the user does not have professional image editing skills, the user can quickly obtain satisfactory editing results through simple operations. The present invention is not only suitable for photo editing on personal social media to improve the attractiveness of shared photos, but is also suitable for professional photography, advertising production and other fields, and provides high-quality expression editing services for professional users.

Description

Technical Field

[0001] The present invention relates to the technical field of expression editing, and in particular to a training method for an expression editing model based on deep learning. Background Art

[0002] In today's society, photography is not only an art form, but also a universal means of recording and sharing personal moments. With the rapid development of smartphones and camera technology, people can capture beautiful moments in life anytime and anywhere. The expression of the characters is particularly important because it can convey the emotions and stories behind the photos. However, due to insufficient light, stiff expressions or unnatural moments, the photos taken may not always reflect the emotions we want. At this time, expression editing technology is particularly important. Traditional expression editing relies on photo editing software such as Adobe Photoshop, which allows users to manually adjust the expressions in photos. Users can fine-tune the curvature of the eyebrows, the arc of the lips, and even change the brightness of the eyes, but this method has obvious limitations: it is not only time-consuming, but also requires users to have certain image editing skills, which may be a challenge for non-professionals.

[0003] With the rise of artificial intelligence and deep learning technology, the field of expression editing has ushered in revolutionary changes. Deep learning-based methods can automatically identify facial features and perform more sophisticated and automated expression adjustments. Most existing methods use a two-stage method based on generative adversarial networks (GANs) for expression editing. However, due to the limited expressive power of GANs, it is easy to cause partial blurring of the face. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a training method for an expression editing model based on deep learning to solve the problems of the existing expression editing model, such as complex editing process, unnatural expression generation, difficulty in maintaining identity features, insufficient expression diversity, unstable generation quality and low operating efficiency.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an expression editing model based on deep learning, which comprises:

[0008] The image processing module is responsible for preprocessing the face video, extracting key points, and generating binary masks of facial features;

[0009] The expression generation module includes an appearance reference network, a posture reference network, and an expression generation network. During training, the source image and the target expression image are photos of the same person, but they do not need to be photos of the same person during the inference stage.

[0010] The appearance reference network is trained based on the latent space diffusion model StableDiffusion and is used to maintain the details of the face. StableDiffusion is a classic latent space diffusion model (LDM, LatentDiffusionModel). LDM uses variational autoencoder VAE. Given an input image, the image is first mapped to the latent space through the encoder, so that diffusion is carried out in the latent space. In the forward process of diffusion, Gaussian noise is gradually added to the latent variable through T iterations. The reverse diffusion process is the inverse process of the forward process. In order to gradually restore the noise to the initial latent variable, it is finally restored to the RGB space through the decoder to obtain the final image.

[0011] Pose reference network: The pose reference network is a convolutional network used to extract the pose of facial expressions. Referring to the design similar to ControlNet, pose reference encodings of different scales are obtained to correspond to the input scales of different modules in the expression image generation diffusion model. At the same time, pose images of the source image and the target expression image are input. The pose reference network promotes the interaction between the source image and the target expression image. This process provides additional clues for the network to understand the correlation between facial key points and appearance, thereby helping to generate images with more accurate expressions.

[0012] Expression generation network, which is trained using the pre-trained StableDiffusion model and is used to generate the corresponding target expression images;

[0013] The cross-attention module combines features from the expression generation network and the appearance reference network using cross-network fusion. After each layer or block is processed, the generated image retains the global features of the source image (such as color, facial structure, etc.) while achieving the change of the target expression. The model can generate expressions without destroying facial details.

[0014] The self-attention module automatically adjusts the focus based on the local and global features of the image. The model will automatically identify the key parts of the image (such as eyes, mouth and other key facial areas) and pay more attention to these areas during the expression generation process, thereby improving the naturalness of expression generation and detail retention.

[0015] Furthermore, it also includes: an expression alignment module, which, during the expression editing process, keeps the positions of facial features and facial postures of the source image and the target expression image consistent, so that the generated target expression is natural and highly matched with the source image.

[0016] In a second aspect, the present invention provides a training method for an expression editing model based on deep learning, comprising:

[0017] Step S1, preprocessing the face video, cropping the face video with the face as the center to obtain a 512×512 face video;

[0018] Step S2, key point extraction, use Mediapipe to extract 478-dimensional 3D facial key points from the facial features of each frame of the face video, draw the inner and outer contours of the facial features including eyebrows, eye contours, pupils and lips, guide the generation of new expressions, extract the positions of the inner and outer contours of the facial features, generate a binary mask of the facial features based on the extracted inner and outer contour positions, expand the facial features, adjust the size to the size required by the latent space, and obtain the expanded facial features binary mask I m , used to calculate the facial features loss function during training;

[0019] Step S3, randomly select two pictures from the video of the same person, one as the source picture I source , and another as the target expression picture I targe ;

[0020] Step S4: source image I source Input into the appearance reference network and CLIP image encoder. The CLIP image encoder retains the overall color and structure of the person, while the appearance reference network can retain the details of the face.

[0021] Step S5, set source image I source and target expression picture I targe The posture diagrams are and The posture diagram Input into the posture reference network to obtain the posture code z p As expression gesture control;

[0022] Step S6: Encode the posture z p Add to the randomly generated hidden noise and input to the expression generation network;

[0023] Step S7, the appearance details extracted by the appearance reference network and the CLIP image encoder are input into the expression generation network through the cross attention module, and the target expression image is obtained through cyclic denoising.

[0024] Furthermore, in the appearance reference network, the training steps include:

[0025] Source Image I source Input to the pre-trained CLIP image encoder ε c In the formula, it is expressed as: zc =ε c (I source ), z c Represents the pre-trained CLIP image encoder ε c Source Image I source The encoding vector obtained after encoding is z c Input into the expression generation network through the cross attention module;

[0026] Source Image I source Input into the pre-trained VAE encoder ε, it becomes the latent space encoding z = ε (I source ), input z into the appearance reference network τ ref The facial detail features are extracted from the network and input into the expression generation network through the cross attention module.

[0027] Furthermore, in the appearance reference network, the training step also includes:

[0028] Perform feature fusion and transform the expression generation network ∈ θ and the appearance reference network τ ref The key and value of each attention module in are combined, and the subscripts θ and ref represent the expression generation network and the appearance reference network respectively. The attention fusion form is: Among them, Q θ represents the query vector of the expression generation network, K θ represents the key vector of the expression generation network, that is, the encoding of the input features, K ref represents the key vector of the appearance reference network, corresponding to the features in the appearance reference network, V θ Represents the value vector of the expression generation network, that is, the feature value extracted from the network, V ref Represents the value vector of the appearance reference network, that is, the feature value extracted from the appearance reference network, d k Represents the dimension of the key vector, which is used to normalize the dot product result, cat(K θ ,K ref ) and cat(V θ ,V ref ), cat represents the concatenation operation, cat(K θ ,K ref ) concatenates the key vectors of the expression generation network and the appearance reference network, cat(V θ ,V ref ) is the concatenation of value vectors, T represents the transpose operation, and the softmax function converts the result of the dot product into a probability distribution; thereby effectively combining the features of the expression generation network with the features of the appearance reference network, so that the generated image can change the expression while maintaining the details and consistency of the face.

[0029] Furthermore, in the posture reference network, the training steps include:

[0030] The pose graph of the source image and the pose graph of the target expression image are combined Input to the pose reference network τ p middle;

[0031] Pose Reference Network τ p The posture code obtained It is directly added to the randomly generated 4-channel latent space Gaussian noise and input into the expression generation network together.

[0032] Furthermore, in the expression generation network, the training steps include:

[0033] Generate random Gaussian noise in latent space Plus the pose code z extracted by the pose reference network p , we get ∈ p ,∈ p represents the pose encoding z generated by randomly generated Gaussian noise and the pose reference network p The combined latent space input;

[0034] Will ∈ p The facial appearance features extracted by CLIP are fused with the features of the expression generation network. At the same time, the features extracted by the appearance reference network are fused with the features of the expression generation network. θ and the appearance reference network τ ref The key and value of each attention module are merged, the feature encoding of the expression generation network output is finally passed through the VAE decoder to obtain the target expression map.

[0035] Furthermore, in the expression generation network, the training steps also include:

[0036] The loss function is minimized and the Adam optimizer is used for optimization iteration. The Loss function of the diffusion model is:

[0037] Among them, τ p ,τ ref ,∈ θ They are the pose reference network, appearance reference network and expression generation network, which are the trainable parts of the model, ε c It is a pre-trained CLIP image encoder and does not participate in training. represents the expected value operator, is the diffusion loss function;

[0038] The loss function is the real target expression picture I tgt And the generated expression picture I gen The perceptual loss between vgg is the pre-trained VGG19 network, l is the l-th layer output of the VGG19 network, The loss function is expressed as:

[0039] The facial features loss function is introduced, and the facial features binary mask I obtained by the image processing module is used m , the facial features loss function is:

[0040] in For the expression generation network ∈ θ The predicted latent space code, z is the latent space code of the target expression image;

[0041] The overall loss function is:

[0042] In a third aspect, the present invention provides an inference method for an expression editing model based on deep learning, comprising:

[0043] The model structure of the inference process of the expression editing model is similar to that of the training process, with the addition of an expression alignment module;

[0044] The reasoning process for the target expression is:

[0045] User input source image;

[0046] The source image and the target expression image are processed by an image processing module to extract a posture graph of the source image and a posture graph of the target expression image;

[0047] The pose graph of the source image and the pose graph of the target expression image are passed through the expression alignment module to obtain the pose graph of the source image and the aligned pose graph of the target expression. The 3D facial key points of the pose graph of the source image and the pose graph of the target expression image are extracted using Mediapipe, and the key points are projected onto a 2D plane to obtain the facial key points at the corresponding image position.

[0048] Use the relative relationship between the facial key points projected onto the 2D plane and the 3D facial key points to obtain the 3D key points of the pupil;

[0049] Calculate the relative relationship between the facial key points projected onto the 2D plane and the 3D facial key points, and apply the transformation matrix and camera matrix of the relative standard space in Mediapipe to obtain the target expression posture graph after projecting the target expression from the 3D space to the 2D plane of the source image, that is, the aligned target expression posture graph.

[0050] Further, the source image is input into the appearance reference network and the CLIP image encoder;

[0051] The pose graph of the source image and the aligned target expression pose graph are input into the pose reference network to obtain the pose encoding as the expression pose control;

[0052] The posture code is added to the randomly generated hidden noise and input into the expression generation network. The appearance details extracted by the appearance reference network and the CLIP image encoder are input into the expression generation network through the cross attention module. After cyclic denoising, the target expression image is obtained.

[0053] The beneficial effects of the present invention are:

[0054] The present invention uses a diffusion model to gradually generate and denoise expressions in latent space, so that the generated expression images have higher detail retention capabilities. Combined with the appearance reference network and the CLIP image encoder, the model can retain the facial details and overall structure of the character while generating expression changes. Compared with traditional GAN, the phenomenon of facial detail blurring and expression deformation is significantly reduced, and the generated results are more realistic and natural.

[0055] The present invention introduces a posture reference network to provide accurate posture control for expression generation, so that users can specify specific expression targets more flexibly.

[0056] The present invention introduces an appearance reference network, which combines the diffusion mechanism of the Stable Diffusion model and can maintain the identity information in the input image during the editing process. Traditional methods often cause facial features (such as face shape, facial features ratio, etc.) to shift during expression changes, causing the generated image to lose its original identity features. The facial details extracted by the appearance reference network not only conform to the expected expression changes, but also retain the facial features of the original person.

[0057] The present invention aims to provide a convenient and fast expression editing method. Users only need to upload source pictures, that is, pictures of people whose expressions need to be edited, and provide target expression pictures or select from preset expression templates. The efficiency and convenience of the present invention significantly improve the editing speed and reduce labor costs, so that even if users do not have professional image editing skills, they can quickly obtain satisfactory editing results through simple operations. The present invention is not only suitable for photo editing on personal social media to improve the attractiveness of shared photos, but is also suitable for professional photography, advertising production and other fields to provide high-quality expression editing services for professional users. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0059] Figure 1 A training diagram of the expression editing model of the present invention;

[0060] Figure 2 It is a reasoning diagram of the expression editing model of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0064] Example 1, reference Figure 1 and Figure 2 , this embodiment provides an expression editing model based on deep learning, the model comprising:

[0065] The image processing module is responsible for preprocessing the face video, extracting key points, and generating binary masks of facial features;

[0066] The expression generation module includes an appearance reference network, a posture reference network, and an expression generation network. During training, the source image and the target expression image are photos of the same person, but they do not need to be photos of the same person during the inference stage.

[0067] The appearance reference network is trained based on the latent space diffusion model StableDiffusion and is used to maintain the details of the face. StableDiffusion is a classic latent space diffusion model (LDM, LatentDiffusionModel). LDM uses variational autoencoder VAE. Given an input image, the image is first mapped to the latent space through the encoder, so that diffusion is carried out in the latent space. In the forward process of diffusion, Gaussian noise is gradually added to the latent variable through T iterations. The reverse diffusion process is the inverse process of the forward process. In order to gradually restore the noise to the initial latent variable, it is finally restored to the RGB space through the decoder to obtain the final image.

[0068] Pose reference network: The pose reference network is a convolutional network used to extract the pose of facial expressions. Referring to the design similar to ControlNet, pose reference encodings of different scales are obtained to correspond to the input scales of different modules in the expression image generation diffusion model. At the same time, pose images of the source image and the target expression image are input. The pose reference network promotes the interaction between the source image and the target expression image. This process provides additional clues for the network to understand the correlation between facial key points and appearance, thereby helping to generate images with more accurate expressions.

[0069] Expression generation network, which is trained using the pre-trained StableDiffusion model and is used to generate the corresponding target expression images;

[0070] The cross-attention module combines features from the expression generation network and the appearance reference network using cross-network fusion. After each layer or block is processed, the generated image retains the global features of the source image (such as color, facial structure, etc.) while achieving the change of the target expression. The model can generate expressions without destroying facial details.

[0071] The self-attention module automatically adjusts the focus based on the local and global features of the image. The model automatically identifies the key parts of the image (such as eyes, mouth and other key facial areas) and pays more attention to these areas during expression generation, thereby improving the naturalness and detail retention of expression generation.

[0072] The expression alignment module keeps the positions of facial features and facial postures of the source and target expression images consistent during the expression editing process, so that the generated target expression is natural and highly matched with the source image.

[0073] Example 2, reference Figure 1 This embodiment provides a training method for an expression editing model based on deep learning, comprising the following steps:

[0074] Step S1, preprocessing the face video, cropping the face video with the face as the center to obtain a 512×512 face video;

[0075] Step S2, key point extraction, use Mediapipe to extract 478-dimensional 3D facial key points from the facial features of each frame of the face video, draw the inner and outer contours of the facial features including eyebrows, eye contours, pupils and lips, guide the generation of new expressions, extract the positions of the inner and outer contours of the facial features, generate a binary mask of the facial features based on the extracted inner and outer contour positions, expand the facial features, adjust the size to the size required by the latent space, and obtain the expanded facial features binary mask I m , used to calculate the facial features loss function during training;

[0076] Step S3, randomly select two pictures from the video of the same person, one as the source picture I source , and another as the target expression picture I targe ;

[0077] Step S4: source image I source Input into the appearance reference network and CLIP image encoder. The CLIP image encoder retains the overall color and structure of the person, while the appearance reference network can retain the details of the face.

[0078] Step S5, set source image I source and target expression picture I targe The posture diagrams are and The posture diagram Input into the posture reference network to obtain the posture code z p As expression gesture control;

[0079] Step S6: Encode the posture z p Add to the randomly generated hidden noise and input to the expression generation network;

[0080] Step S7, the appearance details extracted by the appearance reference network and the CLIP image encoder are input into the expression generation network through the cross attention module, and the target expression image is obtained through cyclic denoising;

[0081] In the appearance reference network, the training steps include:

[0082] Source Image I source Input to the pre-trained CLIP image encoder In, it is expressed as: z c Represents the pre-trained CLIP image encoder Source Image I sourceThe encoding vector obtained after encoding is z c Input into the expression generation network through the cross attention module;

[0083] Source Image I source Input to the pre-trained VAE encoder , becomes a latent space encoding (I source ), input z into the appearance reference network τ ref The facial detail features are extracted from the expression generation network through the cross attention module.

[0084] In the appearance reference network, the training step also includes:

[0085] Perform feature fusion and transform the expression generation network ∈ θ and the appearance reference network τ ref The key and value of each attention module in are combined, and the subscripts θ and ref represent the expression generation network and the appearance reference network respectively. The attention fusion form is: Among them, Q θ represents the query vector of the expression generation network, K θ represents the key vector of the expression generation network, that is, the encoding of the input features, K ref represents the key vector of the appearance reference network, corresponding to the features in the appearance reference network, V θ Represents the value vector of the expression generation network, that is, the feature value extracted from the network, V ref Represents the value vector of the appearance reference network, that is, the feature value extracted from the appearance reference network, d k Represents the dimension of the key vector, which is used to normalize the dot product result, cat(K θ ,K ref ) and cat(V θ ,V ref ), cat represents the concatenation operation, cat(K θ ,K ref ) concatenates the key vectors of the expression generation network and the appearance reference network, cat(V θ ,V ref ) is the concatenation of value vectors, T represents the transposition operation, and the softmax function converts the result of the dot product into a probability distribution; thereby effectively combining the features of the expression generation network with the features of the appearance reference network, so that the generated image can change the expression while maintaining the details and consistency of the face;

[0086] In the pose reference network, the training steps include:

[0087] The pose graph of the source image and the pose graph of the target expression image are combined Input to the pose reference network τ p middle;

[0088] Pose Reference Network τ p The posture code obtained It is directly added to the randomly generated 4-channel latent space Gaussian noise and input into the expression generation network together;

[0089] In the expression generation network, the training steps include:

[0090] Generate random Gaussian noise in latent space Plus the pose code z extracted by the pose reference network p , we get ∈ p ,∈ p represents the pose encoding z generated by randomly generated Gaussian noise and the pose reference network p The combined latent space input;

[0091] Will ∈ p The facial appearance features extracted by CLIP are fused with the features of the expression generation network. At the same time, the features extracted by the appearance reference network are fused with the features of the expression generation network. θ and the appearance reference network τ ref The key and value of each attention module are combined, the feature encoding of the expression generation network output is finally passed through the VAE decoder to obtain the target expression map;

[0092] In the expression generation network, the training steps also include:

[0093] The loss function is minimized and the Adam optimizer is used for optimization iteration. The Loss function of the diffusion model is:

[0094] Among them, τ p ,τ ref ,∈ θ They are the pose reference network, appearance reference network and expression generation network, which are the trainable parts of the model, ε c It is a pre-trained CLIP image encoder and does not participate in training. represents the expected value operator, is the diffusion loss function;

[0095] The loss function is the real target expression picture I tgt And the generated expression picture I gen The perceptual loss between vggis the pre-trained VGG19 network, l is the l-th layer output of the VGG19 network, The loss function is expressed as:

[0096] The facial features loss function is introduced, and the facial features binary mask I obtained by the image processing module is used m , the facial features loss function is:

[0097] in For the expression generation network ∈ θ The predicted latent space code, z is the latent space code of the target expression image;

[0098] The overall loss function is:

[0099] Example 3, reference Figure 2 , this embodiment provides a reasoning method for an expression editing model based on deep learning, comprising the following steps:

[0100] The model structure of the inference process of the expression editing model is similar to that of the training process, with the addition of an expression alignment module;

[0101] The reasoning process for the target expression is:

[0102] User input source image;

[0103] The source image and the target expression image are processed by an image processing module to extract a posture graph of the source image and a posture graph of the target expression image;

[0104] The pose graph of the source image and the pose graph of the target expression image are passed through the expression alignment module to obtain the pose graph of the source image and the aligned pose graph of the target expression. The 3D facial key points of the pose graph of the source image and the pose graph of the target expression image are extracted using Mediapipe, and the key points are projected onto a 2D plane to obtain the facial key points at the corresponding image position.

[0105] Use the relative relationship between the facial key points projected onto the 2D plane and the 3D facial key points to obtain the 3D key points of the pupil;

[0106] Calculate the relative relationship between the facial key points projected onto the 2D plane and the 3D facial key points, and apply the transformation matrix and camera matrix of the relative standard space in Mediapipe to obtain the target expression pose graph after projecting the target expression from the 3D space onto the 2D plane of the source image, i.e. the aligned target expression pose graph;

[0107] The reasoning process of the target expression also includes:

[0108] Input the source image into the appearance reference network and CLIP image encoder;

[0109] The pose graph of the source image and the aligned target expression pose graph are input into the pose reference network to obtain the pose encoding as the expression pose control;

[0110] The posture code is added to the randomly generated hidden noise and input into the expression generation network. The appearance details extracted by the appearance reference network and the CLIP image encoder are input into the expression generation network through the cross attention module. After cyclic denoising, the target expression image is obtained.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A training method for an expression editing model based on deep learning, characterized in that: The deep learning-based expression editing model includes: The image processing module is responsible for preprocessing the face video, extracting key points, and generating binary masks of facial features; The expression generation module includes an appearance reference network, a posture reference network, and an expression generation network. During training, the source image and the target expression image are photos of the same person, but they do not need to be photos of the same person during the inference stage. The appearance reference network is trained based on the latent space diffusion model StableDiffusion and is used to preserve the details of the face. Pose Reference Network,The pose reference network is a convolutional network used to extract the pose of facial expressions; Expression generation network, which is trained using the pre-trained StableDiffusion model and is used to generate the corresponding target expression images; Cross-attention module, which combines features from the expression generation network and the appearance reference network using cross-network fusion; Self-attention module, which automatically adjusts the focus based on the local and global features of the image; Expression alignment module, which makes the positions of facial features and facial postures of the source and target expression images consistent during expression editing; The steps of the training method of the expression editing model based on deep learning include: Step S1, preprocessing the face video, cropping the face video with the face as the center to obtain a 512×512 face video; Step S2, key point extraction, use Mediapipe to perform 478-dimensional 3D face key point extraction on the facial features of each frame of the face video, draw the inner and outer contours of the facial features including eyebrows, eye contours, pupils and lips, extract the positions of the inner and outer contours of the facial features, generate a binary mask of the facial features according to the extracted inner and outer contour positions, expand the facial features, adjust the size to the size required by the latent space, and obtain the expanded facial features binary mask I m ; Step S3, randomly select two pictures from the video of the same person, one as the source picture I source , and another as the target expression picture I targe ; Step S4: source image I source Input to the appearance reference network and CLIP image encoder; Step S5, set source image I source and target expression picture I targe The posture diagrams are and The posture diagram Input into the posture reference network to obtain the posture code z p As expression gesture control; Step S6: Encode the posture z p Add to the randomly generated hidden noise and input to the expression generation network; Step S7, the appearance details extracted by the appearance reference network and the CLIP image encoder are input into the expression generation network through the cross attention module, and the target expression image is obtained through cyclic denoising; In the appearance reference network, the training steps include: Source Image I source Input to the pre-trained CLIP image encoder ε c In the formula, it is expressed as: z c =ε c (I source ), z c Represents the pre-trained CLIP image encoder ε c Source Image I source The encoding vector obtained after encoding is z c Input into the expression generation network through the cross attention module; Source Image I source Input into the pre-trained VAE encoder ε, it becomes the latent space encoding z = ε (I source ), input z into the appearance reference network τ ref Extract facial detail features from the network, and input the extracted facial detail features into the expression generation network through the cross attention module; Perform feature fusion and transform the expression generation network ∈ θ and the appearance reference network τ ref The keys and values ​​of each attention module in are merged together, and the subscripts θ and ref represent the expression generation network and the appearance reference network respectively. The attention fusion form is: Among them, Q θ represents the query vector of the expression generation network, K θ represents the key vector of the expression generation network, that is, the encoding of the input features, K ref represents the key vector of the appearance reference network, corresponding to the features in the appearance reference network, V θ Represents the value vector of the expression generation network, that is, the feature value extracted from the network, V ref Represents the value vector of the appearance reference network, that is, the feature value extracted from the appearance reference network, d k Represents the dimension of the key vector, which is used to normalize the dot product result, cat(K θ ,K ref ) and cat(V θ ,V ref ), cat represents the concatenation operation, cat(K θ ,K ref ) concatenates the key vectors of the expression generation network and the appearance reference network, cat(V θ ,V ref ) is the concatenation of the value vectors, T represents the transpose operation, and the softmax function converts the result of the dot product into a probability distribution; In the pose reference network, the training steps include: The pose map of the source image and the pose map of the target expression image Input to the pose reference network τ p middle; Pose Reference Network τ p The posture code obtained It is directly added to the randomly generated 4-channel latent space Gaussian noise and input into the expression generation network together; In the expression generation network, the training steps include: Generate random Gaussian noise in latent space Plus the pose code z extracted by the pose reference network p , we get ∈ p ,∈ p represents the pose encoding z generated by randomly generated Gaussian noise and the pose reference network p The combined latent space input; Will ∈ p The facial appearance features extracted by CLIP are fused with the features of the expression generation network. At the same time, the features extracted by the appearance reference network are fused with the features of the expression generation network. θ and the appearance reference network τ ref The keys and values ​​of each attention module are merged, the feature encoding of the expression generation network output is finally passed through the VAE decoder to obtain the target expression map.

2. The method for training an expression editing model based on deep learning according to claim 1, characterized in that: In the expression generation network, the training steps also include: The loss function is minimized and the Adam optimizer is used for optimization iteration. The Loss function of the diffusion model is: Among them, τ p ,τ ref ,∈ θ They are the pose reference network, appearance reference network and expression generation network, which are the trainable parts of the model, ε c It is a pre-trained CLIP image encoder and does not participate in training. represents the expected value operator, is the diffusion loss function; The loss function is the real target expression picture I tgt And the generated expression picture I gen The perceptual loss between vgg is the pre-trained VGG19 network, l is the l-th layer output of the VGG19 network, The loss function is expressed as: The facial features loss function is introduced, and the facial features binary mask I obtained by the image processing module is used m , the facial features loss function is: in For the expression generation network ∈ θ The predicted latent space code, z is the latent space code of the target expression image; The overall loss function is:

3. A reasoning method for an expression editing model based on deep learning, characterized in that: The training method of the expression editing model based on deep learning according to claim 1 comprises the following steps: User input source image; The source image and the target expression image are processed by an image processing module to extract a posture graph of the source image and a posture graph of the target expression image; The pose graph of the source image and the pose graph of the target expression image are passed through the expression alignment module to obtain the pose graph of the source image and the aligned pose graph of the target expression. The 3D facial key points of the pose graph of the source image and the pose graph of the target expression image are extracted using Mediapipe, and the key points are projected onto a 2D plane to obtain the facial key points at the corresponding image position. Use the relative relationship between the facial key points projected onto the 2D plane and the 3D facial key points to obtain the 3D key points of the pupil; Calculate the relative relationship between the facial key points projected onto the 2D plane and the 3D facial key points, and apply the transformation matrix and camera matrix of the relative standard space in Mediapipe to obtain the target expression posture graph after projecting the target expression from the 3D space to the 2D plane of the source image, that is, the aligned target expression posture graph.

4. The reasoning method of the expression editing model based on deep learning according to claim 3 is characterized in that: The inference methods for the target expression also include: Input the source image into the appearance reference network and CLIP image encoder; The pose graph of the source image and the aligned target expression pose graph are input into the pose reference network to obtain the pose encoding as the expression pose control; The posture code is added to the randomly generated hidden noise and input into the expression generation network. The appearance details extracted by the appearance reference network and the CLIP image encoder are input into the expression generation network through the cross attention module. After cyclic denoising, the target expression image is obtained.

Images