Fine-grained living body face attribute editing method and system based on text guidance

Through the facial identity feature and facial key point feature extraction module combined with the conditional diffusion model and the cross attention adjustment module, the problems of fine-grained editing and identity consistency in the existing technology are solved, and high accuracy and diversity of faces are achieved Properties editing.

CN119027996AActive Publication Date: 2024-11-26SOUTH CHINA UNIV OF TECH

Patent Information

Application Number
CN202410970065.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-26
Estimated Expiration
2044-07-19

AI Technical Summary

Technical Problem

The existing text-guided face attribute editing algorithm is difficult to achieve fine-grained and diverse editing, while maintaining the consistency of face identity, and coarse-grained text descriptions cannot accurately control image generation.

Method used

The facial identity feature extraction module and the facial key point feature extraction module are used to obtain multi-scale features, combine the conditional diffusion model and the cross attention adjustment module to optimize the alignment of image and text modes, and achieve fine-grainedness through two-stage training and text interpolation. property editing.

Benefits of technology

It realizes refined and diverse editing of face attributes, maintains the consistency of face identity, and improves the accuracy and diversity of generated images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027996A_ABST
    Figure CN119027996A_ABST
Patent Text Reader

Abstract

The invention discloses a fine-grained living body face attribute editing method and system based on text guidance, and the method comprises the following steps: obtaining a face image data set and a text data set, and carrying out the data preprocessing; based on a face identity feature extraction module and a face key point feature extraction module, respectively extracting multi-scale face identity features and multi-scale face key point features of the face image; performing feature splicing on the multi-scale face identity features and the multi-scale face key point features based on a multi-scale feature mixing module, and performing weighted average to obtain face identity structure features; respectively embedding the face image and the original text into the conditional diffusion model as an image mode and a text mode; carrying out two-stage training; and obtaining a final text condition vector based on the original text and the guide text, inputting the final text condition vector into the trained condition diffusion model, and generating a fine-grained face attribute editing picture. According to the invention, refinement and diversification of face attribute editing are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of face attribute editing, and in particular to a text-guided fine-grained living face attribute editing method and system. Background Art

[0002] Face attribute editing technology can generate a variety of images based on existing real faces, such as predicting the gradual aging process of faces, simulating facial expressions, and the change process of hairstyles. By editing the facial attributes of faces, the face liveness detection model can be exposed to a large amount of facial data of the same face in different scenarios during the training process. On the one hand, it can greatly improve the generalization and robustness of the current face liveness detection model and avoid misjudgment of the identity of faces in unknown scenarios. On the other hand, it can reduce the large workload of manually collecting facial data, and protect the face data from being leaked, solving privacy and security issues. In addition, by controlling the change of a certain facial attribute of the face, the facial situation of faces with different facial attribute differences can be simulated, which helps to better explain the working mechanism of the face liveness detection model, such as the sensitivity of the face recognition system to facial attributes, the impact of specific facial editing on the recognition results, etc. Therefore, face attribute editing technology has important practical significance, and its main application scenarios include access control recognition systems, bank and mobile payment verification systems, and urban monitoring systems.

[0003] Early face attribute editing used generative adversarial networks such as GAN or VAE, combined with attribute labels or text descriptions to modify specific style vectors in the latent space to achieve editing of specific attributes. Most existing research methods use CLIP text encoders and image encoders to introduce multimodal conditions into the diffusion model to generate high-resolution images, which can achieve convenient and efficient attribute modification based on natural language descriptions, improving the diversity and fidelity of generated images.

[0004] Existing text-guided face attribute editing algorithms still have some shortcomings. They usually use a general image encoder to encode the input face, ignoring the consistency of the face's unique identity features and facial structure features during the generation process; secondly, existing algorithms lack text that accurately describes face attributes. A piece of text usually contains many descriptions of face attributes, and editing one attribute often leads to changes in other attributes. In addition, coarse-grained text descriptions often cannot represent the fine-grained information in the image, resulting in the inability of text to accurately edit the image and a lack of diversity in generation. Therefore, how to use text to achieve refined, fine-grained, and diverse editing of face attributes while maintaining the consistency of face identity is the research focus and difficulty in this field. Summary of the invention

[0005] In order to overcome the defects and shortcomings of the prior art, the present invention provides a text-guided fine-grained live face attribute editing method and system. The present invention utilizes a face identity feature extraction module and a facial key point feature extraction module to obtain face identity features and facial structure features, and adopts a multi-scale feature extraction strategy so that the global features and local features of the face can be fully characterized, wherein the low-level features contain more position and detail information, while the high-level features contain stronger deep semantic features of the image; utilizing a cross-attention adjustment module based on a conditional diffusion model, respectively utilizing image modality and text modality to reconstruct the input image, optimizing the similarity of the image modality and text modality attention feature vectors, and achieving the degree of alignment between the text and the image at the latent space level; utilizing the interpolation of the original text encoding and the guided text encoding as the query matrix and the content matrix, and cross-attention is performed with the face identity structure feature index matrix, and the face attribute editing based on the guide word is gradually realized through the conditional diffusion model, thereby achieving the refinement and diversification of attribute editing.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a text-guided fine-grained living face attribute editing method, comprising the following steps:

[0008] Obtain a face image dataset and a text dataset, divide the dataset, unify the resolution of the face images, and for each face image, split the text dataset into descriptions of the attributes according to different facial attributes;

[0009] Constructing a face identity feature extraction module and a facial key point feature extraction module, wherein the face identity feature extraction module extracts multi-scale face identity features of a face image, and the facial key point feature extraction module extracts multi-scale facial key point features of a face image;

[0010] Construct a multi-scale feature mixing module, which combines multi-scale face identity features and multi-scale facial key point features according to corresponding scales to obtain identity structure features, and performs weighted average of identity structure features at each scale to obtain face identity structure features;

[0011] The text data is passed through a text encoder to obtain the original text embedding, and the face image and the original text embedding are injected into the conditional diffusion model as the image modality and the text modality, respectively, to train the conditional diffusion model;

[0012] Two-stage training is carried out. In the first stage of training, the face identity structure features are used as image modal conditions, and the face identity structure features and the original text embedding are injected into the conditional diffusion model as multimodal conditions. The corresponding attention feature vector is calculated, and the alignment degree of the face identity structure features and the original text embedding is optimized based on the attention loss. The parameters of the face identity feature extraction module and the facial key point feature extraction module are optimized by the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized by the latent vector reconstruction loss. In the second stage of training, the semantic consistency loss is used to optimize the semantic consistency between the guided text and the original text.

[0013] The original text and the guide text are interpolated to obtain the final text conditional vector, which is input into the trained conditional diffusion model to generate a fine-grained face attribute editing image.

[0014] As a preferred technical solution, the face identity feature extraction module includes a multi-scale encoder and a pre-trained face identity feature extractor;

[0015] The multi-scale encoder includes a convolution layer, a maximum pooling layer, a plurality of residual networks based on ResNet, and a pooling network, and the residual network based on ResNet extracts multi-scale features;

[0016] The pre-trained face identity feature extractor includes a face identity feature extraction network based on ArcFace;

[0017] The multi-scale encoder outputs a multi-scale feature vector f=[f 1 , f 2 , ..., f N ];

[0018] The multi-scale feature vector is input into the pre-trained face identity feature extractor to extract the multi-scale face identity feature f id =[f 1 id , f 2 id , ..., f N id ], the identity feature of each scale is expressed as:

[0019]

[0020] in, represents the face identity feature of the i-th scale, F Arc represents the face identity feature extraction network based on ArcFace, and N represents the size of the division scale.

[0021] As a preferred technical solution, the facial key point feature extraction module includes a pre-trained face key point extraction network PFLD and a facial key point adjustment network;

[0022] The pre-trained face key point extraction network PFLD includes a convolutional layer, a Bottleneck module and a multi-scale fusion network. The feature representation of the face image extracted by the pre-trained face key point extraction network is:

[0023] f sp =F Bottleneck (F DsConv (F Conv (I img )))

[0024] Among them, f sp represents the output feature of the Bottleneck module, F Bottleneck represents the Bottleneck module of the residual network, F DSConv represents a deep separable convolutional network, F Conv represents the convolutional network, I img Represents the input face image;

[0025] The output feature f sp Input to the facial key point adjustment network, which has multiple convolution kernels of different sizes and outputs multi-scale facial key point features Among them, N represents the size of the division scale;

[0026] The multi-scale face identity feature and the multi-scale facial key point feature have the same dimension.

[0027] As a preferred technical solution, the face identity structure feature is expressed as:

[0028]

[0029] Among them, C id represents the face identity structure feature, N represents the size of the division scale, Cancat represents the feature concatenation of multi-scale face identity features and multi-scale facial key point features, represents the face identity feature of the i-th scale, Represents the facial key point feature of the i-th scale.

[0030] As a preferred technical solution, the face image and the original text are respectively embedded as the image modality and the text modality and injected into the conditional diffusion model, and the conditional diffusion model is trained, specifically including:

[0031] Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network;

[0032] The alignment of image semantics and text semantics at the latent space level is achieved by optimizing the similarity of the attention feature vectors of text and image. The attention feature vector is expressed as:

[0033]

[0034] Where K, Q, and V represent the index matrix, content matrix, and query matrix corresponding to the image modality and text modality, respectively. d is the projection dimension of the index matrix and query matrix. att Represents the output of the cross attention layer as the input of the convolutional layer of the conditional diffusion model;

[0035] Each intermediate sample z of the conditional diffusion model t The following conditions must be met:

[0036]

[0037] Among them, α t represents the attenuation coefficient of the conditional diffusion model, z t-1 represents the intermediate sample of the t-1-step diffusion process, ∈ t is the noise that obeys Gaussian distribution;

[0038] The conditional diffusion model is trained by minimizing the loss function, expressed as:

[0039]

[0040] Among them, ∈ θ It is the U-Net network of the denoising generation model, ∈ is the noise obeying the standard Gaussian distribution, t represents the tth step of the diffusion process, C is the input modal feature vector, ||·||2 represents the L2 loss, and z0 represents random noise.

[0041] As a preferred technical solution, given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network, which are specifically expressed as:

[0042]

[0043]

[0044] Q orign =W t Q ×C orign

[0045]

[0046] C orign =E T (T orign )

[0047] Among them, K orign 、V orign , Q orign K represents the index matrix, content matrix, and query matrix corresponding to the original text embedding. id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, W t Q , are six linear transformation matrices with different weights, T orign Represents the original text, E T Represents CLIP text encoder, C orign Represents the raw text embedding.

[0048] As a preferred technical solution, in the first stage of training, it is implemented in two steps:

[0049] Step 1: In the forward and reverse diffusion processes, the face identity structure features are injected into the conditional diffusion model as image modal conditions. The sampling process is expressed as:

[0050]

[0051] Among them, α i is the attenuation coefficient of the conditional diffusion model, z t-1 、z t are the latent vectors of the diffusion process in the t-1th and tth steps in step 1, σ t Indicates z t The variance of e is the random noise that follows the standard Gaussian distribution, ∈ θ The U-Net network of the denoising generation model, C id represents the structural features of face identity, and t represents the tth step of the diffusion process;

[0052] Calculate the corresponding attention feature vector, expressed as:

[0053]

[0054] Among them, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, K id 、V id, Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features;

[0055] Step 2: In the forward and reverse diffusion processes, the face identity structure features and the original text embedding are injected into the conditional diffusion model as multimodal conditions. The index matrix, content matrix, and query matrix of the cross attention layer are K orign 、V id , Q orign , the sampling process is expressed as:

[0056]

[0057] in, are the hidden vectors of the diffusion process of the t-1th and tth steps in step 2, express The variance, C orign represents the original text embedding;

[0058] Calculate the corresponding attention feature vector, expressed as:

[0059]

[0060] The overall loss function of the first stage training is expressed as:

[0061] L=k1L attention +k2L rec +k3L per +k4L landmark

[0062]

[0063] Among them, k1, k2, k3, and k4 are the attention loss L attention , latent vector reconstruction loss L rec , identity-aware loss L per , facial key point loss L landmark The weight coefficient, E T Represents CLIP text encoder, C l , H l , W l is the number of channels, height, and width of the feature map of the lth layer, φ l (I img ) and φ l (I rec ) represent face images I img And the face I reconstructed by the conditional diffusion model recThe output features after the lth layer of VGGFace, ijk represents the result of the activation output of the i-th convolution kernel at position (j, k) after the intermediate layer, C represents the number of key points of the face, and They represent the key points of the input face and the reconstructed face respectively, and ||·||1 represents the L1 loss.

[0064] As a preferred technical solution, in the second stage of training, the original text and the guide text are input into the CLIP text encoder to obtain the original text embedding and the guide text embedding, and the semantic consistency between the guide text and the original text is optimized by the semantic consistency loss. The semantic consistency loss is expressed as:

[0065]

[0066] Among them, i represents the word vector in each text embedding, β is a hyperparameter, and C orign represents the original text embedding, C target Represents a guided text embedding.

[0067] As a preferred technical solution, the original text and the guide text are interpolated to obtain the final text condition vector, which is specifically expressed as:

[0068] C insert =η×C target +(1-η)×C orign

[0069] Among them, η∈(0,1) represents the interpolation coefficient, C orign represents the original text embedding, C target represents the guided text embedding, C insert represents the final text condition vector;

[0070] The face identity structure features and the final text conditional vector are injected into the conditional diffusion model as image modality conditions and text modality conditions. The sampling process is expressed as:

[0071]

[0072] in, Respectively represent the latent vectors of the t-th and t-1-th step diffusion processes, express The variance of i is the attenuation coefficient of the conditional diffusion model, z t is the latent vector of the t-th step diffusion process, ∈ θ is the U-Net network of the denoising generation model, e is random noise that obeys the standard Gaussian distribution, when t is 0, Represents the Gaussian noise of the input conditional diffusion model. When t is T, It represents the target image after T-step sampling and editing according to the final text condition vector prompt;

[0073] When η=0, the model output is the target image, that is, the input original image, and when η=1, the model output is the image edited based on the guided text embedding.

[0074] The present invention also provides a text-guided fine-grained living face attribute editing system, which is used to implement the above-mentioned text-guided fine-grained living face attribute editing method, and the system includes: a data set acquisition module, a data set preprocessing module, a face identity feature extraction module, a facial key point feature extraction module, a multi-scale feature mixing module, a text encoder, a conditional diffusion model construction module, a two-stage training module, a text interpolation module, and a face attribute editing picture output module;

[0075] The data set acquisition module is used to acquire a face image data set and a text data set;

[0076] The data set preprocessing module is used to divide the data set, unify the resolution size of the face image, and for each face image, split the text data set into description texts for the attribute according to different facial attributes;

[0077] The face identity feature extraction module is used to extract multi-scale face identity features of face images;

[0078] The facial key point feature extraction module is used to extract multi-scale facial key point features of the face image;

[0079] The multi-scale feature mixing module is used to perform feature splicing on the multi-scale face identity features and the multi-scale facial key point features according to the corresponding scales to obtain the identity structure features, and to perform weighted averaging on the identity structure features of each scale to obtain the face identity structure features;

[0080] The text encoder is used to encode text data into raw text embedding;

[0081] The conditional diffusion model construction module is used to construct a conditional diffusion model, and inject the face image and the original text embedding into the conditional diffusion model as the image modality and the text modality respectively, and train the conditional diffusion model;

[0082] The two-stage training module is used to perform two-stage training. In the first stage of training, the face identity structure feature is used as an image modal condition, and the face identity structure feature and the original text embedding are respectively injected into the conditional diffusion model as a multimodal condition, and the corresponding attention feature vector is calculated. The alignment degree of the face identity structure feature and the original text embedding is optimized based on the attention loss, and the parameters of the face identity feature extraction module and the facial key point feature extraction module are optimized by the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized by the latent vector reconstruction loss. In the second stage of training, the semantic consistency between the guided text and the original text is optimized by the semantic consistency loss.

[0083] The text interpolation module is used to obtain a final text condition vector by interpolating the original text and the guide text;

[0084] The face attribute edited picture output module is used to input the final text conditional vector into the trained conditional diffusion model to generate a fine-grained face attribute edited picture.

[0085] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0086] (1) The present invention extracts multi-scale face identity features and multi-scale facial key point features of face images based on a face identity feature extraction module and a facial key point feature extraction module, respectively, obtains face identity structure features, solves the problem of mismatch between the generated face and the input face identity, and retains the facial structure of the input face.

[0087] (2) Based on face datasets and text datasets, the present invention conducts separate two-stage training for each attribute in the text description of different attributes, ensuring that each face attribute does not interfere with each other, facilitating the precise control of the generation of face attributes by text, and improving the reliability and stability of the generated image.

[0088] (3) The present invention reconstructs the input image through two modalities, namely, picture and original text, and uses the cross-attention mechanism and reconstruction loss to achieve alignment between text and image at the latent space level, which can improve the accuracy of text-generated pictures. In the process of editing facial attributes, the present invention adopts a progressive method of original text and guided text interpolation, which can gradually edit facial attributes in a fine-grained manner and continuously enrich the details of the image. For example, the attribute of "beard" can be gradually added to the face, and the generated beard transitions from thick to dense, and the generated image is more diverse. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a structural schematic diagram of a face identity feature extraction module of the present invention;

[0090] Figure 2 It is a structural schematic diagram of a facial key point feature extraction module of the present invention;

[0091] Figure 3 This is a schematic diagram of the training process in the first stage of the present invention;

[0092] Figure 4 This is a schematic diagram of the training process in the second stage of the present invention;

[0093] Figure 5 This is a schematic diagram of the face attribute editing and interpolation process in the testing phase of the present invention;

[0094] Figure 6 Editing and interpolating five facial attributes in the test phase of the present invention to generate a facial schematic diagram;

[0095] Figure 7 This is a schematic diagram comparing the effects of face attribute editing in the testing phase of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0097] Example 1

[0098] This embodiment uses the celebrity face image dataset CelebA-HQ and the corresponding text dataset CelebA-Dialog for training and testing as an example to introduce the specific implementation process of the present invention in detail. The CelebA-HQ dataset consists of high-resolution face images and corresponding 40 attribute labels, including more than 1,000 celebrities, including multiple face attributes such as age, gender, hairstyle, expression, etc., with a total of 30,000 pictures. The CelebA-Dialog dataset is used to study how to use natural language descriptions to edit and generate face images. The dataset provides rich and highly fine-grained natural language descriptions for the 30,000 face images in CelebA-HQ, including descriptions of five attributes: "bangs", "eyeglasses", "no beard", "smile", and "age" and an overall description ("overall_caption"). The acquisition and use of privacy, portraits and other information in the present invention are in compliance with the laws and regulations of relevant countries and regions.

[0099] This embodiment is deployed on the Linux system, based on the Pytorch1.7.0 deep learning framework, the configuration environment is Anaconda, the hardware configuration is two NVIDIA GeForce RTX 3090Ti, the graphics card driver is Driver Version: 470.239.06, the CUDA version is 11.4, the running container is Conda, and the main dependent libraries are opencv, imageio, pytorch-lightning, transformers, omegaconf, etc.

[0100] like Figure 1 , Figure 2 As shown, this embodiment provides a text-guided fine-grained living face attribute editing method, comprising the following steps:

[0101] S1: Data preprocessing;

[0102] In this embodiment, the image data set and the text data set are divided into a training set, a test set, and a validation set in a ratio of 7:2:1. The five attribute descriptions in the text data set CelebA-Dialog are split using the json toolkit and saved as captions_hq_Bangs.json, captions_hq_Beard.json, captions_hq_Eyeglasses.json, captions_hq_Age.json, and captions_hq_Eyeglasses.json files. The resolution of the face image is adjusted to H×W×C, where H, W, and C are the image height, width, and number of channels, respectively. In this embodiment, the values ​​are H=512, W=512, and C=3, respectively.

[0103] S2: Constructing face identity feature extraction module;

[0104] In this embodiment, the face identity feature extraction module is mainly used to extract multi-scale identity features of the face. The main structure includes a multi-scale encoder F Φ and the pre-trained F Arc Face identity feature extractor. The multi-scale encoder F Φ It mainly includes a 7×7 convolution layer, a 3×3 maximum pooling layer, 4 residual networks based on ResNet, a pooling network, and F Arc The face identity feature extractor includes a face identity feature extraction network based on ArcFace;

[0105] The specific process is as follows: the input face image resolution is H×W×C. First, the image is input into a convolutional neural network with a convolution kernel size of 7×7, a step size of 1, a padding of 3, and a channel number of 256. Then it is input into a 3×3 maximum pooling network to obtain a size of Then the initial features are input into four residual networks to extract multi-scale features. The residual network uses two types of convolution kernels: 1×1 and 3×3. The 1×1 convolution kernel is used to change the number of output channels, and the 3×3 convolution changes the output feature size. BatchNorm is used as batch normalization, and the mean and standard deviation of small batches are used to make the intermediate output values ​​of each layer of the neural network more stable. The activation function uses the RELU function.

[0106] After the residual network, the sizes are obtained respectively The multi-scale features are further input into the pooling network, which includes global average pooling, global maximum pooling, activation network and 1×1 convolution layer. Global maximum pooling is mainly used to extract the most relevant features in each channel and eliminate redundant data. Global average pooling is mainly used to uniformly obtain all features in the feature map to minimize feature loss. The activation network mainly uses Sigmoid function and 1×1 convolution layer to unify the number of feature channels of each scale to 256. After the multi-scale encoder F Φ After that, the multi-scale feature vector f of the input image is obtained, f = [f 1 , f 2 , ..., f N ], where N represents the size of the division scale. In this embodiment, N=4 is preferred, that is, f=[f 1 , f 2 , f 3 , f 4 ], the number of feature vector channels at each scale is 256.

[0107] These multi-scale features are then fed into the pre-trained F Arc The face identity feature extractor extracts the multi-scale identity features of the face f id =[f 1 id , f 2 id , f 3 id , f 4 id ], the identity characteristics of each scale are shown in the following equation:

[0108]

[0109] where f iid represents the face identity feature of the i-th scale; the key to ArcFace is to design the angle interval loss function to introduce additional intervals in the angle space, which enhances the compactness within the class and the difference between classes, and improves the ability to extract face identity features.

[0110] S3, constructing a facial key point feature extraction module;

[0111] In this embodiment, the facial key point feature extraction module constrains the geometric structure of the face through the PFLD network, and simultaneously retains the overall posture and local structural features of the face by predicting 68 facial key points. The main structure includes a pre-trained facial key point extraction network PFLD and a facial key point adjustment network. The PFLD network includes a 3×3 convolution layer, a Bottleneck module, and a multi-scale fusion network, the purpose of which is to fuse with the multi-scale identity features extracted by the first branch. The whole process can be expressed by the following equation:

[0112] f sp =F Bottleneck (F DSConv (F Conv (I img )))

[0113] Among them, I img represents the input face image, F conv represents a common convolutional network, F DSConv represents a deep separable convolutional network, F Bottleneck It is the Bottleneck module of the residual network.

[0114] The specific process is as follows: the input face image resolution is H×W×C. First, the image is input into a convolutional neural network with a convolution kernel size of 3×3, a step size of 1, a padding of 3, and a channel number of 256. Then it is input into a deep separable convolutional network to obtain a size of The initial features of . In the depthwise separable convolution, each channel of the image uses a separate convolution kernel to process information, which mainly includes two steps: Point Convlution and Depthwise Convlution. Point Convlution refers to the use of a 1×1 convolution kernel to process cross-channel information, and multiple 1×1 convolution kernels can be used to obtain multiple output feature maps. Depthwise Convlution refers to the use of a 3×3 convolution kernel to process only the information of a certain channel of the input image, that is, the spatial information in the length and width directions. The depthwise separable convolution greatly reduces the number of parameters and calculations, and improves the efficiency of convolution operations. Then the initial features are input into the Bottleneck module of the residual network, including 6 ResNet128 and 2 ResNet16 Bottleneck layers, and the size is obtained. The characteristic f sp .

[0115] After that, the facial key point features output by the PFLD network are input into the facial key point adjustment network. The main structure of the facial key point adjustment network includes 7×7, 5×5, 3×3, and 1×1 convolution kernels, a convolutional neural network with 256 channels, and an output size of Multi-scale facial key point features f s ' p , Wherein, N represents the size of the division scale, and in this embodiment, N=4 is preferred, that is, Multi-scale facial key point features f s ' p and the multi-scale identity feature f id have the same dimensions.

[0116] S4, construct a multi-scale feature mixing module;

[0117] In this embodiment, the multi-scale feature mixing module obtains the face identity structure feature by fusing the identity feature and the facial key point feature. First, the multi-scale feature mixing module converts the face identity feature f id =[f 1 id , f 2 id , f 3 id , f 4 id ] and facial key point features The features are spliced ​​according to the corresponding scale as the identity structure feature at that scale. Secondly, the identity structure feature of each scale is weighted averaged to obtain the face identity structure feature C id, which contains the unique identity features of the entire face and the facial geometric structure features. The whole process can be expressed by the following equation:

[0118]

[0119] Where α is a weight coefficient, which is set to 0.5 in this embodiment, indicating that identity and facial structure features are equally important. N represents the size of the division scale, which is set to 4 in this embodiment. Cancat represents the feature concatenation of face identity features and facial key point features at different scales.

[0120] S5. Construct a cross-attention adjustment module based on the conditional diffusion model;

[0121] In this embodiment, the main function of this module is to optimize the alignment between the original text embedding and the input face image in the latent space. First, the original text is passed through the text encoder to obtain the original text embedding. The process can be expressed as the following equation:

[0122] C orign =E T (T orign )

[0123] Among them, T orign Represents the original text, E T Represents CLIP text encoder, C orign represents the original text embedding;

[0124] The conditional diffusion model realizes text-guided image generation by inputting text prompts into the cross-attention layer. Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network. The process can be expressed as the following equation:

[0125]

[0126]

[0127] Q orign =W t Q ×C orign

[0128]

[0129] Among them, K orign 、V orign , Q orign K represents the index matrix, content matrix, and query matrix corresponding to the original text embedding. id 、V id , Q idRepresents the index matrix, content matrix, and query matrix corresponding to the face identity structure features; W t Q , are six linear transformation matrices with different weights;

[0130] Furthermore, in the two-stage training, the similarity of the attention feature vectors of the text and the image is optimized to achieve the alignment of the image semantics and the text semantics at the latent space level. The calculation of the attention feature vector is shown in the following equation:

[0131]

[0132] Where K, Q, and V represent the index matrix, content matrix, and query matrix corresponding to the image modality and text modality, respectively, and d is the projection dimension of the index matrix and query matrix; f att is the output of the cross attention layer, which is used as the input of the subsequent diffusion model convolution layer;

[0133] The conditional diffusion model is essentially a denoising generative model that trains the generative model to predict image z from random noise Z0 T , each intermediate sample z t (t=0,1,2,...T) all satisfy the following equation:

[0134]

[0135] Where T represents the number of diffusion steps, α t is the attenuation coefficient of the conditional diffusion model, ∈ t is the noise that obeys the standard Gaussian distribution;

[0136] The input face image and the original text embedding are injected into the conditional diffusion model as the image modality and the text modality respectively. The conditional diffusion model is trained by minimizing the following loss function:

[0137]

[0138] Among them, ∈ θ It is the backbone network of the denoising generation model, which is essentially a U-Net network. ∈ is the noise that obeys the standard Gaussian distribution. The value range of t is [0, T], which indicates the diffusion of the tth step. C is the input modality feature vector. This method includes two modalities: image and text. ||·||2 indicates the L2 loss.

[0139] S6, construct a two-stage training module;

[0140] In this embodiment, a two-stage training module is constructed so that the text can accurately edit the attributes of the face. The purpose of the first stage is to align the face image with the original text prompt at the latent space level, and the purpose of the second stage is to add new guide word embedding as the guide text based on the original text, and realize the face attribute editing based on the description of the guide word. The two-stage training process is as follows Figure 3 , Figure 4 shown.

[0141] The specific training process uses the Xavier parameter initialization method to initialize the model parameters. The small batch gradient descent optimization algorithm is used, and the batch size is set to 2. The Adam optimizer is used, the initial learning rate lr of the first stage is set to 0.001, the initial learning rate lr of the first stage is set to 1e-6, the weight decay parameter is 0.0006, the momentum parameter momentum is set to 0.9, and the learning rate is reduced by 20% every 50 epochs.

[0142] The first stage is implemented in two steps, the specific process is as follows:

[0143] Step 1: In the forward and reverse diffusion process, the face identity structure feature C id As the image modality condition, it is injected into the conditional diffusion model. The index matrix, content matrix, and query matrix of the cross attention layer are K id 、V id , Q id , d represents the projection dimension of the index matrix and the query matrix. The sampling process is shown in the following equation:

[0144]

[0145] where α i is the attenuation coefficient of the conditional diffusion model, z t-1 、z t are the latent vectors of the diffusion process in the t-1th and tth steps in step 1, σ t Indicates z t The variance of e is the random noise that follows the standard Gaussian distribution, ∈ θ It is the backbone network of the denoising generation model, which is essentially a U-Net network. According to the calculation formula of the attention feature vector, the attention feature vector corresponding to step 1 can be calculated, as shown in the following equation:

[0146]

[0147] Step 2: In the forward and reverse diffusion process, the face identity structure feature C id Embed C with the original text orignAs a multimodal condition injected into the conditional diffusion model, the index matrix, content matrix, and query matrix of the cross attention layer are K orign 、V id , Q orign The content matrix V in the attention layer of the diffusion model id Controlling the overall structure of the face should be consistent with the forward process to avoid major changes in the structure of the face image. orign and content matrix Q orign It mainly controls the detailed attributes of the face, such as expressions, etc. These features are provided by the original text embedding. The sampling process is shown in the following equation:

[0148]

[0149] in, are the hidden vectors of the diffusion process of the t-1th and tth steps in step 2, express The variance of , e is the random noise that obeys the standard Gaussian distribution.

[0150] Similarly, according to the calculation formula of the attention feature vector, the attention feature vector corresponding to step 2 can be calculated, as shown in the following equation:

[0151]

[0152] In this embodiment, the attention loss function L is designed attention To further optimize the face identity structure feature C id Embed C with the original text orign Using attention reweighting, we can fine-tune the CLIP text encoder E without changing the overall structure of the image. T To improve the fine control editing of text features on local features of the image. attention The expression of is shown in Equation 11:

[0153]

[0154] where ||·||2 represents the L2 loss. Since the diffusion process requires z T The noise image z0 is obtained by adding noise in T steps, and then based on L ldm The loss is restored to the original latent vector z by predicting the noise of the U-Net model T In actual diffusion, the hidden vector z is reversed at each step t Slight errors will inevitably occur. After T steps, these accumulated errors will be gradually amplified, causing the reversed image to deviate from the original image. Therefore, this method proposes a latent vector reconstruction loss L recTo constrain each step of the diffusion process, by fine-tuning the CLIP text encoder E T Improve the ability of text features to represent image features. rec The expression of is shown in the following equation:

[0155]

[0156] In order to constrain the face identity encoder to more effectively extract the identity features and facial structure features of the original face, this method proposes an identity-aware loss L per and facial keypoint loss L landmank To train the model. per The expression of is shown in the following equation:

[0157]

[0158] Among them is I img Input face, I rec is the face reconstructed by the diffusion model. φ(·) is the pre-trained face recognition model VGGFace, l represents the lth layer of the VGGFace network, φ l (I img ) and φ l (I rec ) represent I img and I rec The output features after the lth layer of VGGFace. ijk represents the result of the activation output of the i-th convolution kernel at position (j, k) after passing through the intermediate layer. C l , H l , W l is the number of channels, height, and width of the feature map of the lth layer. per It focuses more on facial identity features and is closer to human visual perception. landmank Mainly to avoid the problem of facial structure key point distortion during the model generation process, L landmank The expression of is shown in the following equation:

[0159]

[0160] Where C represents the number of facial key points, and They represent the key points of the input face and the reconstructed face respectively, i is the key point index number, and ||·||1 represents the L1 loss.

[0161] The overall loss function expression of the first stage is shown in the following equation:

[0162] L=k1L attention +k2L rec +k3Lper +k4L landmark

[0163] Where k1, k2, k3, k4 are the attention loss L attention , latent vector reconstruction loss L rec , identity-aware loss L per , facial key point loss L landmark The weight coefficient of , in this embodiment, is 1.

[0164] The specific process of the second stage of training is as follows:

[0165] Fix the face identity encoding module and face identity structure feature C trained in the first stage id As the embedding of the content matrix, the original text and the guide text are input into the CLIP text encoder to obtain the original text embedding C orign Embed C with boot text target In order to avoid the addition of guide words affecting the syntactic and semantic properties of the original text and causing language drift, this method designs a semantic consistency loss L sem , by maximizing C target and C orign The cosine similarity between L sem The expression of is shown in the following equation:

[0166]

[0167] Where i represents the word vector in each text embedding, and β is a hyperparameter with a value of 0.5. orign After the first stage of training, the input face can be completely reconstructed through cross attention. It can be considered that C orign It contains most of the facial information of the original image, thus ensuring the high fidelity of the generated face.

[0168] S7, construct test evaluation module;

[0169] In this embodiment, the test evaluation module is mainly used for text-guided fine-grained face attribute editing, and uses relevant evaluation indicators to evaluate the quality of the generated model.

[0170] The specific process is as follows: Figure 5 As shown in Figure 1, load the trained model parameters and weights, fix the trained face identity encoding module, facial key point encoding module, CLIP text encoder, and conditional diffusion model parameters, and input the test set images into the network for face attribute editing. In order to achieve fine-grained editing based on guide words, this method adopts C target With C orignThe face image is gradually edited by linear interpolation. The text modal condition injected by the conditional diffusion model can be expressed as the following equation:

[0171] C insert =η×C target +(1-η)×C orign

[0172] Among them, η∈(0,1) represents the interpolation coefficient. Then, C id With C insert They are injected into the conditional diffusion model as image modality conditions and text modality conditions respectively. The sampling process is shown in the following equation:

[0173]

[0174] in, represents the latent vector of the t-th step diffusion process, express The variance of , t∈(0,1);

[0175] When t is 0, Represents the Gaussian noise of the input conditional diffusion model. When t is T, Indicates the output after T steps of sampling according to the interpolation text C insert Prompt the edited target image; when η=0, the model output is the target image, which is the input original image; when η=1, the model output is based on C target The edited image, in the process of η changing from 0 to 1, can achieve smooth and fine-grained editing of the input image;

[0176] This example evaluates the designed network through two dimensions: qualitative and quantitative:

[0177] (1) Qualitative evaluation: The experimental results are as follows: Figure 6As shown, from left to right, these are the input face images, and the images after face attribute editing under the interpolation coefficients η of 0.2, 0.4, 0.8, and 1.0 respectively. It can be seen that the text-guided multimodal face attribute editing method proposed in the present invention can achieve fine-grained face attribute editing for the five attributes of "beard", "age", "smile", "bangs", and "glasses" by guiding the text without changing the identity characteristics of the input face and the overall facial structure characteristics. By controlling η, the richness and delicacy of the generated attributes can be controlled, and at the same time, the mutual influence between different attributes during the editing process can be avoided. In order to further qualitatively evaluate the generation advantages of the present invention, a comparative experiment is set up to compare the results of the present invention with those of the latest generation model. The selected generation models include TediGAN based on the GAN network, the backbone network SD-Edit of Stable Diffusion, and the DreamSalon network. The experimental comparison results are shown in the figure. Figure 7 As shown, it can be seen that the present invention is superior to the existing models in maintaining the face identity and character background, and the generated face is more consistent with the characteristics of the input face and the conditions described by the guiding text.

[0178] (2) Quantitative evaluation: In this embodiment, FID, Clip-Score, and LPIPS are used to evaluate the edited face images. FID (Frechet Inception Distance) is used to evaluate the quality of images synthesized by different methods. The Inception v3 model is used to extract the features of the real image and the generated image, and the distance between the generated image and the real image feature distribution is output at a certain layer of the model. The expression of FID is shown in the following equation:

[0179]

[0180] where μ r and∑ r are the mean and covariance matrices of the real image, μ g and∑ g are the mean and covariance matrices of the generated image, respectively, and Tr represents the trace of the matrix. The lower the FID, the better the quality of the synthesized image.

[0181] CLIP-Score is calculated as the cosine similarity between the normalized image and text embeddings. The image and text are projected into a common feature space using a CLIP image encoder (such as a convolutional neural network) and a CLIP text encoder (such as a Transformer), respectively. A higher score generally indicates a higher consistency between the output image and the text caption. The expression of Clip-Score is shown in the following equation:

[0182]

[0183] Among them, f i 、f t represent the image feature vector and text feature vector respectively, · represents the dot product, and ||·||2 represents the L2 norm.

[0184] LPIPS is an indicator used to perceive the similarity between two images. It calculates the similarity in the feature space and can overcome the shortcomings of the traditional pixel-level similarity indicator. In this embodiment, VGGFace is used as the perception network, which is mainly used to evaluate the identity consistency between the edited face and the generated face. The expression of LPIPS is shown in the following equation:

[0185]

[0186] Among them, I1 and I2 represent real images and generated images, i represents the i-th layer of the VGGFace network, is the normalized representation of the feature vector of the input image I in the i-th layer of the VGGFace network, w i represents the weight coefficient, and ||·||2 represents the L2 norm.

[0187] This embodiment uses three commonly used indicators, FID, Clip-Score, and LPIPS, to evaluate the generated images from three dimensions: the quality of the generated images, the consistency between the generated images and the guide text, and the identity consistency between the generated face images and the input faces. The widely used generation models are selected for comparison, including the StyleMapGAN model and the TediGAN model based on the generative adversarial GAN ​​network, the SD-Edit network and the DreamSalon model based on the Stable Diffusion network. The experimental results of the three indicators are shown in the following table, and the bold data are the best results;

[0188] Table 1 Comparison of FID evaluation index results

[0189]

[0190] Table 2 Comparison of CLIP-Score evaluation index results

[0191]

[0192] Table 3 Comparison of LPIPS evaluation index results

[0193]

[0194] This embodiment conducts quantitative evaluation on five facial attributes respectively, and averages the five attributes to obtain the evaluation value of the whole image. It can be seen from Table 1, Table 2 and Table 3 that the values ​​of FID, CLIP-Score and LPIPS of the present invention are 129.56, 0.82 and 17.11 respectively, which are generally better than the existing generation model, and achieves excellent performance in the quality of generated images, the matching degree between text and image, and the consistency of the identity of the face in the generated image, which verifies the effectiveness of the present invention.

[0195] Example 2

[0196] This embodiment provides a text-guided fine-grained living face attribute editing system, which is used to implement the text-guided fine-grained living face attribute editing method of the above-mentioned embodiment 1. The system includes: a data set acquisition module, a data set preprocessing module, a face identity feature extraction module, a facial key point feature extraction module, a multi-scale feature mixing module, a text encoder, a conditional diffusion model construction module, a two-stage training module, a text interpolation module, and a face attribute editing picture output module;

[0197] In this embodiment, the data set acquisition module is used to acquire a face image data set and a text data set;

[0198] In this embodiment, the data set preprocessing module is used to divide the data set, unify the resolution size of the face image, and for each face image, split the text data set into description texts for the attribute according to different facial attributes;

[0199] In this embodiment, the face identity feature extraction module is used to extract multi-scale face identity features of face images;

[0200] In this embodiment, the facial key point feature extraction module is used to extract multi-scale facial key point features of the face image;

[0201] In this embodiment, the multi-scale feature mixing module is used to perform feature splicing on the multi-scale face identity features and the multi-scale facial key point features according to the corresponding scales to obtain the identity structure features, and perform weighted averaging on the identity structure features of each scale to obtain the face identity structure features;

[0202] In this embodiment, the text encoder is used to encode the text data into raw text embedding;

[0203] In this embodiment, the conditional diffusion model construction module is used to construct a conditional diffusion model, and the face image and the original text embedding are respectively injected into the conditional diffusion model as the image modality and the text modality, and the conditional diffusion model is trained;

[0204] In this embodiment, a two-stage training module is used to perform two-stage training. In the first stage of training, the face identity structure feature is used as an image modal condition, and the face identity structure feature and the original text embedding are injected into the conditional diffusion model as a multimodal condition, and the corresponding attention feature vector is calculated. The alignment degree of the face identity structure feature and the original text embedding is optimized based on the attention loss, and the parameters of the face identity feature extraction module and the facial key point feature extraction module are optimized by the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized by the latent vector reconstruction loss. In the second stage of training, the semantic consistency loss is used to optimize the semantic consistency between the guided text and the original text.

[0205] In this embodiment, the text interpolation module is used to obtain a final text condition vector by interpolating the original text and the guide text;

[0206] In this embodiment, the face attribute edited picture output module is used to input the final text conditional vector into the trained conditional diffusion model to generate a fine-grained face attribute edited picture.

[0207] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A text-guided fine-grained live face attribute editing method. It is characterized in that The steps include: Obtain a face image dataset and a text dataset, divide the dataset, unify the resolution of the face images, and for each face image, split the text dataset into descriptions of the attributes according to different facial attributes; Constructing a face identity feature extraction module and a facial key point feature extraction module, wherein the face identity feature extraction module extracts multi-scale face identity features of a face image, and the facial key point feature extraction module extracts multi-scale facial key point features of a face image; Construct a multi-scale feature mixing module, which combines multi-scale face identity features and multi-scale facial key point features according to corresponding scales to obtain identity structure features, and performs weighted average of identity structure features at each scale to obtain face identity structure features; The text data is passed through a text encoder to obtain the original text embedding, and the face image and the original text embedding are injected into the conditional diffusion model as the image modality and the text modality, respectively, to train the conditional diffusion model; Two-stage training is carried out. In the first stage of training, the face identity structure features are used as image modal conditions, and the face identity structure features and the original text embedding are injected into the conditional diffusion model as multimodal conditions. The corresponding attention feature vector is calculated, and the alignment degree of the face identity structure features and the original text embedding is optimized based on the attention loss. The parameters of the face identity feature extraction module and the facial key point feature extraction module are optimized by the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized by the latent vector reconstruction loss. In the second stage of training, the semantic consistency loss is used to optimize the semantic consistency between the guided text and the original text. The original text and the guide text are interpolated to obtain the final text conditional vector, which is input into the trained conditional diffusion model to generate a fine-grained face attribute editing image.

2. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that The face identity feature extraction module includes a multi-scale encoder and a pre-trained face identity feature extractor; The multi-scale encoder includes a convolution layer, a maximum pooling layer, a plurality of residual networks based on ResNet, and a pooling network, and the residual network based on ResNet extracts multi-scale features; The pre-trained face identity feature extractor includes a face identity feature extraction network based on ArcFace; The multi-scale encoder outputs a multi-scale feature vector f=[f 1 , f 2 , ..., f N ]; The multi-scale feature vector is input into the pre-trained face identity feature extractor to extract the multi-scale face identity feature f id =[f 1 id , f 2 id , ..., f N id ], the identity feature of each scale is expressed as: in, represents the face identity feature of the i-th scale, F Arc represents the face identity feature extraction network based on ArcFace, and N represents the size of the division scale.

3. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that The facial key point feature extraction module includes a pre-trained face key point extraction network PFLD and a facial key point adjustment network; The pre-trained face key point extraction network PFLD includes a convolutional layer, a Bottleneck module and a multi-scale fusion network. The feature representation of the face image extracted by the pre-trained face key point extraction network is: f sp =F Bottleneck (F DsConv (F Conv (I img ))) Among them, f sp represents the output feature of the Bottleneck module, F Bottleneck represents the Bottleneck module of the residual network, F DSConv represents a deep separable convolutional network, F Conv represents the convolutional network, I img Represents the input face image; The output feature f sp Input to the facial key point adjustment network, which has multiple convolution kernels of different sizes and outputs multi-scale facial key point features Among them, N represents the size of the division scale; The multi-scale face identity feature and the multi-scale facial key point feature have the same dimension.

4. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that The face identity structure feature is expressed as: Among them, C id represents the face identity structure feature, N represents the size of the division scale, Cancat represents the feature concatenation of multi-scale face identity features and multi-scale facial key point features, represents the face identity feature of the i-th scale, Represents the facial key point feature of the i-th scale.

5. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that The face image and the original text embedding are respectively injected into the conditional diffusion model as the image modality and the text modality, and the conditional diffusion model is trained, specifically including: Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network; The alignment of image semantics and text semantics at the latent space level is achieved by optimizing the similarity of the attention feature vectors of text and image. The attention feature vector is expressed as: Where K, Q, and V represent the index matrix, content matrix, and query matrix corresponding to the image modality and text modality, respectively. d is the projection dimension of the index matrix and query matrix. att Represents the output of the cross attention layer as the input of the convolutional layer of the conditional diffusion model; Each intermediate sample z of the conditional diffusion model t The following conditions must be met: Among them, α t represents the attenuation coefficient of the conditional diffusion model, z t-1 represents the intermediate sample of the t-1-step diffusion process, ∈ t is the noise that obeys Gaussian distribution; The conditional diffusion model is trained by minimizing the loss function, expressed as: Among them, ∈ θ is the U-Net network of the denoising generation model, ∈ is the noise that follows the standard Gaussian distribution, t represents the tth step of the diffusion process, C is the input modal feature vector, ||·|| 2 represents L2 loss, z 0 represents random noise.

6. The text-guided fine-grained living face attribute editing method according to claim 5, It is characterized in that Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated through a fully connected linear network, which are specifically expressed as: Q orign =W t Q ×C orign C orign =E T (T orign ) Among them, K orign 、V orign , Q orign K represents the index matrix, content matrix, and query matrix corresponding to the original text embedding. id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, W t Q , are six linear transformation matrices with different weights, T orign Represents the original text, E T Represents CLIP text encoder, C orign Represents the raw text embedding.

7. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that In the first stage of training, it is implemented in two steps: Step 1: In the forward and reverse diffusion processes, the face identity structure features are injected into the conditional diffusion model as image modal conditions. The sampling process is expressed as: Among them, α i is the attenuation coefficient of the conditional diffusion model, z t-1 、z t are the latent vectors of the diffusion process in the t-1th and tth steps in step 1, σ t Indicates z t The variance of e is the random noise that follows the standard Gaussian distribution, ∈ θ The U-Net network of the denoising generation model, C id represents the structural features of face identity, and t represents the tth step of the diffusion process; Calculate the corresponding attention feature vector, expressed as: Among them, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features; Step 2: In the forward and reverse diffusion processes, the face identity structure features and the original text embedding are injected into the conditional diffusion model as multimodal conditions. The index matrix, content matrix, and query matrix of the cross attention layer are K orign 、V id , Q orign , the sampling process is expressed as: in, are the hidden vectors of the diffusion process of the t-1th and tth steps in step 2, express The variance, C orign represents the original text embedding; Calculate the corresponding attention feature vector, expressed as: The overall loss function of the first stage training is expressed as: L=k 1 L attention +k 2 L rec +k 3 L per +k 4 L landmark Among them, k 1 , k 2 , k 3 , k 4 are the attention loss L attention , latent vector reconstruction loss L rec , identity-aware loss L per , facial key point loss L landmark The weight coefficient, E T Represents CLIP text encoder, C l , H l , W l is the number of channels, height, and width of the feature map of the lth layer, φ l (I img ) and φ l (I rec ) represent face images I img And the face I reconstructed by the conditional diffusion model rec The output features after the lth layer of VGGFace, ijk represents the result of the activation output of the i-th convolution kernel at position (j, k) after the intermediate layer, C represents the number of key points of the face, and Represent the key points of the input face and the reconstructed face, respectively, ||·|| 1 represents L1 loss.

8. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that In the second stage of training, the original text and the guide text are input into the CLIP text encoder to obtain the original text embedding and the guide text embedding. The semantic consistency between the guide text and the original text is optimized by the semantic consistency loss. The semantic consistency loss is expressed as: Among them, i represents the word vector in each text embedding, β is a hyperparameter, and C orign represents the original text embedding, C target Represents a guided text embedding.

9. The text-guided fine-grained living face attribute editing method according to claim 1, It is characterized in that The original text and the guide text are interpolated to obtain the final text condition vector, which is specifically expressed as: C insert =η×C target +(1-n)×C orign Among them, η∈(0,1) represents the interpolation coefficient, C orign represents the original text embedding, C target represents the guided text embedding, C insert represents the final text condition vector; The face identity structure features and the final text conditional vector are injected into the conditional diffusion model as image modality conditions and text modality conditions. The sampling process is expressed as: in, Respectively represent the latent vectors of the t-th and t-1-th step diffusion processes, express The variance of i is the attenuation coefficient of the conditional diffusion model, z t is the latent vector of the t-th step diffusion process, ∈ θ is the U-Net network of the denoising generation model, e is random noise that obeys the standard Gaussian distribution, When t is 0, Represents the Gaussian noise of the input conditional diffusion model. When t is T, It represents the target image after T-step sampling and editing according to the final text condition vector prompt; When η=0, the model output is the target image, that is, the input original image, and when η=1, the model output is the image edited based on the guided text embedding.

10. A text-guided fine-grained live face attribute editing system. It is characterized in that The system is used to implement the text-guided fine-grained live face attribute editing method described in any one of claims 1 to 9, comprising: a data set acquisition module, a data set preprocessing module, a face identity feature extraction module, a facial key point feature extraction module, a multi-scale feature mixing module, a text encoder, a conditional diffusion model construction module, a two-stage training module, a text interpolation module, and a face attribute editing picture output module; The data set acquisition module is used to acquire a face image data set and a text data set; The data set preprocessing module is used to divide the data set, unify the resolution size of the face image, and for each face image, split the text data set into description texts for the attribute according to different facial attributes; The face identity feature extraction module is used to extract multi-scale face identity features of face images; The facial key point feature extraction module is used to extract multi-scale facial key point features of the face image; The multi-scale feature mixing module is used to perform feature splicing on the multi-scale face identity features and the multi-scale facial key point features according to the corresponding scales to obtain the identity structure features, and to perform weighted averaging on the identity structure features of each scale to obtain the face identity structure features; The text encoder is used to encode text data into raw text embedding; The conditional diffusion model construction module is used to construct a conditional diffusion model, and inject the face image and the original text embedding into the conditional diffusion model as the image modality and the text modality respectively, and train the conditional diffusion model; The two-stage training module is used to perform two-stage training. In the first stage of training, the face identity structure feature is used as an image modal condition, and the face identity structure feature and the original text embedding are respectively injected into the conditional diffusion model as a multimodal condition, and the corresponding attention feature vector is calculated. The alignment degree of the face identity structure feature and the original text embedding is optimized based on the attention loss, and the parameters of the face identity feature extraction module and the facial key point feature extraction module are optimized by the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized by the latent vector reconstruction loss. In the second stage of training, the semantic consistency between the guided text and the original text is optimized by the semantic consistency loss. The text interpolation module is used to obtain a final text condition vector by interpolating the original text and the guide text; The face attribute edited picture output module is used to input the final text conditional vector into the trained conditional diffusion model to generate a fine-grained face attribute edited picture.

Citation Information

Patent Citations

  • Image editing method and device, equipment and storage medium

    CN118135062A

  • Method for single-view hair modeling and portrait editing

    US20140233849A1

  • Generating a modified digital image utilizing a human inpainting model

    US20240135511A1

Cited By

  • Face generation method based on key point guided text-based graph diffusion model

    CN119762610A

  • Facial expression migration method and system based on diffusion model and facial key points

    CN119784576A

  • Real-time face blind restoration method and device based on identity constraint and frequency domain enhancement

    CN120318122A

  • Three-dimensional face generation and expression editing method and system based on text picture driving

    CN120495532A

  • Text image driven three-dimensional face generation and expression editing method and system

    CN120495532B