Text-guided fine-grained living face attribute editing method and system

Abstract

The present invention discloses a method and system for fine-grained live facial attribute editing based on text guidance. The method comprises the following steps: obtaining a facial image dataset and a text dataset and performing data preprocessing; extracting multi-scale facial identity features and multi-scale facial key point features from the facial image using a facial identity feature extraction module and a facial key point feature extraction module; performing feature concatenation on the multi-scale facial identity features and multi-scale facial key point features using a multi-scale feature mixing module, and performing weighted averaging to obtain facial identity structural features; embedding the facial image and original text as image modality and text modality, respectively, into a conditional diffusion model; performing two-stage training; and obtaining a final text conditional vector based on the original text and the guiding text, inputting the final text conditional vector into the trained conditional diffusion model, and generating a fine-grained facial attribute editing image. The present invention achieves refinement and diversification of facial attribute editing.

Description

Technical Field

[0001] The present invention relates to the technical field of face attribute editing, and in particular to a text-guided fine-grained living face attribute editing method and system. Background Art

[0002] Facial attribute editing technology can generate diverse images based on existing real-world faces, for example, predicting the gradual aging process, simulating facial expressions, and simulating changes in hairstyle. By editing facial attributes, face liveness detection models can be trained on a large amount of facial data from the same individual in different scenarios. This significantly improves the generalization and robustness of existing face liveness detection models, preventing misidentification in unknown scenarios. Furthermore, it reduces the workload of manually collecting facial data and protects against data leakage, addressing privacy concerns. Furthermore, by controlling the variation of specific facial attributes, facial scenarios can be simulated under varying facial attributes, helping to better understand the working mechanisms of face liveness detection models, such as the sensitivity of face recognition systems to facial attributes and the impact of specific facial edits on recognition results. Therefore, facial attribute editing technology has significant practical significance, with key applications in areas such as access control systems, bank and mobile payment verification systems, and urban surveillance systems.

[0003] Early facial attribute editing used generative adversarial networks such as GAN or VAE, combined with attribute labels or text descriptions to modify specific style vectors in the latent space to achieve editing of specific attributes. Existing research methods mostly use CLIP text encoders and image encoders, introducing multimodal conditions into the diffusion model to generate high-resolution images, which can achieve convenient and efficient attribute modification based on natural language descriptions, improving the diversity and fidelity of generated images.

[0004] Existing text-guided facial attribute editing algorithms still have some shortcomings. They typically use a general image encoder to encode the input face, ignoring the unique identity characteristics of the face and the consistency of facial structural features during the generation process. Second, existing algorithms lack text that accurately describes facial attributes. A piece of text often contains many descriptions of facial attributes, and editing one attribute often leads to changes in other attributes. Furthermore, coarse-grained text descriptions often cannot represent the fine-grained information in the image, resulting in an inability to accurately edit the image and a lack of diversity in the generated text. Therefore, how to use text to achieve refined, fine-grained, and diverse editing of facial attributes while maintaining the consistency of facial identity is a research focus and difficulty in this field. Summary of the Invention

[0005] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a text-guided fine-grained living face attribute editing method and system. The present invention utilizes a face identity feature extraction module and a facial key point feature extraction module to obtain face identity features and facial structure features, and adopts a multi-scale feature extraction strategy so that the global features and local features of the face can be fully represented, wherein the low-level features contain more position and detail information, while the high-level features contain stronger deep semantic features of the image; utilizing a cross-attention adjustment module based on a conditional diffusion model, the input image is reconstructed using image modality and text modality respectively, the similarity of the image modality and text modality attention feature vectors is optimized, and the degree of alignment between text and image at the latent space level is achieved; the interpolation of the original text encoding and the guided text encoding is utilized as the query matrix and content matrix, and cross-attention is performed with the face identity structure feature index matrix, and the face attribute editing based on the guide word is gradually realized through the conditional diffusion model, thereby realizing the refinement and diversification of attribute editing.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a text-guided fine-grained living face attribute editing method, comprising the following steps:

[0008] Obtain a facial image dataset and a text dataset, divide the dataset, unify the resolution of the facial images, and for each facial image, split the text dataset into descriptions of the attributes based on different facial attributes.

[0009] Constructing a face identity feature extraction module and a facial key point feature extraction module, wherein the face identity feature extraction module extracts multi-scale face identity features of a face image, and the facial key point feature extraction module extracts multi-scale facial key point features of a face image;

[0010] Construct a multi-scale feature mixing module, which combines multi-scale face identity features and multi-scale facial key point features according to the corresponding scales to obtain identity structure features. The identity structure features of each scale are weighted averaged to obtain the face identity structure features.

[0011] The text data is passed through a text encoder to obtain the original text embedding. The face image and the original text embedding are injected into the conditional diffusion model as the image modality and text modality respectively, and the conditional diffusion model is trained.

[0012] The training is carried out in two stages. In the first stage of training, the facial identity structure features are respectively injected into the conditional diffusion model as image modal conditions and the facial identity structure features and original text embeddings are respectively injected into the conditional diffusion model as multimodal conditions. The corresponding attention feature vector is calculated, and the alignment degree of the facial identity structure features and original text embeddings is optimized based on the attention loss. The parameters of the facial identity feature extraction module and the facial key point feature extraction module are optimized using the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized using the latent vector reconstruction loss. In the second stage of training, the semantic consistency loss is used to optimize the semantic consistency between the guided text and the original text.

[0013] The original text and the guide text are interpolated to obtain the final text conditional vector, which is input into the trained conditional diffusion model to generate fine-grained facial attribute editing images.

[0014] As a preferred technical solution, the face identity feature extraction module includes a multi-scale encoder and a pre-trained face identity feature extractor;

[0015] The multi-scale encoder includes a convolutional layer, a maximum pooling layer, multiple ResNet-based residual networks, and a pooling network, and the ResNet-based residual network extracts multi-scale features;

[0016] The pre-trained face identity feature extractor includes a face identity feature extraction network based on ArcFace;

[0017] The multi-scale encoder outputs a multi-scale feature vector f=[f 1 , f 2 ,...,f N ];

[0018] The multi-scale feature vector is input into the pre-trained face identity feature extractor to extract the multi-scale face identity feature f id =[f 1 id , f 2 id ,...,f N id ], the identity feature of each scale is expressed as:

[0019]

[0020] in, represents the face identity feature of the i-th scale, F Arc represents the face identity feature extraction network based on ArcFace, and N represents the size of the division scale.

[0021] As a preferred technical solution, the facial key point feature extraction module includes a pre-trained face key point extraction network PFLD and a facial key point adjustment network;

[0022] The pre-trained facial key point extraction network PFLD includes a convolutional layer, a Bottleneck module and a multi-scale fusion network. The feature representation of the facial image extracted by the pre-trained facial key point extraction network is:

[0023] f sp =F Bottleneck (F DSConv (F Conv (I img )))

[0024] Among them, f sp Represents the features output by the Bottleneck module, F Bottleneck Represents the Bottleneck module of the residual network, F DSConv represents a depthwise separable convolutional network, F Conv represents the convolutional network, I img Represents the input face image;

[0025] Output feature f sp Input to the facial key point adjustment network, the facial key point adjustment network has multiple convolution kernels of different sizes and outputs multi-scale facial key point features Among them, N represents the size of the division scale;

[0026] The multi-scale face identity feature and the multi-scale facial key point feature have the same dimension.

[0027] As a preferred technical solution, the facial identity structure feature is expressed as:

[0028]

[0029] Among them, C id Represents the face identity structure feature, N represents the size of the division scale, Cancat represents the feature splicing of multi-scale face identity features and multi-scale facial key point features, represents the face identity feature of the i-th scale, represents the facial key point feature of the i-th scale.

[0030] As a preferred technical solution, facial images and original text are embedded as image modalities and text modalities respectively and injected into the conditional diffusion model, and the conditional diffusion model is trained, specifically including:

[0031] Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network;

[0032] The alignment of image semantics and text semantics at the latent space level is achieved by optimizing the similarity of the attention feature vectors of text and image. The attention feature vector is expressed as:

[0033]

[0034] Where K, Q, and V represent the index matrix, content matrix, and query matrix corresponding to the image modality and text modality, respectively. d is the projection dimension of the index matrix and query matrix. att Represents the output of the cross attention layer as the input of the convolutional layer of the conditional diffusion model;

[0035] Each intermediate sample z of the conditional diffusion model t The following conditions must be met:

[0036]

[0037] Among them, α t represents the attenuation coefficient of the conditional diffusion model, z t-1 represents the intermediate sample of the t-1 step diffusion process, ∈ t is the noise that obeys Gaussian distribution;

[0038] The conditional diffusion model is trained by minimizing the loss function, which is expressed as:

[0039]

[0040] Among them, ∈ θ The U-Net network of the denoising generation model, ∈ is the noise obeying the standard Gaussian distribution, t represents the tth step of the diffusion process, C is the input modal feature vector, ||·||2 represents the L2 loss, and z0 represents random noise.

[0041] As a preferred technical solution, given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network, which are specifically expressed as:

[0042] K orign =W t K ×C orign

[0043]

[0044] V orign =W t V ×C orign

[0045] V id =W p V ×C id

[0046] Q orign =W t Q ×C orign

[0047] Q id =W p Q ×C id

[0048] C orign =E T (T orign )

[0049] Among them, K orign 、V orign , Q orign Represents the index matrix, content matrix, and query matrix corresponding to the original text embedding, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, W t K 、W p K 、W t V 、W p V 、W t Q 、W p Q are six linear transformation matrices with different weights, T orign Represents the original text, E T Represents CLIP text encoder, C orign Represents raw text embedding.

[0050] As a preferred technical solution, the first stage of training is implemented in two steps:

[0051] Step 1: In the forward and reverse diffusion processes, the facial identity structure features are injected into the conditional diffusion model as image modality conditions. The sampling process is expressed as:

[0052]

[0053] Among them, α i is the attenuation coefficient of the conditional diffusion model, z t-1 、xt They are the latent vectors of the diffusion process of step t-1 and step t in step 1, σ t Represents x t The variance of e is the random noise that obeys the standard Gaussian distribution, ∈ θ The U-Net network of the denoising generation model, C id represents the structural features of facial identity, and t represents the tth step of the diffusion process;

[0054] Calculate the corresponding attention feature vector, expressed as:

[0055]

[0056] Among them, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the facial identity structure features;

[0057] Step 2: In the forward and reverse diffusion processes, the facial identity structure features and the original text embedding are injected into the conditional diffusion model as multimodal conditions. The index matrix, content matrix, and query matrix of the cross attention layer are K orign 、V id , Q orign , the sampling process is expressed as:

[0058]

[0059] in, are the latent vectors of the diffusion process of step t-1 and step t in step 2, express The variance, C orign represents the original text embedding;

[0060] Calculate the corresponding attention feature vector, expressed as:

[0061]

[0062] The overall loss function of the first stage training is expressed as:

[0063] L=k1L attention +k2L rec +k3L per +k4L landmark

[0064]

[0065] Among them, k1, k2, k3, and k4 are the attention loss L attention , latent vector reconstruction loss L rec , identity-aware loss L per , facial key point loss L landmark The weight coefficient, E T Represents CLIP text encoder, C l 、H l 、W l is the number of channels, height, and width of the feature map of the first layer, φ l (I img ) and φ l (I rec ) represent face images I img And the face I reconstructed by the conditional diffusion model rec The output features after the lth layer of VGGFace, ijk represents the result of the activation output of the i-th convolution kernel at position (j, k) after the intermediate layer, C represents the number of key points of the face, and They represent the key points of the input face and the reconstructed face respectively, and ||·||1 represents the L1 loss.

[0066] As a preferred technical solution, in the second stage of training, the original text and the guide text are input into the CLIP text encoder to obtain the original text embedding and the guide text embedding. The semantic consistency loss is used to optimize the semantic consistency between the guide text and the original text. The semantic consistency loss is expressed as:

[0067]

[0068] Among them, i represents the word vector in each text embedding, β is a hyperparameter, and C orign represents the original text embedding, C target Represents guided text embedding.

[0069] As a preferred technical solution, the original text and the guide text are interpolated to obtain the final text condition vector, which is specifically expressed as:

[0070] C insert =C×C target +(1-η)×C orign

[0071] Among them, η∈(0,1) represents the interpolation coefficient, C orign represents the original text embedding, C target represents the guided text embedding, C insert represents the final text condition vector;

[0072] The facial identity structure features and the final text conditional vector are injected into the conditional diffusion model as image modality conditions and text modality conditions. The sampling process is expressed as:

[0073]

[0074] in, Represent the latent vectors of the diffusion process of step t and t-1 respectively, express The variance of α i is the attenuation coefficient of the conditional diffusion model, z t is the latent vector of the t-th step diffusion process, ∈ θ is the U-Net network of the denoising generation model, e is the random noise that obeys the standard Gaussian distribution, when t is 0, Represents the Gaussian noise of the input conditional diffusion model. When t is T, It represents the target image after T-step sampling and editing according to the final text condition vector prompt;

[0075] When η=0, the model output is the target image, that is, the input original image. When η=1, the model output is the image edited based on the guided text embedding.

[0076] The present invention also provides a text-guided fine-grained living face attribute editing system for implementing the above-mentioned text-guided fine-grained living face attribute editing method. The system comprises: a data set acquisition module, a data set preprocessing module, a face identity feature extraction module, a facial key point feature extraction module, a multi-scale feature mixing module, a text encoder, a conditional diffusion model construction module, a two-stage training module, a text interpolation module, and a face attribute editing image output module;

[0077] The data set acquisition module is used to acquire a face image data set and a text data set;

[0078] The dataset preprocessing module is used to divide the dataset, unify the resolution size of the face image, and for each face image, split the text dataset into description texts for different facial attributes;

[0079] The face identity feature extraction module is used to extract multi-scale face identity features of face images;

[0080] The facial key point feature extraction module is used to extract multi-scale facial key point features of the face image;

[0081] The multi-scale feature mixing module is used to perform feature splicing on the multi-scale face identity features and the multi-scale facial key point features according to the corresponding scales to obtain the identity structure features, and perform weighted averaging on the identity structure features of each scale to obtain the face identity structure features;

[0082] The text encoder is used to encode text data into raw text embedding;

[0083] The conditional diffusion model construction module is used to construct a conditional diffusion model, embedding the face image and the original text as image modality and text modality respectively into the conditional diffusion model, and training the conditional diffusion model;

[0084] The two-stage training module is used to perform two-stage training. During the first stage of training, the facial identity structure features are respectively used as image modal conditions, and the facial identity structure features and the original text embedding are respectively injected into the conditional diffusion model as multimodal conditions, and the corresponding attention feature vectors are calculated. The alignment degree of the facial identity structure features and the original text embedding is optimized based on the attention loss. The parameters of the facial identity feature extraction module and the facial key point feature extraction module are optimized through the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized through the latent vector reconstruction loss. During the second stage of training, the semantic consistency between the guided text and the original text is optimized through the semantic consistency loss.

[0085] The text interpolation module is used to obtain a final text condition vector by interpolating the original text and the guide text;

[0086] The face attribute edited picture output module is used to input the final text condition vector into the trained conditional diffusion model to generate a fine-grained face attribute edited picture.

[0087] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0088] (1) The present invention extracts multi-scale face identity features and multi-scale face key point features of face images based on a face identity feature extraction module and a facial key point feature extraction module, respectively, obtains face identity structure features, solves the problem of mismatch between the generated face and the input face identity, and retains the facial structure of the input face.

[0089] (2) Based on face datasets and text datasets, the present invention conducts separate two-stage training for each attribute in the text description of different attributes, ensuring that each face attribute does not interfere with each other, facilitating the precise control of the generation of face attributes by text, and improving the reliability and stability of the generated image.

[0090] (3) The present invention reconstructs the input image through two modalities, namely, picture and original text, and uses the cross-attention mechanism and reconstruction loss to achieve alignment of text and image at the latent space level, which can improve the accuracy of text-generated pictures. In the process of face attribute editing, the present invention adopts a progressive method of original text and guided text interpolation, which can gradually edit face attributes in a fine-grained manner and continuously enrich the details of the image. For example, the attribute "beard" can be gradually added to the face, and the generated beard transitions from thick to dense, and the generated image is more diverse. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is a structural diagram of the face identity feature extraction module of the present invention;

[0092] Figure 2 Schematic diagram of the structure of the facial key point feature extraction module of the present invention;

[0093] Figure 3 This is a schematic diagram of the training process in the first stage of the present invention;

[0094] Figure 4 This is a schematic diagram of the training process in the second stage of the present invention;

[0095] Figure 5 This is a schematic diagram of the face attribute editing and interpolation process in the testing phase of the present invention;

[0096] Figure 6 Editing and interpolating the five facial attributes in the test phase of the present invention to generate a facial schematic diagram;

[0097] Figure 7 This is a schematic diagram comparing the effects of face attribute editing during the testing phase of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0099] Example 1

[0100] This example uses the celebrity facial image dataset CelebA-HQ and the corresponding text dataset CelebA-Dialog for training and testing as examples to detail the specific implementation of the present invention. The CelebA-HQ dataset consists of high-resolution facial images and corresponding 40 attribute labels, encompassing over 1,000 celebrities and various facial attributes such as age, gender, hairstyle, and expression, totaling 30,000 images. The CelebA-Dialog dataset is used to study how to edit and generate facial images using natural language descriptions. This dataset provides rich and highly granular natural language descriptions for the 30,000 facial images in CelebA-HQ, including descriptions for five attributes: "bangs," "eyeglasses," "no beard," "smiling," and "age," as well as an overall caption ("overall_caption"). The present invention complies with the laws and regulations of relevant countries and regions regarding the acquisition and use of privacy, portrait, and other information.

[0101] This embodiment is deployed on the Linux system, based on the Pytorch1.7.0 deep learning framework, the configuration environment is Anaconda, the hardware configuration is two NVIDIA GeForce RTX 3090Ti, the graphics card driver is Driver Version: 470.239.06, the CUDA version is 11.4, the running container is Conda, and the main dependent libraries are opencv, imageio, pytorch-lightning, transformers, omegaconf, etc.

[0102] like Figure 1 、 Figure 2 As shown, this embodiment provides a text-guided fine-grained living face attribute editing method, including the following steps:

[0103] S1: data preprocessing;

[0104] In this embodiment, the image dataset and the text dataset are divided into a training set, a test set, and a validation set in a ratio of 7:2:1. The five attribute descriptions in the text dataset CelebA-Dialog are split using the json toolkit and saved as captions_hq_Bangs.json, captions_hq_Beard.json, captions_hq_Eyeglasses.json, captions_hq_Age.json, and captions_hq_Eyeglasses.json files. The resolution of the face image is adjusted to H×W×C, where H, W, and C are the image height, width, and number of channels, respectively. In this embodiment, the values ​​are H=512, W=512, and C=3, respectively.

[0105] S2: Build a face identity feature extraction module;

[0106] In this embodiment, the face identity feature extraction module is mainly used to extract the multi-scale identity features of the face. The main structure includes a multi-scale encoder F Φ and pre-trained F Arc Face identity feature extractor. The multi-scale encoder F Φ It mainly includes a 7×7 convolution layer, a 3×3 maximum pooling layer, 4 residual networks based on ResNet, a pooling network, and F Arc The face identity feature extractor includes a face identity feature extraction network based on ArcFace;

[0107] The specific process is as follows: the input face image resolution is H×W×C. First, the image is input into a convolutional neural network with a convolution kernel size of 7×7, a step size of 1, a padding of 3, and a channel number of 256. Then it is input into a 3×3 maximum pooling network to obtain a size of The initial features are then input into four residual networks to extract multi-scale features. The convolution kernels used in the residual network are of two types: 1×1 and 3×3. The 1×1 convolution kernel is used to change the number of output channels, and the 3×3 convolution changes the output feature size. BatchNorm is used as batch normalization, and the mean and standard deviation of the small batch are used to make the intermediate output values ​​of each layer of the neural network more stable. The activation function uses the RELU function. After the residual network, the sizes of The multi-scale features are further input into the pooling network, which includes global average pooling, global maximum pooling, activation network and 1×1 convolution layer. Global maximum pooling is mainly used to extract the most relevant features in each channel and eliminate redundant data. Global average pooling is mainly used to uniformly obtain all features in the feature map to minimize feature loss. The activation network mainly uses Sigmoid function and 1×1 convolution layer to unify the number of feature channels of each scale to 256. After the multi-scale encoder F Φ After that, the multi-scale feature vector f of the input image is obtained, f=[f 1 , f 2 ,...,f N ], where N represents the size of the division scale. In this embodiment, N=4 is preferred, i.e., f=[f 1 , f 2 , f 3 , f 4 ], the number of feature vector channels at each scale is 256.

[0108] These multi-scale features are then fed into the pre-trained F Arc The face identity feature extractor extracts the multi-scale identity features of the face f id =[f 1 id , f 2 id , f 3 id , f 4 id ], the identity characteristics of each scale are shown in the following equation:

[0109]

[0110] where f i id represents the facial identity feature of the i-th scale; the key to ArcFace is to design the angle interval loss function to introduce additional intervals in the angle space, which enhances the compactness within the class and the difference between classes, and improves the ability to extract facial identity features.

[0111] S3, building a facial key point feature extraction module;

[0112] In this embodiment, the facial landmark feature extraction module constrains the geometric structure of the face through a PFLD network, predicting 68 facial landmarks to simultaneously preserve the overall facial pose and local structural features. The main structure includes a pre-trained facial landmark extraction network (PFLD) and a facial landmark adjustment network. The PFLD network includes a 3×3 convolutional layer, a Bottleneck module, and a multi-scale fusion network to fuse with the multi-scale identity features extracted by the first branch. The entire process can be expressed as follows:

[0113] f sp =F Bottleneck (F DSConv (F Conv (I img )))

[0114] Among them, I ing represents the input face image, F conv represents the ordinary convolutional network, F DSConv represents a depthwise separable convolutional network, F Bottleneck It is the Bottleneck module of the residual network.

[0115] The specific process is as follows: the input face image resolution is H×W×C. First, the image is input into a convolutional neural network with a convolution kernel size of 3×3, a step size of 1, a padding of 3, and a channel number of 256. Then, it is input into a deep separable convolutional network to obtain a size of Initial features. In depthwise separable convolution, each channel of the image uses a separate convolution kernel to process information, which mainly includes two steps: Point Convlution and Depthwise Convlution. Point Convlution refers to the use of a 1×1 convolution kernel to process cross-channel information, and multiple 1×1 convolution kernels can be used to obtain multiple output feature maps. Depthwise Convlution refers to the use of a 3×3 convolution kernel to process only the information of a certain channel of the input image, that is, the spatial information in the length and width directions. Depthwise separable convolution greatly reduces the number of parameters and computations, and improves the efficiency of convolution operations. Then the initial features are input into the Bottleneck module of the residual network, which includes 6 ResNet128 and 2 ResNet16 Bottleneck layers, and the size is obtained. The characteristic f sp .

[0116] After that, the facial key point features output by the PFLD network are input into the facial key point adjustment network. The main structure of the facial key point adjustment network includes 7×7, 5×5, 3×3, and 1×1 convolution kernels, a convolutional neural network with 256 channels, and an output size of Multi-scale facial key point features Wherein, N represents the size of the division scale. In this embodiment, N=4 is preferred, i.e. Multi-scale facial key point features f′ sp and multi-scale identity features f id have the same dimensions.

[0117] S4, construct a multi-scale feature mixing module;

[0118] In this embodiment, the multi-scale feature mixing module obtains the face identity structure feature by fusing the identity feature and the facial key point feature. First, the multi-scale feature mixing module combines the face identity feature f id =[f 1 id , f 2 id , f 3 id , f 4 id ] and facial key point features The features are spliced ​​according to the corresponding scale as the identity structure feature at that scale. Then, the identity structure feature of each scale is weighted averaged to obtain the face identity structure feature C id , which contains the unique identity features of the entire face and the facial geometric structure features. The whole process can be expressed by the following equation:

[0119]

[0120] Where α is the weight coefficient, which is set to 0.5 in this embodiment, assuming that identity and facial structure features are equally important. N represents the size of the division scale, which is set to 4 in this embodiment. Cancat represents the feature splicing of facial identity features and facial key point features at different scales.

[0121] S5. Construct a cross-attention adjustment module based on the conditional diffusion model;

[0122] In this embodiment, the main function of this module is to optimize the alignment between the original text embedding and the input face image in the latent space. First, the original text is passed through the text encoder to obtain the original text embedding. This process can be expressed as the following equation:

[0123] C orign =E T (T orign )

[0124] Among them, T orign Represents the original text, E T Represents CLIP text encoder, C orignrepresents the original text embedding;

[0125] The conditional diffusion model implements text-guided image generation by inputting text prompts into the cross-attention layer. Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network. The process can be expressed as the following equation:

[0126] K orign =W t K ×C orign

[0127] K id =W p K ×C id

[0128] V orign =W t V ×C orign

[0129] V id =W p V ×C id

[0130] Q orign =W t Q ×C orign

[0131] Q id =W p Q ×C id

[0132] Among them, K orign 、V orign , Q orign Represents the index matrix, content matrix, and query matrix corresponding to the original text embedding, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features; W t K 、W p K 、W t V 、W p V 、W t Q 、W p Q are six linear transformation matrices with different weights;

[0133] Furthermore, in the two-stage training, the similarity of the attention feature vectors of text and image is optimized to achieve the alignment of image semantics and text semantics at the latent space level. The calculation of the attention feature vector is shown in the following equation:

[0134]

[0135] Where K, Q, and V represent the index matrix, content matrix, and query matrix corresponding to the image modality and text modality, respectively. d is the projection dimension of the index matrix and query matrix. att is the output of the cross attention layer, which is used as the input of the subsequent diffusion model convolution layer;

[0136] The conditional diffusion model is essentially a denoising generative model that trains the generative model to predict image z from random noise z0. T , each intermediate sample z t (t=0, 1, 2, ...T) all satisfy the following equation:

[0137]

[0138] Where T represents the number of diffusion steps, ɑ t is the attenuation coefficient of the conditional diffusion model, ∈ t is the noise that obeys the standard Gaussian distribution;

[0139] The input face image and original text embedding are injected into the conditional diffusion model as image modality and text modality respectively. The conditional diffusion model is trained by minimizing the following loss function:

[0140]

[0141] Among them, ∈ θ It is the backbone network of the denoising generative model, which is essentially a U-Net network. ∈ is the noise that obeys the standard Gaussian distribution. The value range of t is [0, T], which indicates the diffusion of the tth step. C is the input modality feature vector. This method includes two modalities: image and text. ||·||2 represents the L2 loss.

[0142] S6, construct a two-stage training module;

[0143] In this embodiment, a two-stage training module is constructed to enable accurate editing of facial attributes. The first stage aims to align the facial image with the original text prompt at the latent space level, and the second stage aims to add new guide word embeddings as guide text based on the original text, and implement facial attribute editing based on the description of the guide words. The two-stage training process is as follows: Figure 3 、 Figure 4 shown.

[0144] The training process uses the Xavier parameter initialization method to initialize model parameters. A mini-batch gradient descent optimization algorithm is used with a batch size of 2. The Adam optimizer is used, with an initial learning rate (lr) of 0.001 for the first epoch and 1e-6 for the second epoch. The weight decay parameter is 0.0006, and the momentum parameter is set to 0.9. The learning rate is reduced by 20% every 50 epochs.

[0145] The first phase is implemented in two steps, as follows:

[0146] Step 1: In the forward and reverse diffusion process, the face identity structure feature C id As the image modality condition, it is injected into the conditional diffusion model. The index matrix, content matrix, and query matrix of the cross attention layer are K id 、V id , Q id , d represents the projection dimension of the index matrix and the query matrix. The sampling process is shown in the following equation:

[0147]

[0148] where α i is the attenuation coefficient of the conditional diffusion model, z t-1 、z t They are the latent vectors of the diffusion process of step t-1 and step t in step 1, σ t represents z t The variance of , e is the random noise that obeys the standard Gaussian distribution, ε θ This is the backbone network of the denoising generation model, which is essentially a U-Net network. According to the calculation formula of the attention feature vector, the attention feature vector corresponding to step 1 can be calculated as shown in the following equation:

[0149]

[0150] Step 2: In the forward and reverse diffusion process, the face identity structure feature C id Embed C with the original text orign As a multimodal condition injected into the conditional diffusion model, the index matrix, content matrix, and query matrix of the cross attention layer are K orign 、V id , Q orign The content matrix V in the attention layer of the diffusion model id Controlling the overall structure of the face should be consistent with the forward process to avoid significant changes in the face image structure. orign and content matrix Qorign It mainly controls the detailed attributes of the face, such as expression, etc. These features are provided by the original text embedding. The sampling process is shown in the following equation:

[0151]

[0152] in, are the latent vectors of the diffusion process of step t-1 and step t in step 2, express The variance of , e is the random noise that obeys the standard Gaussian distribution.

[0153] Similarly, according to the calculation formula of the attention feature vector, the attention feature vector corresponding to step 2 can be calculated as shown in the following equation:

[0154]

[0155] In this embodiment, the attention loss function L is designed attention To further optimize the facial identity structure feature C id Embed C with the original text orign Using attention reweighting, we can fine-tune the CLIP text encoder E without changing the overall structure of the image. T To improve the fine control editing of text features on local features of images. attention The expression of is shown in Equation 11:

[0156]

[0157] Where ||·||2 represents the L2 loss. Since the diffusion process requires z T The noise image z0 is obtained by adding noise in T steps, and then based on L ldm The loss is restored to the original latent vector z by the U-Net model prediction noise T In actual diffusion, the hidden vector z is reversed at each step t Slight errors will inevitably occur. After T steps, these accumulated errors will be gradually amplified, causing the reversed image to deviate from the original image. Therefore, this method proposes the latent vector reconstruction loss L rec To constrain each step of the diffusion process, by fine-tuning the CLIP text encoder E T Improve the ability of text features to represent image features. rec The expression is shown in the following equation:

[0158]

[0159] In order to constrain the face identity encoder to more effectively extract the identity features and facial structure features of the original face, this method proposes an identity-aware loss Lper and facial keypoint loss L landmank To train the model. per The expression is shown in the following equation:

[0160]

[0161] Among them is I img Input face, I rec is the face reconstructed by the diffusion model. φ(·) is the pre-trained face recognition model VGGFace, l represents the lth layer of the VGGFace network, φ l (I img ) and φ l (I rec ) represent I img and I rec The output features of the lth layer of VGGFace. ijk represents the result of the activation output of the i-th convolution kernel at position (j, k) after passing through the intermediate layer. C l 、H l 、W l L is the number of channels, height, and width of the feature map of the first layer. per It focuses more on facial identity features and is closer to human visual perception. landmank Mainly to avoid the problem of facial structure key point distortion during the generation process of the model, L landmank The expression is shown in the following equation:

[0162]

[0163] Where C represents the number of facial key points, and They represent the key points of the input face and the reconstructed face respectively, i is the key point index number, and ||·||1 represents the L1 loss.

[0164] The overall loss function expression of the first stage is shown in the following equation:

[0165] L=k1L attention +k2L rec +k3L per +k4L landmark

[0166] Where k1, k2, k3, k4 are the attention loss L attention , latent vector reconstruction loss L rec , identity-aware loss L per , facial key point loss L landmark The weight coefficient of , in this embodiment, is 1.

[0167] The specific process of the second stage of training is as follows:

[0168] Fix the face identity encoding module and face identity structure feature C trained in the first stage id As the embedding of the content matrix, the original text and the guide text are input into the CLIP text encoder to obtain the original text embedding C orign Embed C with boot text target In order to avoid the addition of guide words affecting the syntactic and semantic properties of the original text and causing language drift, this method designs the semantic consistency loss L sem , by maximizing C target and C orign The cosine similarity between L sem The expression of is shown in the following equation:

[0169]

[0170] Where i represents the word vector in each text embedding, β is a hyperparameter with a value of 0.5. orign After the first stage of training, the input face can be completely reconstructed through cross attention, and it can be considered that C orign It contains most of the facial information of the original image, thus ensuring the high fidelity of the generated face.

[0171] S7. Construct test evaluation module;

[0172] In this embodiment, the test evaluation module is mainly used for fine-grained face attribute editing based on text guidance, and uses relevant evaluation indicators to evaluate the quality of the generated model.

[0173] The specific process is as follows: Figure 5 As shown in Figure 1, load the trained model parameters and weights, fix the trained face identity encoding module, facial key point encoding module, CLIP text encoder, and conditional diffusion model parameters, and input the test set images into the network for face attribute editing. In order to achieve fine-grained editing based on guide words, this method adopts C target with C orign The face image is gradually edited by linear interpolation. The text modal condition injected by the conditional diffusion model can be expressed as the following equation:

[0174] C insert =η×C target +(1-η)×C orign

[0175] Among them, η∈(0,1) represents the interpolation coefficient, and then C id with C insertThey are injected into the conditional diffusion model as image modal conditions and text modal conditions respectively. The sampling process is shown in the following equation:

[0176]

[0177] in, represents the latent vector of the t-th step diffusion process, express The variance of , t∈(0,1);

[0178] When t is 0, Represents the Gaussian noise of the input conditional diffusion model. When t is T, Indicates the output after T-step sampling based on the interpolation text C insert Prompt the edited target image; when η=0, the model output is the target image, which is the input original image; when η=1, the model output is based on C target The edited image can achieve smooth and fine-grained editing of the input image when η changes from 0 to 1;

[0179] This example evaluates the designed network using both qualitative and quantitative dimensions:

[0180] (1) Qualitative evaluation: The experimental results are as follows: Figure 6 As shown, from left to right, these are the input face images, and the images after face attribute editing under the interpolation coefficients η of 0.2, 0.4, 0.8, and 1.0 respectively. It can be seen that the multimodal face attribute editing method based on text guidance proposed in the present invention can guide the text, without changing the identity features of the input face and the overall facial structure features, to achieve fine-grained face attribute editing for the five attributes of "beard", "age", "smile", "bangs", and "glasses". By controlling η, the richness and delicacy of the generated attributes can be controlled, and at the same time, the mutual influence between different attributes during the editing process can be avoided. In order to further qualitatively evaluate the generation advantages of the present invention, a comparative experiment was set up to compare the results of the present invention with the results of the latest generation model. The selected generation models include TediGAN based on the GAN network, the backbone network SD-Edit of Stable Diffusion, and the DreamSalon network. The experimental comparison results are shown in the figure below. Figure 7 As shown, it can be seen that the present invention is superior to existing models in maintaining facial identity and character background, and the generated face is more consistent with the characteristics of the input face and the conditions described by the guiding text.

[0181] (2) Quantitative Evaluation: In this example, FID, Clip-Score, and LPIPS are used to evaluate the edited face images. FID (Frechet Inception Distance) is used to evaluate the quality of images synthesized by different methods. The Inception v3 model is used to extract features of the real image and the generated image. The distance between the feature distribution of the generated image and the real image is output at a certain layer of the model. The expression of FID is shown in the following equation:

[0182]

[0183] where μ r and ∑ r are the mean and covariance matrices of the real image, μ g and ∑ g are the mean and covariance matrices of the generated image, respectively, and Tr represents the trace of the matrix. The lower the FID, the better the quality of the synthesized image.

[0184] The CLIP-Score is calculated as the cosine similarity between the normalized image and text embeddings. The image and text are projected into a common feature space using a CLIP image encoder (e.g., a convolutional neural network) and a CLIP text encoder (e.g., a Transformer). A higher score generally indicates a higher consistency between the output image and the text caption. The expression for Clip-Score is shown in the following equation:

[0185]

[0186] Among them, f i 、f t represent the image feature vector and text feature vector respectively, · represents the dot product, and ||·||2 represents the L2 norm.

[0187] LPIPS is an indicator used to perceive the similarity between two images. It calculates similarity in feature space and can overcome the shortcomings of traditional pixel-level similarity indicators. In this embodiment, VGGFace is used as the perception network, mainly used to evaluate the identity consistency between the edited face and the generated face. The expression of LPIPS is shown in the following equation:

[0188]

[0189] Among them, I1 and I2 represent the real image and the generated image, i represents the i-th layer of the VGGFace network, is the normalized representation of the feature vector of the input image I in the i-th layer of the VGGFace network, w i represents the weight coefficient, and ||·||2 represents the L2 norm.

[0190] This example uses three commonly used metrics: FID, Clip-Score, and LPIPS to evaluate generated images from three dimensions: quality of generated images, consistency between generated images and guidance text, and identity consistency between generated face images and input faces. We compare widely used generative models, including the StyleMapGAN and TediGAN models based on generative adversarial GAN ​​networks, and the SD-Edit and DreamSalon models based on stable diffusion networks. The experimental results for these three metrics are shown in the table below, with the bolded data representing the optimal results.

[0191] Table 1 Comparison of FID evaluation index results

[0192]

[0193] Table 2 Comparison of CLIP-Score evaluation index results

[0194]

[0195] Table 3 Comparison of LPIPS evaluation index results

[0196]

[0197] This example quantitatively evaluates five facial attributes separately and averages the five attributes to obtain an overall image evaluation value. As shown in Tables 1, 2, and 3, the FID, CLIP-Score, and LPIPS values ​​of the present invention are 129.56, 0.82, and 17.11, respectively. Overall, it outperforms existing generative models and achieves excellent performance in generated image quality, text-image matching, and consistency of facial identity in generated images, verifying the effectiveness of the present invention.

[0198] Example 2

[0199] This embodiment provides a text-guided fine-grained live face attribute editing system, which is used to implement the text-guided fine-grained live face attribute editing method of the above-mentioned embodiment 1. The system includes: a data set acquisition module, a data set preprocessing module, a face identity feature extraction module, a facial key point feature extraction module, a multi-scale feature mixing module, a text encoder, a conditional diffusion model construction module, a two-stage training module, a text interpolation module, and a face attribute editing image output module;

[0200] In this embodiment, the dataset acquisition module is used to acquire a face image dataset and a text dataset;

[0201] In this embodiment, the dataset preprocessing module is used to divide the dataset, unify the resolution size of the face image, and for each face image, split the text dataset into description texts for different facial attributes;

[0202] In this embodiment, the face identity feature extraction module is used to extract multi-scale face identity features of the face image;

[0203] In this embodiment, the facial key point feature extraction module is used to extract multi-scale facial key point features of the face image;

[0204] In this embodiment, the multi-scale feature mixing module is used to perform feature splicing on the multi-scale face identity features and the multi-scale facial key point features according to the corresponding scales to obtain the identity structure features, and perform weighted averaging on the identity structure features of each scale to obtain the face identity structure features;

[0205] In this embodiment, a text encoder is used to encode text data into raw text embeddings;

[0206] In this embodiment, the conditional diffusion model construction module is used to construct a conditional diffusion model, and the face image and the original text embedding are respectively injected into the conditional diffusion model as the image modality and the text modality, and the conditional diffusion model is trained;

[0207] In this embodiment, a two-stage training module is used to perform two-stage training. During the first stage of training, facial identity structure features are respectively used as image modal conditions, and facial identity structure features and original text embeddings are respectively injected into the conditional diffusion model as multimodal conditions. The corresponding attention feature vector is calculated, and the alignment degree of the facial identity structure features and the original text embeddings is optimized based on the attention loss. The parameters of the facial identity feature extraction module and the facial key point feature extraction module are optimized by the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized by the latent vector reconstruction loss. During the second stage of training, the semantic consistency loss is used to optimize the semantic consistency between the guided text and the original text.

[0208] In this embodiment, the text interpolation module is used to obtain the final text condition vector by interpolating the original text and the guide text;

[0209] In this embodiment, the facial attribute edited picture output module is used to input the final text condition vector into the trained conditional diffusion model to generate a fine-grained facial attribute edited picture.

[0210] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A text-guided fine-grained living face attribute editing method, characterized by: The steps include: Obtain a facial image dataset and a text dataset, divide the dataset, unify the resolution of the facial images, and for each facial image, split the text dataset into descriptions of the corresponding facial attributes according to different facial attributes; Constructing a face identity feature extraction module and a facial key point feature extraction module, wherein the face identity feature extraction module extracts multi-scale face identity features of a face image, and the facial key point feature extraction module extracts multi-scale facial key point features of a face image; Construct a multi-scale feature mixing module, which combines multi-scale face identity features and multi-scale facial key point features according to the corresponding scales to obtain identity structure features. The identity structure features of each scale are weighted averaged to obtain the face identity structure features. The text data is passed through a text encoder to obtain the original text embedding. The face image and the original text embedding are injected into the conditional diffusion model as the image modality and text modality respectively, and the conditional diffusion model is trained. The training is carried out in two stages. In the first stage of training, the facial identity structure features are respectively injected into the conditional diffusion model as image modal conditions and the facial identity structure features and original text embeddings are respectively injected into the conditional diffusion model as multimodal conditions. The corresponding attention feature vector is calculated, and the alignment degree of the facial identity structure features and original text embeddings is optimized based on the attention loss. The parameters of the facial identity feature extraction module and the facial key point feature extraction module are optimized using the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized using the latent vector reconstruction loss. In the second stage of training, the semantic consistency loss is used to optimize the semantic consistency between the guided text and the original text. The original text and the guide text are interpolated to obtain the final text condition vector, and the final text condition vector and the facial identity structure features are input into the trained conditional diffusion model to generate a fine-grained facial attribute editing image.

2. The text-guided fine-grained living face attribute editing method according to claim 1 is characterized in that: The face identity feature extraction module includes a multi-scale encoder and a pre-trained face identity feature extractor; The multi-scale encoder includes a convolutional layer, a maximum pooling layer, multiple ResNet-based residual networks, and a pooling network, and the ResNet-based residual network extracts multi-scale features; The pre-trained face identity feature extractor includes a face identity feature extraction network based on ArcFace; The multi-scale encoder outputs a multi-scale feature vector f=[f 1 , f 2 ,...,f N ]; The multi-scale feature vector is input into the pre-trained face identity feature extractor to extract the multi-scale face identity feature f id =[f 1 id , f 2 id ,...,f N id ], the identity feature of each scale is expressed as: in, represents the face identity feature of the i-th scale, F Arc represents the face identity feature extraction network based on ArcFace, and N represents the size of the division scale.

3. The text-guided fine-grained living face attribute editing method according to claim 1, characterized in that: The facial key point feature extraction module includes a pre-trained face key point extraction network PFLD and a facial key point adjustment network; The pre-trained facial key point extraction network PFLD includes a convolutional layer, a Bottleneck module and a multi-scale fusion network. The feature representation of the facial image extracted by the pre-trained facial key point extraction network is: f sp =F Bottleneck (F DsConv (F Conv (I img ))) Among them, f sp Represents the features output by the Bottleneck module, F Bottleneck Represents the Bottleneck module of the residual network, F DSConv represents a depthwise separable convolutional network, F Conv represents the convolutional network, I img Represents the input face image; Output feature f sp Input to the facial key point adjustment network, the facial key point adjustment network has multiple convolution kernels of different sizes and outputs multi-scale facial key point features Among them, N represents the size of the division scale; The multi-scale face identity feature and the multi-scale facial key point feature have the same dimension.

4. The text-guided fine-grained living face attribute editing method according to claim 1, characterized in that: The face identity structure feature is expressed as: Among them, C id Represents the face identity structure feature, N represents the size of the division scale, α is the weight coefficient, Concat represents the feature splicing of multi-scale face identity features and multi-scale facial key point features, represents the face identity feature of the i-th scale, represents the facial key point feature of the i-th scale.

5. The text-guided fine-grained living face attribute editing method according to claim 1, characterized in that: The facial image and original text embedding are injected into the conditional diffusion model as image modality and text modality respectively, and the conditional diffusion model is trained, specifically including: Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network; The alignment of image semantics and text semantics at the latent space level is achieved by optimizing the similarity of the attention feature vectors of text and image. The attention feature vector is expressed as: Where K, Q, and V represent the index matrix, content matrix, and query matrix corresponding to the image modality and text modality, respectively. d is the projection dimension of the index matrix and query matrix. att Represents the output of the cross attention layer as the input of the convolutional layer of the conditional diffusion model; Each intermediate sample z of the conditional diffusion model t The following conditions must be met: Among them, α t represents the attenuation coefficient of the conditional diffusion model, z t-1 represents the intermediate sample of the t-1 step diffusion process, ε t is the noise that obeys Gaussian distribution; The conditional diffusion model is trained by minimizing the loss function, which is expressed as: Among them, ∈ θ The U-Net network of the denoising generation model, ∈ is the noise obeying the standard Gaussian distribution, t represents the tth step of the diffusion process, C is the input modal feature vector, ||·||2 represents the L2 loss, and z0 represents random noise.

6. The text-guided fine-grained living face attribute editing method according to claim 5, characterized in that: Given the original text embedding and face identity structure features, the index matrix, content matrix, and query matrix are generated respectively through a fully connected linear network, which can be expressed as follows: Among them, K orign 、V orign , Q orign Represents the index matrix, content matrix, and query matrix corresponding to the original text embedding, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, are six linear transformation matrices with different weights, T orign Represents the original text, E T Represents CLIP text encoder, C orign Represents raw text embedding.

7. The text-guided fine-grained living face attribute editing method according to claim 1, characterized in that: In the first stage of training, it is implemented in two steps: Step 1: In the forward and reverse diffusion processes, the facial identity structure features are injected into the conditional diffusion model as image modality conditions. The sampling process is expressed as: Among them, α i is the attenuation coefficient of the conditional diffusion model, They are the latent vectors of the diffusion process of step t-1 and step t in step 1, σ t represents z t The variance of e is the random noise that obeys the standard Gaussian distribution, ∈ θ The U-Net network of the denoising generation model, C id represents the structural features of facial identity, and t represents the tth step of the diffusion process; Calculate the corresponding attention feature vector, expressed as: Among them, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the face identity structure features, K id 、V id , Q id Represents the index matrix, content matrix, and query matrix corresponding to the facial identity structure features; Step 2: In the forward and reverse diffusion processes, the facial identity structure features and the original text embedding are injected into the conditional diffusion model as multimodal conditions. The index matrix, content matrix, and query matrix of the cross attention layer are K orign 、V id , Q orign , the sampling process is expressed as: in, are the latent vectors of the diffusion process of step t-1 and step t in step 2, express The variance, C orign represents the original text embedding; Calculate the corresponding attention feature vector, expressed as: The overall loss function of the first stage training is expressed as: L=k1L attention +k2L rec +k3L per +k4L landmark Among them, k1, k2, k3, and k4 are the attention loss L attention , latent vector reconstruction loss L rec , identity-aware loss L per , facial key point loss L landmark The weight coefficient, E T Represents CLIP text encoder, C l 、H l 、W l is the number of channels, height, and width of the feature map of the first layer, φ l (I img ) and φ l (I rec ) represent face images I img And the face I reconstructed by the conditional diffusion model rec The output features after the lth layer of VGGFace, ijk represents the result of the activation output of the i-th convolution kernel at position (j, k) after the intermediate layer, C represents the number of key points of the face, and They represent the key points of the input face and the reconstructed face respectively, and ||·||1 represents the L1 loss.

8. The text-guided fine-grained living face attribute editing method according to claim 1, characterized in that: In the second stage of training, the original text and the guide text are input into the CLIP text encoder to obtain the original text embedding and the guide text embedding. The semantic consistency loss is used to optimize the semantic consistency between the guide text and the original text. The semantic consistency loss is expressed as: Among them, i represents the word vector in each text embedding, β is a hyperparameter, and C orign represents the original text embedding, C target Represents guided text embedding.

9. The text-guided fine-grained living face attribute editing method according to claim 1, characterized in that: The original text and the guide text are interpolated to obtain the final text condition vector, which is specifically expressed as: C insert =η×C target +(1-n)×C orign Among them, η∈(0,1) represents the interpolation coefficient, C orign represents the original text embedding, C target represents the guided text embedding, C insert represents the final text condition vector; The facial identity structure features and the final text conditional vector are injected into the conditional diffusion model as image modality conditions and text modality conditions. The sampling process is expressed as: in, Represent the latent vectors of the diffusion process of step t and t-1 respectively, express The variance of α i is the attenuation coefficient of the conditional diffusion model, z t is the latent vector of the t-th step diffusion process, ∈ θ is the U-Net network of the denoising generation model, e is random noise that obeys the standard Gaussian distribution, When t is 0, Represents the Gaussian noise of the input conditional diffusion model. When t is T, It represents the target image after T-step sampling and editing according to the final text condition vector prompt; When η=0, the model output is the target image, that is, the input original image. When η=1, the model output is the image edited based on the guided text embedding.

10. A text-guided fine-grained living face attribute editing system, characterized by: The system is used to implement the text-guided fine-grained living face attribute editing method described in any one of claims 1 to 9, comprising: a data set acquisition module, a data set preprocessing module, a face identity feature extraction module, a facial key point feature extraction module, a multi-scale feature mixing module, a text encoder, a conditional diffusion model construction module, a two-stage training module, a text interpolation module, and a face attribute editing image output module; The data set acquisition module is used to acquire a face image data set and a text data set; The data set preprocessing module is used to divide the data set, unify the resolution size of the face image, and for each face image, split the text data set into description texts corresponding to the facial attributes according to different facial attributes; The face identity feature extraction module is used to extract multi-scale face identity features of face images; The facial key point feature extraction module is used to extract multi-scale facial key point features of the face image; The multi-scale feature mixing module is used to perform feature splicing on the multi-scale face identity features and the multi-scale facial key point features according to the corresponding scales to obtain the identity structure features, and perform weighted averaging on the identity structure features of each scale to obtain the face identity structure features; The text encoder is used to encode text data into raw text embedding; The conditional diffusion model construction module is used to construct a conditional diffusion model, inject the face image and the original text embedding as the image modality and the text modality into the conditional diffusion model respectively, and train the conditional diffusion model; The two-stage training module is used to perform two-stage training. During the first stage of training, the facial identity structure features are respectively used as image modal conditions, and the facial identity structure features and the original text embedding are respectively injected into the conditional diffusion model as multimodal conditions, and the corresponding attention feature vectors are calculated. The alignment degree of the facial identity structure features and the original text embedding is optimized based on the attention loss. The parameters of the facial identity feature extraction module and the facial key point feature extraction module are optimized through the identity perception loss and the facial key point loss. The alignment degree of the original text and the input face in the latent space is optimized through the latent vector reconstruction loss. During the second stage of training, the semantic consistency between the guided text and the original text is optimized through the semantic consistency loss. The text interpolation module is used to obtain a final text condition vector by interpolating the original text and the guide text; The face attribute edited picture output module is used to input the final text condition vector and face identity structure features into the trained conditional diffusion model to generate a fine-grained face attribute edited picture.

Images