A face attribute editing method based on mask denoising and feature selection

Abstract

The application discloses a face attribute editing method based on mask denoising and feature selection. The mask denoising module is designed for face attribute editing, can generate only the mask related to the attribute to be changed when generating the attention mask, and remove some background masks containing face information, so that the attribute editing task can be well completed and the consistency of the non-editing area is reserved. The feature selection unit is designed to filter the features of the image at the feature level, retain the features irrelevant to the attribute lost due to the network depth, and perform feature style fusion according to the attribute style code after the information enrichment of the binary attribute vector via the attribute encoder in the decoding process. The pre-trained tamper detection model can better detect whether the generated image is tampered, and improve the authenticity of the generated image.

Description

TECHNICAL FIELD

[0001] The technical scheme of the present application relates to face image attribute editing technology, in particular to a face attribute editing method based on mask denoising and feature selection. BACKGROUND

[0002] Face attribute editing technology is a popular research direction in computer graphics, and the purpose is to edit the attributes of face images according to subjective requirements. The editable attributes include glasses, bangs, beard, mouth opening and closing, etc. Face attribute editing can play a certain role in movie special effects, human-computer interaction, and finding lost people. How to edit one or more attributes of a face image according to the subjective requirements of the user in a more realistic way, and not to change other areas except the editing area, has always been a more difficult task.

[0003] Most of the current research on face attribute editing methods is based on Generative Adversarial Network (GAN) to design and implement. Early GAN methods require the use of the same face attribute pair images for training to obtain a model for editing a single attribute. Later, technology developed to no longer require paired images for attribute editing training, but a single training can only target a single attribute. Later, in 2019, Choi Y, Choi M, Kim M, et al. published "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation" in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, which first proposed using a single model to edit multiple attributes. This paper uses a binary one-hot vector to represent face attributes and uses the one-hot vector to guide the training process, achieving the effect of a single model editing multiple attributes. Subsequent works have further studied on the idea of this article. However, the quality of the generated image is not good when using a single model to edit multiple attributes, and it also causes the generated image to have the effect of attribute bundling, for example, when aging a face image, the hair color will turn white, because old people usually have white hair. CN114494529A discloses a face attribute editing method based on cost-sensitive learning, which provides different weights for different categories of samples in the data set during training of the deep learning model to learn a better solution to the influence of sample imbalance in the data set on the performance of the model. However, this method requires pre-computing the distribution probability of different face attributes in the data set, which requires additional computing resources. CN115082292A discloses a face multi-attribute editing method based on global attribute editing direction, which obtains an attribute association graph, an attribute semantic embedding set, and a scale factor, constructs a global attribute editing network with the attribute association graph, the attribute semantic embedding set, and the scale factor as input and the global attribute editing direction as output, and solves the problem of multiple single attribute editing when editing multiple attributes. This method also needs to calculate the attribute association graph in the data set, and editing face images in other data sets also requires obtaining relevant information in the new data set, which consumes a lot of resources. SUMMARY

[0004] The purpose of this invention is to address the shortcomings of existing single-model face multi-attribute editing techniques by proposing a face attribute editing method based on mask denoising and feature selection.

[0005] This invention, based on generative adversarial networks (GANs), designs a mask denoising module and a feature selection unit. Specifically, this method includes a generator network and a discriminator network. The generator network consists of an encoder-decoder and a mask denoising module. The mask denoising module is structurally similar to the decoder, both utilizing feature selection units. However, unlike the decoder, the mask denoising module includes a coordinate space attention layer designed in this invention, and it learns the single-channel attention mask required for facial feature reconstruction after facial attribute editing. The mask denoising module outputs two masks: one for facial reconstruction and another for facial attribute editing. The former, called the reconstruction mask, contains a large amount of facial contour and background noise information. The latter, called the editing mask, contains not only facial contour and background noise information but also the positional information of the edited attribute regions. Fusing the editing mask and the reconstruction mask yields the specific location of the attribute regions during editing, called the attribute mask, which removes noise related to the facial contour and background. By fusing the original image and the image generated by the decoder with an attribute mask, a face image can be obtained that only has its attributes edited while retaining other regions unchanged. The feature selection unit includes an attribute-independent feature extractor and a target style fusion unit. The attribute-independent feature extractor extracts attribute-independent features layer by layer from the encoder based on the input binary attribute-independent vector. These extracted features are then input into the target style fusion unit, where they are first concatenated channel-wise with the features output from the transposed convolutional layer in the decoder or mask denoising module. The concatenated features are then style-fused with the input binary attribute-related vector. The feature selection unit preserves as many attribute-independent region features as possible, preventing feature loss, and integrates attribute-related styles into the output features of the transposed convolutional layer.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] A face attribute editing method based on mask denoising and feature selection is a face attribute editing method based on a mask denoising module and a feature selection unit. The implementation of this method mainly includes the following:

[0008] First, the dataset needs to be selected and manipulated. The publicly available face dataset is preprocessed before network training by centering and cropping the face to obtain a face image with a pixel size of 128*128.

[0009] A face attribute editing network with mask denoising and feature selection is constructed. The network consists of two parts: a generator and a discriminator. The generator adopts an encoder-decoder structure, on which a mask denoising module and a feature selection unit are added. The encoder consists of 5 convolutional layers, and the decoder consists of 4 transposed selection layers and 1 transposed convolutional layer. The mask denoising module has a similar structure to the decoder, including 4 transposed selection layers, 1 coordinate space attention layer, and 1 transposed convolutional layer.

[0010] The mask denoising module is used to obtain the attribute coordinate position mask map when performing face editing. The input of the mask denoising module is the feature of the image after the encoder. The feature is input into the mask denoising module and passes through 4 layers of transposed selection blocks. The obtained feature is then input into the coordinate space attention layer, which is used to weight the dimension, height, and width of the feature and the spatial position information related to the attribute. The weighted feature is then input into the last transposed convolution block to obtain two attention masks, including an editing mask related to the attribute and a reconstruction mask that only performs image reconstruction and is unrelated to the attribute. The two masks are selectively fused to obtain an attribute mask that only retains the positional region information related to the attribute. The attribute mask is then fused with the image output by the decoder and the original image to obtain the final result.

[0011] The feature selection unit preserves attribute-independent features and fuses attribute-related information with the input features. The feature selection unit includes an attribute-independent feature extractor and a target style fusion unit. The attribute-independent feature extractor contains an attribute-independent encoder, and the target style fusion unit contains an attribute-related encoder. The feature selection unit first inputs the features output from the corresponding convolutional block in the encoder into the attribute-independent feature extractor, extracts attribute-independent features based on the attribute-independent style code, concatenates the extracted attribute-independent features with the features output from the transposed convolutional layer in the corresponding transposed selection block of the decoder or mask denoising module, and then inputs the concatenated features into the target style fusion unit to fuse with the attribute-related style code. The output is the feature that preserves attribute-independent information and incorporates attribute-related style information. Specifically, the attribute-independent style code is obtained by encoding a binary attribute-independent vector by the attribute-independent encoder, and the attribute-related style code is obtained by encoding an input binary attribute-related vector by the attribute-related encoder. The attribute-independent vector is obtained by inverting the attribute-related vector.

[0012] The discriminator consists of two discriminators. One discriminator employs a pre-trained model in the field of image manipulation, proposed by He Y, Yu N, Keuper M, et al. in their paper "Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis" published at the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021). This invention pre-trains this model using images trained on the original generative adversarial network and images from a real dataset, selecting the best-performing pre-trained model as an additional discriminator. The other discriminator comprises a 5-layer convolutional module, an adversarial head for adversarial loss to determine whether an image is real or fake, and a classification head for image classification. Both the adversarial head and the classification head are fully connected layers. By inputting the face image into these 5 convolutional layers, the resulting features are then input into the adversarial head and the classification head respectively to obtain the corresponding probability values.

[0013] The convolutional block consists of a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a batch normalization layer (BN), and a Leaky ReLU activation function. The transpose selection block includes a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, and a feature selection unit. The transposed convolutional block includes a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a batch normalization layer (BN), and a ReLU activation function. The feature selection unit includes an attribute-independent feature extractor and a target style fusion unit. The attribute-independent feature extractor includes a transposed convolutional layer, an Adain adaptive instance normalization layer, a ReLU activation layer, and an attribute-independent encoder. The target style fusion unit includes a channel concatenation operation, an Adain adaptive instance normalization layer, a ReLU activation layer, and an attribute-dependent encoder. Both the attribute-independent encoder and the attribute-dependent encoder consist of three fully connected modules, each including a fully connected layer, a PixelNorm normalization layer, and a Leaky ReLU activation layer.

[0014] The specific steps of the face attribute editing method based on mask denoising and feature selection are as follows:

[0015] The first step is to extract features from the input face image data layer by layer through the encoder network;

[0016] Step 1.1: Center-crop the input face image data, adjust the resolution to 128*128, normalize, and tensor the data to obtain the preprocessed face image data, which is then used as the original input image Img. real ;

[0017] Step 1.2: The face data from step 1.1 above is input into the first convolutional block of the encoder. The size of a single face image is 128*128*3. After passing through a convolutional layer with a kernel size of 4*4, a stride of 2, and padding of 1, it passes through a batch normalization layer (BN) and a LeakyReLU activation function to obtain a feature map with a size of 64*64*64.

[0018] Step 1.3: The output of the previous convolutional block is used as the input of the next convolutional block for convolution operation, as shown in the following formula (1);

[0019]

[0020] In equation (1), This represents the feature output of the i-th convolutional block of the encoder. Represents the features output by the (i-1)th convolutional block. Conv indicates the operation of inputting the features into the convolutional block. The specific operation is as follows: first, perform a convolution operation with a kernel size of 4*4, a stride of 2, and padding of 1; then, perform batch normalization (BN); and finally, perform LeakyReLU activation.

[0021] The second step is to use the feature selection unit to extract attribute-independent features from the features of different layers of the encoder and retain the attribute-independent features.

[0022] The encoder features of different layers are respectively input into the attribute-independent vector in the attribute-independent feature extractor in the feature selection unit to extract attribute-independent features, as shown in the following formula (2);

[0023]

[0024] In equation (2), V dif This represents an attribute-independent vector that is independent of the attributes of a face image. This represents feature information unrelated to the attributes of the i-th layer corresponding to the number of layers in the decoder or mask denoising module. Cfe represents the feature of the corresponding layer in the encoder corresponding to the i-th layer of the decoder or mask denoising module. The specific operation method is shown in the following formula (3).

[0025]

[0026] In equation (3), ConvT represents the transpose convolution operation on the input features, Adain represents the adaptive instance normalization operation on the features, ReLU represents the ReLU activation function activation on the input features, and Emb... dThis indicates that an attribute-independent encoder is used for encoding. The attribute-independent encoder contains three consecutive sets of FC fully connected layer mappings, PixelNorm normalization, and LeakyReLU activation function. The attribute-independent vector is input into the attribute-independent encoder and processed sequentially to obtain the attribute-independent style code, as shown in the following formula (4).

[0027] S dif =Emb d (V dif (4)

[0028] In equation (4), S dif This represents the attribute-independent style code obtained after passing through the attribute-independent encoder.

[0029] The third step involves using a decoder and a feature selection unit to decode the features of the last layer of the encoder, resulting in an image for preliminary facial attribute editing.

[0030] Step 3.1: Input the output features of the last layer of the encoder into the transposed convolutional layer in the first layer of the decoder to obtain the intermediate features without feature selection, as shown in the following formula (6);

[0031]

[0032] In equation (5), The intermediate features are obtained by passing the output features of the last layer of the encoder through the transposed convolutional layer in the first layer of the decoder.

[0033] Step 3.2, the features obtained in 3.1 Input attribute correlation vector V con Features unrelated to the attributes of the first layer of the corresponding decoder obtained in step two. The input is fed into the target style fusion unit of the feature selection unit in the first layer, as shown in the following formula (6);

[0034]

[0035] In equation (6), Tsf is the output of the first layer of the decoder, and Tsf is the target style fusion unit. The specific operation is shown in the following formula (7);

[0036]

[0037] In equation (7), Concat represents the channel splicing operation, and Emb... cThis indicates that the attribute-related vector is encoded using an attribute-related encoder. The attribute-related encoder contains three consecutive sets of FC fully connected layer mappings, PixelNorm normalization, and LeakyReLU activation function. The attribute-related vector is input into the attribute-related encoder and processed sequentially to obtain the attribute-related style code, as shown in the following formula (8).

[0038] S con =Emb c (V con (8)

[0039] In equation (8), S con This represents the attribute-related style code obtained after the attribute-related vector has been processed by the attribute-related encoder.

[0040] Step 3.3: Use the output of the previous layer of the decoder as the input of the next layer for direct calculation, as shown in the following formula (9);

[0041]

[0042] In equation (9), This represents the output of the i-th layer of the decoder. This represents the output of the (i-1)th layer. This represents the output of the attribute-independent feature extractor in the i-th layer; Tsf is the target style fusion unit.

[0043] Step 3.4: The output of the fourth layer of the decoder is input into the fifth layer of the decoder, sequentially fed into a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a batch normalization layer (BN), and a ReLU activation function layer, to obtain the output of the last layer of the decoder.

[0044] Step 3.5, take the final result from step 3.4. After passing through a transposed convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, and a Tanh activation function layer, a preliminary synthesized face image Img1 is obtained;

[0045] The fourth step is to use the mask denoising module and feature selection unit to decode the features of the last layer of the encoder to obtain the final attribute mask.

[0046] Step 4.1: The target attribute label and the original attribute label are sequentially input into the mask denoising module along with the last layer features of the encoder. The operations of formulas (5), (6), (7), (8), and (9) are performed to obtain the feature output of the fourth layer of the mask denoising module, which are the output features with the target attribute vector as input. Output features with the original attribute vector as input

[0047] Step 4.2, use the features obtained in step 4.1 and The inputs are respectively fed into the coordinate space attention layer CorSpa, as shown in the following formulas (10) and (11);

[0048]

[0049]

[0050] In equation (10), F mt for The result after the coordinate space attention layer; in equation (11), F mo for The result after passing through the coordinate space attention layer;

[0051] Step 4.3: The results obtained in step 4.2 are input into the last layer of the mask denoising module, passing sequentially through a transposed convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, a batch normalization layer (BN), and a ReLU activation function layer, to obtain the final output features of the mask denoising module, which are the target output features with the target attribute vector as input. and the original output features with the original attribute vector as input.

[0052] In step 4.4, the target output features and the original output features obtained in step 4.3 are sequentially input into a transposed convolutional layer and a sigmoid layer with a kernel size of 3×3, a stride of 1, and padding of 1, to obtain the final mask image, which is Mask. t and Mask o Mask t This is an edit mask that takes the target attribute vector as input. The black areas in the edit mask contain the attribute regions to be edited, as well as noise regions with identity and background information. o The reconstruction mask is taken as input from the original attribute vector. The black areas in the reconstruction mask only contain noisy areas with identity and background.

[0053] Step 4.5, apply the editing mask obtained in step 4.4. t and reconstructed mask o Selective blending is performed as shown in the following formula (12), which removes the black areas that are the same as the noise areas with identity and background in the reconstruction mask, and highlights the black attribute areas unique to the editing mask;

[0054] Mask f =1+Mask t -Mask o (12)

[0055] In equation (12), Mask f This is the final attribute mask obtained after the mask denoising module and selection fusion;

[0056] Step 5, apply the mask f The initial synthesized face image Img1 and the input original image Img real The final image is obtained by fusion, as shown in formula (13) below;

[0057] Img f =Img real *Mask f +Img1*(1-Mask f (13)

[0058] In formula (13), Img f This represents the final synthesized face image obtained after the generator has processed it.

[0059] The fifth step is to use a discriminator to perform image discrimination operations;

[0060] Step 5.1, input the image data (final synthesized face image) into the pre-trained tampering detection model proposed by He Y, Yu N, Keuper M et al. in "Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis" published in the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021) to determine the probability that the image has been tampered with, as shown in the following formula (14);

[0061] Edit_P fake =D edit (X) (14)

[0062] In equation (14), D edit This is a pre-trained tamper detection model, where X is the input image data and Edit_P is the input image data. fake The probability that the image has been tampered with is detected after the tamper detection model is used;

[0063] Step 5.2: The second discriminator consists of a 5-layer convolutional module, an adversarial head for adversarial loss (determining whether an image is real or fake), and a classification head for image classification. Both the adversarial head and the classification head are fully connected layers. Image data (real face images and the final synthesized face images) are input into the first layer of the second discriminator's convolutional module, passing sequentially through a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, an IN instance normalization layer, and a LeakyReLU activation function layer, to obtain the output of the first convolutional layer of the discriminator.

[0064] Step 5.3, take the output of the previous layer of the second discriminator as the input of the next convolutional layer, as shown in the following formula (15);

[0065]

[0066] In equation (15), This represents the output of the i-th layer convolutional module of the second discriminator. This represents the output of the i*1th layer convolutional module of the second discriminator;

[0067] Step 5.4, The input is fed into the classification head of the second discriminator to obtain the discrimination probability of the image with respect to each attribute, as shown in the following formula (16);

[0068]

[0069] In equation (16), D cls This represents the probability value of each attribute obtained by the second discriminator; Linear represents a fully connected layer, which performs fully connected mapping.

[0070] Step 5.5, will The input is fed into the adversarial head of the second discriminator to obtain the probability that the image is judged as a real image, as shown in the following formula (17);

[0071]

[0072] In equation (17), D adv This represents the probability that an image is judged as a real image after passing through a second discriminator.

[0073] At this point, the above steps complete the construction of the generator and discriminator network structure for the face attribute editing method based on mask denoising and feature selection. The final generation result of the generator and the discrimination result obtained by the discriminator can be obtained from the input image data.

[0074] Loss function setting for a face attribute editing method based on mask denoising and feature selection;

[0075] The loss function setting for the face attribute editing method generator based on mask denoising and feature selection is shown in the following formula (18);

[0076] L g =λ1*L adv +λ2*L rec +λ3*L cls +λ4*L edit (18)

[0077] In equation (18), L g Let λ be the total loss function of the generator. i ,i∈(1,2,3,4) are the hyperparameters balancing the various losses, L adv For the adversarial loss of the generator, L rec L is the reconstruction loss of the generator. cls L is the classification loss of the generator. edit The tamper detection loss is calculated after the image generated by the generator is processed by the tamper detection model.

[0078] The adversarial loss of the face attribute editing method generator based on mask denoising and feature selection is shown in the following formula (19);

[0079] L adv =-E[D adv (G(img,label) t (19)

[0080] In equation (19), E[·] represents the mean operation, and D adv This indicates that the second discriminator, acting as an adversarial head, judges the realism of the image. G represents the process of the generator generating the image, img is the original image data input to the generator, and label... t The attribute tag for the target attribute;

[0081] The reconstruction loss of the face attribute editing generator based on mask denoising and feature selection is shown in the following formula (20);

[0082] L rec =E||img-G(img,label) o )||1 (20)

[0083] In equation (20), ||·||1 is the L1 norm, and label o These are the original attribute labels corresponding to the input image;

[0084] The classification loss of the face attribute editing generator based on mask denoising and feature selection is shown in the following formula (21);

[0085] L cls =CrossEntropy(label) t D cls (G(img,label) t ))) (twenty one)

[0086] In equation (21), D cls The second discriminator classification head performs classification predictions for each attribute of the input image, with CrossEntropy being the cross-entropy loss.

[0087] The tamper detection loss of the face attribute editing method generator based on mask denoising and feature selection is shown in the following formula (22);

[0088] L edit =CrossEntropy(1,D) edit (G(img,label) t ))) (twenty two)

[0089] In equation (22), D edit To introduce a pre-trained tamper detection discriminator, the probability of the input image after tamper detection by the tamper detection model is subjected to cross-entropy loss with the all-1 tensor to obtain the final tamper detection loss.

[0090] The loss function for the second discriminator is set as shown in formula (23) below;

[0091] L d =λ5*L adv +λ6*L cls (twenty three)

[0092] In equation (23), λ5 and λ6 are hyperparameters of the weights between the discriminator adjustment loss, and L d For the total loss of the discriminator, L adv For the adversarial loss of the second discriminator, L cls The classification loss of the second discriminator;

[0093] The adversarial loss of the second discriminator is shown in equation (24);

[0094] L adv =-E[D adv (img)]+E[D adv (G(img,label) t ))] (twenty four)

[0095] The classification loss of the second discriminator is shown in equation (25);

[0096] L cls =CrossEntropy(label) o D cls (img)) (25)

[0097] After the above operations, the overall process of the face attribute editing method based on mask denoising and feature selection, including model building, training and optimization, is completed. The final trained model can be used to perform face attribute editing operations.

[0098] Compared with existing methods, the advantages and significant advancements of this invention are as follows:

[0099] (1) This invention proposes a method for editing facial attributes, specifically a facial attribute editing method based on mask denoising and feature selection. The mask denoising module generates a mask for facial image reconstruction and an editing mask for facial attribute editing. By selectively fusing the two masks, a mask containing only the regions to be edited can be obtained by removing background noise. By combining this attribute mask with the original image and the image generated by the decoder, a final result can be obtained that accurately edits and preserves the non-edited attribute regions unchanged.

[0100] (2) This invention proposes a feature selection unit that obtains multi-scale information of the image from different layers of the encoder. Then, it extracts multi-scale features and fuses them with attribute-independent style codes to extract attribute-independent feature information. Then, it concatenates the features output by the transposed convolutional layer in the corresponding layer of the decoder or mask denoising module and fuses them with the attribute style codes to edit the features. This can more accurately retain attribute-independent features, prevent the loss of image features after passing through multiple network layers, and accurately edit attribute-related features.

[0101] (3) This invention introduces a model from the field of image tampering detection into the discriminator. The image tampering detection model is pre-trained using images generated by a generator network, enabling it to detect whether an image has been tampered with. This pre-trained discriminator is then added to the network for further training. This approach improves the realism of the images generated by the model, allowing it to generate images that better reflect real-world situations. Attached Figure Description

[0102] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0103] Figure 1 This is a flowchart illustrating the overall training process of the face attribute editing method based on mask denoising and feature selection in this invention.

[0104] Figure 2 This is a flowchart of the generator part of the face attribute editing method based on mask denoising and feature selection in this invention.

[0105] Figure 3 This is a flowchart of the discriminator part of the face attribute editing method based on mask denoising and feature selection in this invention.

[0106] Figure 4 This is a flowchart of the feature selection unit of the face attribute editing method based on mask denoising and feature selection in this invention.

[0107] Figure 5This is a flowchart of the attribute-dependent encoder and attribute-independent encoder of the face attribute editing method based on mask denoising and feature selection of the present invention.

[0108] Figure 6 This is a flowchart of the coordinate space attention layer of the face attribute editing method based on mask denoising and feature selection in this invention.

[0109] Figure 7 This is a flowchart of the coordinate attention module in the coordinate space attention layer of the face attribute editing method based on mask denoising and feature selection in this invention.

[0110] Figure 8 This is a flowchart of the spatial attention module in the coordinate space attention layer of the face attribute editing method based on mask denoising and feature selection in this invention.

[0111] Figure 9 This is a generated image of the face attribute editing method based on mask denoising and feature selection according to the present invention. Detailed Implementation

[0112] This invention relates to a face attribute editing method based on mask denoising and feature selection, which includes the following:

[0113] First, the publicly available face dataset is preprocessed to obtain the original input images, with each face image corresponding to a binary attribute label.

[0114] Construct a face attribute editing network that performs mask denoising and feature selection. The network consists of two parts: a generator and a discriminator.

[0115] The generator adopts an encoder-decoder structure, and a mask denoising module is added on this basis. The encoder includes 5 convolutional blocks, the decoder includes 4 transposed selection blocks and 1 transposed convolutional block, and the mask denoising module includes 4 transposed selection blocks, 1 coordinate space attention layer and 1 transposed convolutional block.

[0116] The transpose selection block includes a transpose convolutional layer and a feature selection unit. The feature selection unit includes an attribute-independent feature extractor and a target style fusion unit. The attribute-independent feature extractor includes a transpose convolutional layer, an Adain adaptive instance normalization layer, a ReLU activation layer, and an attribute-independent encoder. The target style fusion unit includes a channel concatenation operation, an Adain adaptive instance normalization layer, a ReLU activation layer, and an attribute-related encoder. Both the attribute-independent encoder and the attribute-related encoder are composed of three fully connected modules, each including a fully connected layer, a PixelNorm normalization layer, and a LeakyReLU activation layer.

[0117] The feature selection unit first inputs the features output by the corresponding convolutional block in the encoder into the attribute-independent feature extractor, and extracts attribute-independent features according to the attribute-independent style code. The extracted attribute-independent features are then concatenated with the features output by the transposed convolutional layer in the corresponding transposed selection block in the decoder or mask denoising module. The concatenated features are then input into the target style fusion unit and fused with the attribute-related style code. The output is the feature after retaining the attribute-independent information and fusing the attribute-related style information.

[0118] The attribute-independent style code is obtained by inputting the attribute-independent vector into the attribute-independent encoder, the attribute-dependent style code is obtained by inputting the attribute-dependent vector into the attribute-dependent encoder, and the attribute-independent vector is obtained by inverting the attribute-dependent vector.

[0119] Figure 1 The illustrated embodiment shows that the overall training process of the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0120] The input image is preprocessed, and then the encoder extracts deep features layer by layer, called encoder features, which are then input into the decoder and mask denoising module. The decoder decodes layer by layer and uses a feature selection unit to filter features based on the input target attribute vector, ultimately generating a preliminary edited image. The mask denoising module decodes layer by layer and uses a feature selection unit to filter features based on whether the input attribute vector is the target attribute vector or the original attribute vector. An editing mask is generated based on the input target attribute vector, and a reconstruction mask is generated based on the input original attribute vector. The editing mask and the reconstruction mask are fused to generate an attribute mask. The generated preliminary edited image is fused with the attribute mask and the input original image to generate the final image. The generated image is input into the discriminator module, which judges and classifies the generated image to determine its authenticity and outputs the results. Finally, the loss function is calculated based on the discriminator's results, and the entire training network is optimized through backpropagation.

[0121] Figure 2 The illustrated embodiment shows that the generation process of the generator for the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0122] The generator includes an encoder, a decoder, and a mask denoising module. The encoder consists of 5 sets of convolutional blocks, and the decoder consists of 4 transposed selection blocks and 1 transposed convolutional block. The mask denoising module has a similar structure to the decoder, consisting of 4 transposed selection blocks, 1 coordinate space attention layer, and 1 transposed convolutional block. The mask denoising module shares weights with the transposed selection blocks of the decoder. The output of the last transposed convolutional block of the decoder is a preliminary synthesized face after initial editing. The output of the last transposed convolutional block in the mask denoising module is a reconstruction mask and an editing mask, depending on the type of the input attribute vector. The reconstruction mask and the editing mask are fused to obtain the attribute mask. The preliminary synthesized face, the attribute mask, and the input face are fused to generate the final synthesized face image. The transposed selection block includes a transposed convolutional layer and a feature selection unit, which are connected sequentially. The transposed convolutional block includes a 4×4 kernel, a stride of 2, and padding of 1, as well as a batch normalization layer and a ReLU activation function.

[0123] Figure 3 The illustrated embodiment shows that the overall process of the discriminator part of the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0124] The discriminator component comprises two discriminators. One discriminator employs a pre-trained model in the field of image manipulation, proposed by He Y, Yu N, Keuper M, et al. in their paper "Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis" published at the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021). This invention pre-trains this model using images trained on the original generative adversarial network and images from a real dataset, selecting the best-performing pre-trained model as an additional discriminator. The other discriminator consists of a 5-layer convolutional module, an adversarial head for determining whether an image is real or fake using adversarial loss, and a classification head for image classification. Both the adversarial head and the classification head are fully connected layers. The classification head is used for image attribute classification, while the adversarial head determines whether the generated image is real or fake.

[0125] Figure 4 The illustrated embodiment shows that the overall process of the feature selection unit in the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0126] The feature selection unit comprises an attribute-independent feature extractor and a target style fusion unit, which are connected sequentially. The attribute-independent feature extractor includes a transposed convolutional layer, an adaptive instance normalization layer, and an attribute-independent encoder. The target style fusion unit includes an attribute-related encoder, an adaptive instance normalization layer, and a channel concatenation operation. The input to the feature selection unit is the features from different layers of the encoder and the features output by the transposed convolutional layer in the corresponding layer of the decoder or mask denoising module. The input to the attribute-independent feature extractor is the features from different layers of the encoder and attribute-independent vectors. The input to the target style fusion unit is the features output by the transposed convolutional layer in the corresponding layer of the decoder or mask denoising module and attribute-related vectors. The input target attribute vector or original attribute vector is set as an attribute-related vector. The attribute-related vector is inverted to obtain an attribute-independent vector. The attribute-independent vector is input to the attribute-independent encoder to obtain an attribute-independent style code. Based on the attribute-independent style code, the encoder features processed by the transposed convolutional layer are subjected to adaptive instance normalization and ReLU activation to filter attribute-independent features. The output of the attribute-independent feature extractor is the output of the ReLU activation function after adaptive instance normalization.

[0127] Then, the features output by the attribute-independent feature extractor and the features output by the transposed convolutional layer in the corresponding layer transpose selection block of the decoder or mask denoising module are input into the target style fusion unit. In the target style fusion unit, the two are first concatenated on the channel side. Then, the concatenated features are adaptively normalized and activated by the ReLU function according to the attribute-related style code obtained by the attribute-related encoder through the attribute-related vector. The final feature is output, which retains the attribute-independent features and modifies the attribute-related features.

[0128] Figure 5 The illustrated embodiments demonstrate that the overall process of the attribute-dependent encoder and attribute-independent encoder in the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0129] The input to the attribute-related encoder is a binary one-hot attribute vector. The input vector first passes through a fully connected (FC) layer to expand the dimension of the binary attribute vector to 32, then through PixelNorm pixel normalization and LeakyReLU activation function, then through another fully connected layer to expand the dimension to 64, and then through PixelNorm pixel normalization and LeakyReLU activation function again. Finally, through another fully connected layer, the dimension of the result after the above operations is expanded to 128, and then through PixelNorm pixel normalization and LeakyReLU activation function. The final output is the attribute-related style code after attribute encoding.

[0130] The input to the attribute-independent encoder is an attribute vector in binary one-hot form after inversion, called the attribute-independent vector. The process is the same as that of the attribute-related encoder, and the final output is the attribute-independent style code.

[0131] Figure 6 The illustrated embodiment shows that the overall flow of the coordinate space attention layer of the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0132] The input features first pass through the coordinate attention module, and the resulting features are then multiplied with the input features before being input into the spatial attention module. The features processed by the spatial attention module are then multiplied with the features input into the spatial attention module to obtain the final output features.

[0133] Figure 7 The illustrated embodiment shows that the overall process of the coordinate attention module in the coordinate space attention layer of the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0134] The input features are first subjected to average pooling in the X direction of C×H×1 and average pooling in the Y direction of C×1×W. The two pooled features are then concatenated along the channel. Then, a convolutional layer with a kernel size of 1×1, a stride of 1, and padding of 0 is input. Next, a normalized BN layer and a ReLU activation function layer are input. The output features are segmented along the channel into two features of C×H×1 and C×1×W. The two output features of different sizes are then passed through a convolutional layer (Conv) and a Sigmoid activation function layer, respectively, and then fused to obtain the final output feature with dimensions of C×H×W.

[0135] Figure 8 The illustrated embodiment shows that the overall process of the spatial attention module in the coordinate space attention layer of the face attribute editing method based on mask denoising and feature selection of the present invention is as follows:

[0136] The input features are first processed by max pooling and average pooling, then by a convolutional layer (Conv) and a sigmoid activation function layer before the final features are output.

[0137] Example 1

[0138] The face attribute editing method based on mask denoising and feature selection in this embodiment has the following specific steps:

[0139] The first step is to extract features from the input face image data layer by layer through the encoder network;

[0140] Step 1.1: The input face image data is centered and cropped, the resolution is adjusted to 128*128, normalized, and tensorized to obtain the preprocessed face image data. In this embodiment, the dataset uses the open dataset CelebA from the Chinese University of Hong Kong, which contains 202,599 face images of 10,177 celebrities. Each face is labeled with more than 40 attributes. This invention selects several of these attributes for effect demonstration.

[0141] Step 1.2: The face data from step 1.1 above is input into the first convolutional block of the encoder. The size of a single face image is 128*128*3. After passing through a convolutional layer with a kernel size of 4*4, a stride of 2, and padding of 1, it passes through a batch normalization layer (BN) and a LeakyReLU activation function to obtain a feature map with a size of 64*64*64.

[0142] Step 1.3: The output of the previous convolutional block is used as the input of the next convolutional block for convolution operation, as shown in the following formula (1);

[0143]

[0144] In equation (1), This represents the feature output of the i-th convolutional block of the encoder. Represents the features output by the (i-1)th convolutional block. Conv indicates the operation of inputting the features into the convolutional block. The specific operation is as follows: first, perform a convolution operation with a kernel size of 4*4, a stride of 2, and padding of 1; then, perform batch normalization (BN); and finally, perform LeakyReLU activation.

[0145] The second step is to use the feature selection unit to extract attribute-independent features from the features of different layers of the encoder and retain the attribute-independent features.

[0146] The encoder features of different layers are respectively input into the attribute-independent vector in the attribute-independent feature extractor in the feature selection unit to extract attribute-independent features, as shown in the following formula (2);

[0147]

[0148] In equation (2), V dif This represents an attribute-independent vector that is independent of the attributes of a face image. This represents feature information unrelated to the attributes of the i-th layer corresponding to the number of layers in the decoder or mask denoising module. Cfe represents the feature of the corresponding layer in the encoder corresponding to the i-th layer of the decoder or mask denoising module. The specific operation method is shown in the following formula (3).

[0149]

[0150] In equation (3), ConvT represents the transpose convolution operation on the input features, Adain represents the adaptive instance normalization operation on the features, ReLU represents the ReLU activation function activation on the input features, and Emb... d This indicates that an attribute-independent encoder is used for encoding. The attribute-independent encoder contains three consecutive sets of FC fully connected layer mappings, PixelNorm normalization, and LeakyReLU activation function. The attribute-independent vector is input into the attribute-independent encoder and processed sequentially to obtain the attribute-independent style code, as shown in the following formula (4).

[0151] S dif =Emb d (V dif (4)

[0152] In equation (4), S dif This represents the attribute-independent style code obtained after passing through the attribute-independent encoder.

[0153] The third step involves using a decoder and a feature selection unit to decode the features of the last layer of the encoder, resulting in an image for preliminary facial attribute editing.

[0154] Step 3.1: Input the output features of the last layer of the encoder into the transposed convolutional layer in the first layer of the decoder to obtain intermediate features without feature selection, as shown in the following formula (6);

[0155]

[0156] In equation (5), The intermediate features are obtained by passing the output features of the last layer of the encoder through the transposed convolutional layer in the first layer of the decoder.

[0157] Step 3.2, combine the features obtained in 3.1 with the input attribute correlation vector V. con Features unrelated to the attributes of the first layer of the corresponding decoder obtained in step two. The input is fed into the target style fusion unit of the feature selection unit in the first layer, as shown in the following formula (6);

[0158]

[0159] In equation (6), Tsf is the output of the first layer of the decoder, and Tsf is the target style fusion unit. The specific operation is shown in the following formula (7);

[0160]

[0161] In equation (7), Concat represents the channel splicing operation, and Emb... c This indicates that the attribute-related vector is encoded using an attribute-related encoder. The attribute-related encoder contains three consecutive sets of FC fully connected layer mappings, PixelNorm normalization, and LeakyReLU activation function. The attribute-related vector is input into the attribute-related encoder and processed sequentially to obtain the attribute-related style code, as shown in the following formula (8).

[0162] S con =Emb c (V con (8)

[0163] In equation (8), S con This represents the attribute-related style code obtained after the attribute-related vector has been processed by the attribute-related encoder.

[0164] Step 3.3: Use the output of the previous layer of the decoder as the input of the next layer for direct calculation, as shown in the following formula (9);

[0165]

[0166] In equation (9), This represents the output of the i-th layer of the decoder. This represents the output of the (i-1)th layer. This represents the output of the attribute-independent feature extractor in the i-th layer;

[0167] Step 3.4: The output of the fourth layer of the decoder is input into the fifth layer of the decoder, sequentially fed into a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a batch normalization layer (BN), and a ReLU activation function layer, to obtain the output of the last layer of the decoder.

[0168] Step 3.5, the final result of 3.4 After passing through a transposed convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, and a Tanh activation function layer, a preliminary synthesized face image is obtained, i.e., the preliminary generated image Img1 is obtained;

[0169] The fourth step is to use the mask denoising module and feature selection unit to decode the features of the last layer of the encoder to obtain the final attribute mask.

[0170] Step 4.1: The target attribute label and the original attribute label are sequentially input into the mask denoising module along with the last layer features of the encoder. The operations of formulas (5), (6), (7), (8), and (9) are performed to obtain the feature output of the fourth layer of the mask denoising module, which are the output features with the target attribute vector as input. Output features with the original attribute vector as input

[0171] Step 4.2, use the features obtained in 4.1 and The inputs are respectively fed into the coordinate space attention layer CorSpa, as shown in the following formulas (10) and (11);

[0172]

[0173]

[0174] In equation (10), F mt for The result after the coordinate space attention layer; in equation (11), F mo for The result after passing through the coordinate space attention layer;

[0175] Step 4.3: The results obtained in 4.2 are input into the last layer of the mask denoising module, passing sequentially through a transposed convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, a batch normalization layer (BN), and a ReLU activation function layer, to obtain the final output features of the mask denoising module, which are the target output features with the target attribute vector as input. and the original output features with the original attribute vector as input.

[0176] In step 4.4, the target output features and the original output features obtained in step 4.3 are sequentially input into a transposed convolutional layer and a sigmoid layer with a kernel size of 3×3, a stride of 1, and padding of 1, to obtain the final mask image, which is Mask. t and Mask o Among them, Mask t This is an edit mask that takes the target attribute vector as input. The black areas in the edit mask contain the attribute regions to be edited, as well as noise regions with identity and background information. o The reconstruction mask is taken as input from the original attribute vector. The black areas in the reconstruction mask only contain noisy areas with identity and background.

[0177] Step 4.5, apply the editing mask obtained in step 4.4. t and reconstructed mask o Selective blending is performed as shown in the following formula (12), which removes the black areas that are the same as the noise areas with identity and background in the reconstruction mask, and highlights the black attribute areas unique to the editing mask;

[0178] Mask f =1+Mask t -Masko (12)

[0179] In equation (12), Mask f This is the final attribute mask obtained after passing through the mask denoising module and the mask fusion module;

[0180] Step 5, apply the mask f Img1 and the input raw image Img real The final image is obtained by fusion, as shown in formula (13) below;

[0181] Img f =Img real *Mask f +Img1*(1-Mask f (13)

[0182] In formula (13), Img f This represents the final synthesized face image obtained after the generator has processed it.

[0183] The fifth step is to use a discriminator to perform image discrimination operations;

[0184] Step 5.1, the final synthesized face image is input into the pre-trained model proposed by He Y, Yu N, Keuper M et al. in "Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis" published in the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021) to determine the probability that the image has been tampered with, as shown in the following formula (14);

[0185] Edit_P fake =D edit (X) (14)

[0186] In equation (14), D edit This is a pre-trained tamper detection model, where X is the input image data and Edit_P is the input image data. fake The probability that the image has been tampered with is detected after the tamper detection model is used;

[0187] Step 5.2: The image data is input into the first convolutional layer of the second discriminator, passing sequentially through a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, an IN instance normalization layer, and a LeakyReLU activation function layer, to obtain the output of the first convolutional layer of the second discriminator.

[0188] Step 5.3, take the output of the previous layer of the second discriminator as the input of the next convolutional layer, as shown in the following formula (15);

[0189]

[0190] In equation (15), This represents the output of the i-th layer convolutional module of the second discriminator. This represents the output of the i*1th layer convolutional module of the second discriminator;

[0191] Step 5.4, The input is fed into the classification head of the second discriminator to obtain the discrimination probability of the image with respect to each attribute, as shown in the following formula (16);

[0192]

[0193] In equation (16), D cls This represents the probability value of each attribute obtained by the discriminator; Linear represents a fully connected layer, which performs fully connected mapping.

[0194] Step 5.5, will The input is fed into the adversarial head of the second discriminator to obtain the probability that the image is judged as a real image, as shown in the following formula (17);

[0195]

[0196] In equation (17), D adv This represents the probability that an image is judged as a real image after passing through a second discriminator.

[0197] At this point, the above steps complete the construction of the generator and discriminator network structure for the face attribute editing method based on mask denoising and feature selection. The final generation result of the generator and the discrimination result obtained by the discriminator can be obtained from the input image data.

[0198] Loss function setting for a face attribute editing method based on mask denoising and feature selection;

[0199] The generator loss function is set as shown in formula (18) below;

[0200] L g =λ1*L adv +λ2*L rec +λ3*L cls +λ4*L edit (18)

[0201] In equation (18), L g Let λ be the total loss function of the generator. i ,i∈(1,2,3,4) are the hyperparameters balancing the various losses, L adv For the adversarial loss of the generator, Lrec L is the reconstruction loss of the generator. cls L is the classification loss of the generator. edit The tamper detection loss is calculated after the image generated by the generator is processed by the tamper detection model.

[0202] The adversarial loss of the generator is shown in the following formula (19);

[0203] L adv =-E[D adv (G(img,label) t (19)

[0204] In equation (19), E[·] represents the mean operation, and D adv This represents the discriminator's adversarial head's judgment of the image's realism, G represents the generator's image generation process, img is the original image data input to the generator, and label is... t The attribute tag for the target attribute;

[0205] The reconstruction loss of the generator is shown in Equation (20);

[0206] L rec =E||img-G(img,Label) o )||1 (20)

[0207] In equation (20), ||·||1 is the L1 norm, and label o These are the original attribute labels corresponding to the input image;

[0208] The classification loss of the generator is shown in the following formula (21);

[0209] L cls =CrossEntropy(Label) t D cls (G(img,label) t ))) (twenty one)

[0210] In equation (21), D cls CrossEntropy is used to perform classification prediction of each attribute of the input image by the discriminator classification head;

[0211] The tamper detection loss of the generator is shown in the following formula (22);

[0212] L edit =CrossEntropy(1,D) edit (G(img,label) t ))) (twenty two)

[0213] In equation (22), D edit To introduce a pre-trained tamper detection discriminator, the probability of the input image after tamper detection by the tamper detection model is subjected to cross-entropy loss with the all-1 tensor to obtain the final tamper detection loss.

[0214] The loss function for the second discriminator is set as shown in formula (23) below;

[0215] L d =λ5*L adv +λ6*L cls (twenty three)

[0216] In equation (23), λ5 and λ6 are hyperparameters of the weights between the discriminator adjustment loss, and L d For the total loss of the discriminator, L adv For the adversarial loss of the second discriminator, L cls The classification loss of the second discriminator;

[0217] The adversarial loss of the second discriminator is shown in equation (24);

[0218] L adv =-E[D adv (img)]+E[D adv (G(img,label) t ))] (twenty four)

[0219] The classification loss of the second discriminator is shown in equation (25);

[0220] L cls =CrossEntropy(label) o D cls (img)) (25)

[0221] After the above operations, the overall process of the face attribute editing method based on mask denoising and feature selection, including model building, training and optimization, is completed. The final trained model can be used to perform face attribute editing operations.

[0222] Figure 9 The image shows the generated effect of the face attribute editing method based on mask denoising and feature selection of the present invention. The input image is reconstructed, bangs are added, glasses are put on, mouth opening and closing is adjusted, beard is added, and gender is changed. The generated image shows that the present invention can accurately edit the attributes of face images and ensure the invariance of other areas except the edited area.

[0223] This invention addresses facial attribute editing. The designed mask denoising module generates only the mask relevant to the attribute to be changed during attention mask generation, while removing background masks containing facial information. This effectively completes the attribute editing task while preserving the consistency of non-edited areas. The designed feature selection unit filters image features at the feature level, retaining attributes-irrelevant features lost due to network depth. During decoding, it performs style fusion of features based on the attribute style code enriched by the attribute encoder from the binary attribute vector. The pre-trained tamper detection model improves the authenticity of generated images by better detecting whether they have been tampered with.

[0224] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A face attribute editing method based on mask denoising and feature selection, the method comprising the following: First, the publicly available face dataset is preprocessed to obtain the original input image Img. real Each face image corresponds to a binary attribute label; Construct a face attribute editing network that performs mask denoising and feature selection. The network consists of two parts: a generator and a discriminator. The generator adopts an encoder-decoder structure, and a mask denoising module is added on this basis. The encoder includes 5 convolutional blocks, the decoder includes 4 transposed selection blocks and 1 transposed convolutional block, and the mask denoising module includes 4 transposed selection blocks, 1 coordinate space attention layer and 1 transposed convolutional block. The mask denoising module generates a mask for face image reconstruction and an editing mask for face attribute editing. Selective fusion of the two masks can obtain a mask that removes background noise and contains only the region attribute mask that needs to be edited. This attribute mask is then combined with the original image and the image generated by the decoder. The transpose selection block includes a transpose convolutional layer and a feature selection unit. The feature selection unit includes an attribute-independent feature extractor and a target style fusion unit. The attribute-independent feature extractor includes a transpose convolutional layer, an Adain adaptive instance normalization layer, a ReLU activation layer, and an attribute-independent encoder. The target style fusion unit includes a channel concatenation operation, an Adain adaptive instance normalization layer, a ReLU activation layer, and an attribute-related encoder. Both the attribute-independent encoder and the attribute-related encoder are composed of three fully connected modules, each including a fully connected layer, a PixelNorm normalization layer, and a LeakyReLU activation layer. The feature selection unit first inputs the features output by the corresponding convolutional block in the encoder into the attribute-independent feature extractor, and extracts attribute-independent features according to the attribute-independent style code. The extracted attribute-independent features are then concatenated with the features output by the transposed convolutional layer in the corresponding transposed selection block in the decoder or mask denoising module. The concatenated features are then input into the target style fusion unit and fused with the attribute-related style code. The output is the feature after retaining the attribute-independent information and fusing the attribute-related style information. The attribute-independent style code is obtained by inputting the attribute-independent vector into the attribute-independent encoder, the attribute-dependent style code is obtained by inputting the attribute-dependent vector into the attribute-dependent encoder, and the attribute-independent vector is obtained by inverting the attribute-dependent vector.

2. The face attribute editing method based on mask denoising and feature selection according to claim 1, characterized in that, The convolutional block consists of a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a batch normalization layer (BN), and a LeakyReLU activation function. The transposed selection block consists of a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, and a feature selection unit. The transposed convolutional block consists of a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a Batch Normalization (BN) layer, and a ReLU activation function.

3. The face attribute editing method based on mask denoising and feature selection according to claim 1, characterized in that, The coordinate space attention layer includes a coordinate attention module and a spatial attention module. The input features first pass through the coordinate attention module, and the resulting features are then multiplied with the input features before being input into the spatial attention module. The features that pass through the spatial attention module are multiplied with the features that are input into the spatial attention module to obtain the final output features. The overall process of the coordinate attention module is as follows: The input features are first subjected to average pooling in the X direction of C×H×1 and average pooling in the Y direction of C×1×W. The two pooled features are concatenated along the channel. Then, a convolutional layer with a kernel size of 1×1, a stride of 1, and padding of 0 is input. Then, a normalized BN layer and a ReLU activation function layer are input. The output features are segmented along the channel into two features of C×H×1 and C×1×W. The two output features of different sizes are passed through a convolutional layer Conv and a Sigmoid activation function layer respectively and then fused to obtain the final output feature with dimensions of C×H×W. The overall process of the spatial attention module is as follows: the input features first undergo max pooling and average pooling operations in sequence, and then pass through a convolutional layer (Conv) and a sigmoid activation function layer to output the final features.

4. The face attribute editing method based on mask denoising and feature selection according to claim 1, characterized in that, The mask denoising module is used to obtain attribute coordinate position mask maps during face editing. The input of the mask denoising module is the features of the image after the encoder. The features are input into the mask denoising module and pass through 4 layers of transposed selection blocks. The obtained features are then input into the coordinate space attention layer to weight the dimension, height, and width of the features, and to weight the spatial position information related to the attributes. The weighted features are then input into the last transposed convolution block. Depending on whether the input is the target attribute label or the original attribute label, two attention masks are obtained: an editing mask related to the attributes and a reconstruction mask that only reconstructs the image and is unrelated to the attributes. The two masks are selectively fused to obtain an attribute mask that retains only the positional region information related to the attributes. The attribute mask is then fused with the image output by the decoder and the original image to obtain the final result.

5. The face attribute editing method based on mask denoising and feature selection according to claim 4, characterized in that, The selective fusion process is carried out according to formula (12): Remove black areas that are identical to the noisy areas with identity and background in the reconstruction mask, and highlight the black attribute areas unique to the editing mask; Mask f =1+Mask t -Mask o (12) In equation (12), Mask f The final attribute mask is obtained after the mask denoising module and selective fusion; Mask t To edit the mask, Mask o To reconstruct the mask.

6. The face attribute editing method based on mask denoising and feature selection according to claim 1, characterized in that, The discriminator includes two discriminators. The first discriminator is a tamper detection model used to determine the probability that the image has been tampered with. The second discriminator consists of a 5-layer convolutional module, an adversarial head used to determine whether the image is real or fake against the adversarial loss, and a classification head used to classify the image. Both the adversarial head and the classification head are fully connected layers.

7. The face attribute editing method based on mask denoising and feature selection according to claim 6, characterized in that, The convolutional module of the second discriminator includes a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, an IN instance normalization layer, and a LeakyReLU activation function layer connected in sequence.

8. A face attribute editing method based on mask denoising and feature selection, the specific steps of which are: The first step is to extract features from the input face image data layer by layer through the encoder network; Step 1.1: Center-crop the input face image data, adjust the resolution to 128*128, normalize and tensor the data to obtain the preprocessed face image data, which is used as the input original image Img. real ; Step 1.2: The face data from step 1.1 above is input into the first convolutional block of the encoder. The size of a single face image is 128*128*3. After passing through a convolutional layer with a kernel size of 4*4, a stride of 2, and padding of 1, it passes through a batch normalization layer (BN) and a LeakyReLU activation function to obtain a feature map with a size of 64*64*64. Step 1.3: The output of the previous convolutional block is used as the input of the next convolutional block for convolution operation, as shown in the following formula (1); In equation (1), This represents the feature output of the i-th convolutional block of the encoder. Represents the features output by the (i-1)th convolutional block. Conv indicates the operation of inputting the features into the convolutional block. The specific operation is as follows: first, perform a convolution operation with a kernel size of 4*4, a stride of 2, and padding of 1; then, perform batch normalization (BN); and finally, perform LeakyReLU activation. The second step is to use the feature selection unit to extract attribute-independent features from the features of different layers of the encoder and retain the attribute-independent features. The encoder features of different layers are respectively input into the attribute-independent vector in the attribute-independent feature extractor in the feature selection unit to extract attribute-independent features, as shown in the following formula (2); In equation (2), V dif This represents an attribute-independent vector that is independent of the attributes of a face image. This represents feature information unrelated to the attributes of the i-th layer corresponding to the number of layers in the decoder or mask denoising module. Cfe represents the feature of the corresponding layer in the encoder corresponding to the i-th layer of the decoder or mask denoising module. The specific operation method is shown in the following formula (3). In equation (3), ConvT represents the transpose convolution operation on the input features, Adain represents the adaptive instance normalization operation on the features, ReLU represents the ReLU activation function activation on the input features, and Emb... d This indicates that an attribute-independent encoder is used for encoding. The attribute-independent encoder contains three consecutive sets of FC fully connected layer mappings, PixelNorm normalization, and LeakyReLU activation function. The attribute-independent vector is input into the attribute-independent encoder and processed sequentially to obtain the attribute-independent style code, as shown in the following formula (4). S dif =Emb d (V dif ) (4) In equation (4), S dif This represents the attribute-independent style code obtained after passing through the attribute-independent encoder. The third step involves using a decoder and a feature selection unit to decode the features of the last layer of the encoder, resulting in an image for preliminary facial attribute editing. Step 3.1: Input the output features of the last layer of the encoder into the transposed convolutional layer in the first layer of the decoder to obtain the intermediate features without feature selection, as shown in the following formula (6); In equation (5), The intermediate features are obtained by passing the output features of the last layer of the encoder through the transposed convolutional layer in the first layer of the decoder. Step 3.2, the features obtained in 3.1 Input attribute correlation vector V con Features unrelated to the attributes of the first layer of the corresponding decoder obtained in step two. The input is fed into the target style fusion unit of the feature selection unit in the first layer, as shown in the following formula (6); In equation (6), Tsf is the output of the first layer of the decoder, and Tsf is the target style fusion unit. The specific operation is shown in the following formula (7); In equation (7), Concat represents the channel splicing operation, and Emb... c This indicates that the attribute-related vector is encoded using an attribute-related encoder. The attribute-related encoder contains three consecutive sets of FC fully connected layer mappings, PixelNorm normalization, and LeakyReLU activation function. The attribute-related vector is input into the attribute-related encoder and processed sequentially to obtain the attribute-related style code, as shown in the following formula (8). S con =Emb c (V con ) (8) In equation (8), S con This represents the attribute-related style code obtained after the attribute-related vector has been processed by the attribute-related encoder. Step 3.3: Use the output of the previous layer of the decoder as the input of the next layer for direct calculation, as shown in the following formula (9); In equation (9), This represents the output of the i-th layer of the decoder. This represents the output of the (i-1)th layer, where Tsf is the target style fusion unit; Step 3.4: The output of the fourth layer of the decoder is input into the fifth layer of the decoder, sequentially fed into a transposed convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, a batch normalization layer (BN), and a ReLU activation function layer, to obtain the output of the last layer of the decoder. Step 3.5, take the final result from step 3.

4. After passing through a transposed convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, and a Tanh activation function layer, a preliminary synthesized face image Img1 is obtained; The fourth step is to use the mask denoising module and feature selection unit to decode the features of the last layer of the encoder to obtain the final attribute mask. Step 4.1: The target attribute label and the original attribute label are sequentially input into the mask denoising module along with the last layer features of the encoder. The operations of formulas (5), (6), (7), (8), and (9) are performed to obtain the feature output of the fourth layer of the mask denoising module, which are the output features with the target attribute vector as input. Output features with the original attribute vector as input Step 4.2, use the features obtained in step 4.1 and The inputs are respectively fed into the coordinate space attention layer CorSpa, as shown in the following formulas (10) and (11); In equation (10), F mt for The result after the coordinate space attention layer; in equation (11), F mo for The result after passing through the coordinate space attention layer; Step 4.3: The results obtained in step 4.2 are input into the last layer of the mask denoising module, passing sequentially through a transposed convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, a batch normalization layer (BN), and a ReLU activation function layer, to obtain the final output features of the mask denoising module, which are the target output features with the target attribute vector as input. and the original output features with the original attribute vector as input. In step 4.4, the target output features and the original output features obtained in step 4.3 are sequentially input into a transposed convolutional layer and a sigmoid layer with a kernel size of 3×3, a stride of 1, and padding of 1, to obtain the final mask image, which is Mask. t and Mask o Among them, Mask t This is an edit mask that takes the target attribute vector as input. The black areas in the edit mask contain the attribute regions to be edited, as well as noise regions with identity and background information. o The reconstruction mask is taken as input from the original attribute vector. The black areas in the reconstruction mask only contain noisy areas with identity and background. Step 4.5, apply the editing mask obtained in step 4.

4. t and reconstructed mask o Selective fusion is performed as shown in the following formula (12), which removes the black areas that are the same as the noise areas with identity and background in the reconstruction mask, and highlights the black attribute areas unique to the editing mask. Mask f =1+Mask t -Mask o (12) In equation (12), Mask f This is the final attribute mask obtained after the mask denoising module and selective fusion; Step 4.6, apply the mask f The initial synthesized face image Img1 and the input original image Img real The final image is obtained by fusion, as shown in formula (13) below; Img f =Img real *Mask f +Img1*(1-Mask f ) (13) In formula (13), Img f This represents the final synthesized face image obtained after the generator has processed it. The fifth step is to use a discriminator to perform image discrimination operations; Step 5.1: Input the final synthesized face image into the first discriminator of the pre-trained tamper detection to determine the probability that the image has been tampered with, as shown in the following formula (14); Edit_P fake =D edit (X) (14) In equation (14), D edit This is the first discriminator for tamper detection after pre-training, where X is the input image data and Edit_P fake The first discriminator after tamper detection detects the probability that the image has been tampered with; Step 5.2: The second discriminator consists of a 5-layer convolutional module, an adversarial head for determining whether an image is real or fake due to adversarial loss, and a classification head for classifying the image. Both the adversarial head and the classification head are fully connected layers. Image data is input into the first convolutional module of the second discriminator, passing sequentially through a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, an IN instance normalization layer, and a LeakyReLU activation function layer, to obtain the output of the first convolutional layer of the second discriminator. Step 5.3, take the output of the previous layer of the second discriminator as the input of the next convolutional layer, as shown in the following formula (15); In equation (15), This represents the output of the i-th convolutional module of the discriminator. This represents the output of the (i-1)th convolutional module of the discriminator; Step 5.4, The input is fed into the classification head of the second discriminator to obtain the discrimination probability of the image with respect to each attribute, as shown in the following formula (16); In equation (16), D cls This represents the probability value of each attribute obtained by the second discriminator; Linear represents a fully connected layer, which performs fully connected mapping. Step 5.5, will The input is fed into the adversarial head of the second discriminator to obtain the probability that the image is judged as a real image, as shown in the following formula (17); In equation (17), D adv This represents the probability that an image is judged as a real image by the discriminator. At this point, the above steps complete the construction of the generator and discriminator network structure for the face attribute editing method based on mask denoising and feature selection, and obtain the final generation result of the generator and the discrimination result obtained by the discriminator through the input image data.

9. The face attribute editing method based on mask denoising and feature selection according to claim 8, wherein the loss function of the method is set as shown in the following formula (18); L g =λ1*L adv +λ2*L rec +λ3*L cls +λ4*L edit (18) In equation (18), L g Let λ be the total loss function of the generator. i ,i∈(1,2,3,4) are the hyperparameters balancing the various losses, L adv For the adversarial loss of the generator, L rec L is the reconstruction loss of the generator. cls L is the classification loss of the generator. edit The tamper detection loss is calculated after the image generated by the generator is processed by the tamper detection model. The adversarial loss of the generator is shown in the following formula (19); L adv =-E[D adv (G(img,lkable t ))] (19) In equation (19), E[·] represents the mean operation, and D adv This indicates that the second discriminator, acting as an adversarial head, judges the realism of the image. G represents the process of the generator generating the image, img is the original image data input to the generator, and label... t The attribute tag for the target attribute; The reconstruction loss of the generator is shown in Equation (20); L rec =E||img-G(img,lable o )||1 (20) In equation (20), ||·||1 is the L1 norm, and label o These are the original attribute labels corresponding to the input image; The classification loss of the generator is shown in the following formula (21); L cls =CrossEntropy(lable t ,D cls (G(img,lable t ))) (21) In equation (21), D cls The second discriminator classification head performs classification predictions for each attribute of the input image, with Crossentropy being the cross-entropy loss; The tamper detection loss of the generator is shown in the following formula (22); L edit =CrossEntropy(1,D edit (G(img,lable t ))) (22) In Equation (22), the probability of the input image after being detected by the tamper detection model is subjected to cross-entropy loss with the full-1 tensor to obtain the final tamper detection loss; The loss function for the second discriminator is set as shown in formula (23) below; L d =λ5*L adv +λ6*L cls (23) In equation (23), λ5 and λ6 are hyperparameters of the weights between the discriminator adjustment loss, and L d For the total loss of the discriminator, L adv For the adversarial loss of the second discriminator, L cls The classification loss of the second discriminator; The adversarial loss of the second discriminator is shown in equation (24); L adv =-E[D adv (img)]+E[D adv (G(img,lable t ))] (24) The classification loss of the second discriminator is shown in equation (25); L cls =CrossEntropy(lable o ,D cls (img) (25)。 10. The face attribute editing method based on mask denoising and feature selection according to any one of claims 1-9, characterized in that, The dataset used is the CelebA open dataset from the Chinese University of Hong Kong.

Images