A face feature conversion method based on feature information mapping and attention mechanism

Abstract

The application discloses a face feature conversion method based on feature information mapping and an attention mechanism, and comprises the following steps: 1) constructing a face feature editing model; the face feature editing model comprises a generator, a discriminator and a loss network; 2) taking a target domain label to be converted and an original image as inputs of the model, and outputting a converted generated image; first, the target domain label is converted into deep feature information data by a feature information mapping network in the generator, and then is added to an image generation network in the process of converting the original image into the generated image to realize conversion, finally, the generated image is input into the discriminator to evaluate the authenticity of the generated image. The method is modified by modifying the structure of the generator and the structure of the discriminator, and a cyclic structure in the model is constructed, which not only helps correct generation of the image, and by adding an attention mask generation network and a context loss, the performance of preserving the original information of the image is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of face feature conversion, and particularly relates to a face feature conversion method based on feature information mapping and an attention mechanism. BACKGROUND

[0002] In recent years, the technology in the fields of computer graphics and computer vision is developing rapidly, which also promotes the progress of the fields related to the modification, editing and conversion of face features. With the birth of the generative adversarial network, the content generation artificial intelligence has become a hot topic of the times and an important research direction. The editing and modification of face features is a multi-field to multi-field conversion task, which can be applied to optimize distorted images and low-quality images in real life, better highlight face features, and help face images cope with detection and analysis more effectively; or polish the completed images, reduce the work difficulty and workload of film and television practitioners, and help art creators and enthusiasts better realize the display of inspiration and the creation of artistic works. However, the current related technology still has some problems in this field: (1) the current feature conversion method based on multi-field image conversion still has weak control over discrete feature labels, and the effective control degree of labels on features and attributes needs to be further strengthened, and the control precision of labels on feature generation also needs to be further improved. (2) The existing method is easy to lose detailed features in the process of feature conversion and attribute editing, and the commonly used cycle consistency loss reduces the conversion effect of features and attributes of images in some application scenarios to a certain extent. SUMMARY

[0003] The purpose of the present application is to overcome the shortcomings of the prior art and provide a face feature conversion method based on feature information mapping and an attention mechanism. By modifying the structure of the generator and the discriminator, a cyclic structure is constructed inside the model, which not only helps to generate correct images, but also improves the performance of preserving the original information of the image by adding an attention mask generation network and a context loss.

[0004] A face feature conversion method based on feature information mapping and an attention mechanism, comprising the following steps:

[0005] Step 1, constructing a face feature editing model FFEGAN;

[0006] The face feature editing model FFEGAN comprises a generator, a discriminator and a loss network.

[0007] The generator is composed of a feature information mapping network, an image generation network and an attention mask generation network.

[0008] Step 2: Use the labels of the target domain to be transformed and the original image as a face feature editing model.

[0009] The FFEGAN model takes as input and outputs a generated image that transforms features from the original domain to the target domain.

[0010] First, the target domain label is transformed into deep feature information data by the feature information mapping network in the generator. Then, it is added to the image generation network during the process of converting the original image into the generated image to control the conversion. Finally, the generated image enters the discriminator for evaluation, assesses the realism of the generated image, and classifies the generated image into its domain.

[0011] Further, in step 1, the generator G is an image generation network G that generates the target image through an encoder-decoder structure. gen Attention Mask Generation Network G, which utilizes a multi-head self-attention mechanism to generate attention masks. mask It consists of three parts: a feature information mapping network M that converts one-hot encoding into high-dimensional latent information;

[0012] Assume the original face image is x, and the feature label assigned to the original domain is c. 0 The target domain is assigned the feature label c t The generator uses the original face image x and the target domain label c. t As input, generate a target image y, and match its feature labels with c. t It matches the original facial information and attributes while preserving them.

[0013] The target domain label c t The target domain label c will be generated randomly in the generator during model training. t It will be mapped by the feature information mapping network M into a high-dimensional latent code k containing implicit feature information. t This vector will be injected into the image generation network G multiple times. gen The image generation network G enhances the control of the generated images by adding labels. gen The mathematical logic expression is:

[0014] G gen (x,k t )=y (1)

[0015] The mathematical logic expression for generator G is:

[0016] G(x,c t ) = G gen (x,M(c t ))=y (2)

[0017] In formulas (1) and (2), x is the original image input into the generator, y is the new image generated by the generator; M is a feature information mapping network.

[0018] When the generation work of the generator runs to the decoder part, the high-dimensional latent code k t of the feature information is implied when the information of different network layers is up-sampled. gen The label will be injected into the image generation network G gen multiple times at different layers to strengthen the control of the label at different layers of the decoder.

[0019] Further, the encoder part of the image generation network G gen is composed of three down-sampling layers and the same number of bottleneck layers, the down-sampling layers are used to perform convolution operations on the input data to extract the feature information contained in the image, and adaptive instance normalization method is used for normalization processing after each convolution operation.

[0020] The decoder part of the image generation network G gen is symmetrical in structure with the encoder part, and the decoder part is composed of three bottleneck layers and the same number of up-sampling layers, the up-sampling layers are used to perform transpose convolution operations, and the bottleneck layers use residual blocks for jump connection, the working principle of the residual block in the image generation network G gen is represented by formula as:

[0021] y=σ(F(x,W)+x) (3)

[0022] In formula (3), σ(·) represents an activation function, F(·) represents a residual function, and W represents all weights in the residual block;

[0023] For a deeper layer L, its relationship with the previous layer l is:

[0024]

[0025] The adaptive instance normalization method is used for normalization processing after each convolution operation, and the calculation process of adaptive instance normalization is:

[0026] AdaIN(X,k t )=(1+f(k t )1)(IN(X))+f(k t )2 (5)

[0027] In formula (5), X is the feature distribution of the image, and f(k t )1 and f(k t )2 are two parameters generated by the slicing operation.

[0028] Further, the attention mask generation network G mask The architecture of the image generation network G gen The architecture of the attention mask generation network G mask After the last layer of the bottleneck layer, a calculation operation for multi-head self-attention mechanism calculation is added, the number of heads of the multi-head self-attention mechanism is set to 3, and the attention mask generation network G mask The up-sampling layer part of the attention mask generation network G mask The number of convolution kernels of the last layer of the bottleneck layer is set to 1, and the attention mask generation network G mask The original image x is taken as input, and finally an attention mask AttM with one channel and the same resolution as the original face image x is output, the mask AttM contains information of relevant feature positions of the original image and weight information of different regions in the conversion work;

[0029] The attention mask generation network G mask Based on the correct generation of the attention mask AttM, the image generation network G gen Only the relevant regions marked by the mask need to be converted, and other regions remain unchanged to complete a successful conversion, and the generator G outputs the image y * The process of the image generation network G

[0030] y * =(1-AttM)·x+AttM·y' (6)

[0031] In formula (6), x is the original image input into the generator, y' is the relevant region marked by the mask, and AttM is the attention mask;

[0032] Then the working principle of the generator G as a whole is represented as:

[0033] G(x,c t )=G mask (x)·G gen (x,M(c t ))+(1-G mask (x))·x (7)。

[0034] Further, in step 1, the discriminator includes two discriminant branches, the two discriminant branches are respectively an adversarial discriminant branch D adv and a classification discriminant branch D cls ; the adversarial discriminant branch D advIn the full convolution form, after the input image is subjected to the convolution operation, the output result will continue to be projected into an N-order projection matrix containing discriminative information through the convolution operation, and each value in the N-order projection matrix corresponds to the evaluation result of the discriminator on a certain block region of the image; the classification discriminant branch D cls for outputting a probability distribution describing the features of the input image, the probability distribution being used to assist the face feature editing model FFEGAN in learning various feature attributes and classifying them into correct feature labels;

[0035] The overall structure of the discriminator is represented by the formula as follows:

[0036] D={D adv ,D cls} (8)。

[0037] Further, the function of the feature information mapping network is represented by the formula as follows:

[0038] k t =M(c t ) (9)

[0039] In formula (9), M is the feature information mapping network, c t is the target domain label, and k t is a high-dimensional latent code containing feature information;

[0040] The input of the feature information mapping network is an n-dimensional one-hot encoding, which needs to pass through nine dense layers and an activation layer after each dense layer after entering the feature information mapping network. The neurons in the one-hot encoding will perform affine transformation on the one-hot encoding when passing through the dense layers, and the one-hot encoding will be subjected to nonlinear transformation after passing through the activation layer. After passing through the nine dense layers and the nine activation layers, the feature information mapping network will output a high-dimensional latent code. The operation of affine transformation of the one-hot encoding in the dense layer is represented by the formula as follows:

[0041]

[0042] In formula (10), z is the final output value of the affine transformation; ω k represents the weight between any neuron in the kth layer and any neuron in the k+1th layer; m is the maximum number of layers of the feature information mapping network; and c is an additional bias for offsetting the affine transformation bias in the calculation process.

[0043] The input layer and the following seven dense layers of the feature information mapping network are composed of 512 neurons, and the output layer of the feature information mapping network is composed of N neurons.

[0044] The activation layer adopts a leaky linear rectifier unit Leaky ReLU as an activation function, and the mathematical expression of the Leaky ReLU activation function is:

[0045]

[0046] Further, the image generation network G gen By using a loss network pre-trained in the image classification field to calculate the context loss, the generated image and the original image are input into the loss network, the feature distribution of the image is extracted in the last activation layer before the dense layer, and the context loss is calculated, and the feature information extracted in the layer can best reflect the potential deep features of the image.

[0047] Further, in the training process of the FFEGAN model, in order to ensure that the generator G can maximize the preservation of other information of the original image x while converting the original feature to the target feature when generating a new image y with the target feature, the reorganization loss is applied to the generator, and the definition of the reorganization loss is represented as:

[0048] L re =||x-G(G(x,c t ),c o )||1 (12)

[0049] In formula (12), x is the original image input into the FFEGAN model; y is the generated image output by the FFEGAN model; c o is the label of the original domain corresponding to the original image;

[0050] For the two different discriminant branches of the discriminator, the adversarial discriminant branch D adv and the classification discriminant branch D cls , in the process of model training, due to the different purposes, the goal of the generator G is to continuously reduce and reduce the adversarial loss L adv , while the goal of the discriminator D is to continuously increase and increase the adversarial loss; the corresponding loss functions are also completely different;

[0051] The loss function of the adversarial discriminant branch D adv is represented as:

[0052]

[0053] In formula (13), L adv is the adversarial loss; x is the original image input into the FFEGAN model, D adv (x) is the probability distribution of the image output by the adversarial discriminant branch of the discriminator; c t is the target domain label of the conversion;

[0054] The classification discriminant branch D cls The loss function is represented as:

[0055] ① When training the discriminator D, the function of the classification discriminant branch D cls is to evaluate the attribute category of the real image with the classification loss, so the loss function is represented as:

[0056]

[0057] In formula (14), D cls (c o |x) is the confidence probability distribution of the classification discriminant branch on whether the input image is the output feature attribute; x is the original image input into the FFEGAN model, and c o is the original domain label of the original image feature attribute;

[0058] ② When training the generator G, the function of the classification discriminant branch D cls helps the generator G to generate images conforming to the target domain label c t , so the loss function is represented as:

[0059]

[0060] In formula (15), D cls (c t |G(x, c t )) is the confidence probability distribution of the classification discriminant branch on whether the input image belongs to the target feature attribute; x is the original image input into the FFEGAN model, and c t represents the target domain of the conversion task, i.e., the labeled label of the target feature attribute.

[0061] Further, the context loss, for the original input image x and the generated image y, regards them as the set of all features contained in the respective images, denoted as: X1={x i}, Y1={y j}; In the calculation, X1 and Y1 need to be the same size as a set, i.e., |Y1|=|X1|, when the sizes of the two sets are different, additional features need to be added from the larger set;

[0062] First, the cosine distance of the features needs to be normalized, and the principle of the operation is represented as:

[0063]

[0064] In formula (16), d ij is the cosine distance between x i and y jcosine distance; l is an arbitrary positive integer not equal to j; ∈ is the offset of the denominator, ∈ = 1e-5;

[0065] Then, the normalized distance needs to be converted into the similarity between features, and this conversion process is completed by an exponential operation; at the same time, in order to make the similarity between different features have a unified scale when measuring, the similarity is defined as the quotient of the similarity between the original feature and the current feature and the total similarity between the original feature and all other features, and the principle of this process is expressed by the formula:

[0066]

[0067] In formula (17), h is a bandwidth parameter, h > 0; CX ij is the similarity between features i and j;

[0068] When formula (17) is applied to the full image, the definition of the similarity between two sample images is expressed by the formula:

[0069]

[0070] Suppose the FFEGAN model uses VGG19 as the loss network, extracts the feature set of the input image through the VGG19 loss network, and then applies the feature set to the context loss function, the definition of the context loss function is expressed by the formula:

[0071] L CX (x,y,k) = -log(CX(Φ k (x),Φ k (y))) (19)

[0072] In formula (19), Φ k (x) and Φ k (y) are the feature sets extracted from the kth layer of the loss network Φ of images x and y, respectively;

[0073] Further applying formula (19) to the FFEGAN model, the context loss function corresponding to the FFEGAN model is expressed as:

[0074] L CX (G(x),x,k) = -log(CX(Φ k (G(x)),Φ k (x))) (20)

[0075] According to the context loss function provided by formula (20), it can be understood that in the process of generating images, the generator G needs to minimize the context loss L CXThis will help to improve the similarity between the generated image and the original image at the feature set level; therefore, the total loss function of the generator G in the FFEGAN model is:

[0076]

[0077] In formula (21), L G represents the total loss of the generator G; L adv is the adversarial loss of the adversarial discrimination branch of the discriminator; is the classification loss of the classification discrimination branch of the discriminator; L re is the reorganization loss for maintaining the original information of the image; L cx is the context loss; λ cls , λ re and λ cx are weight coefficients of the classification loss, the reorganization loss and the context loss, which will affect the balance between maintaining the original information and feature conversion of the generated image;

[0078] The total loss function of the discriminator D of the FFEGAN model is:

[0079]

[0080] In formula (22), L D is the total loss of the discriminator; L adv is the adversarial loss of the adversarial discrimination branch of the discriminator; is the classification loss of the classification discrimination branch of the discriminator; λ cls is the weight coefficient of the classification loss, which will affect the balance of the discriminator before the adversarial generator and the correct classification label.

[0081] Compared with the prior art, the present application has the following beneficial effects:

[0082] 1、The method of the present application is designed to effectively solve the problem of weak control of one-hot encoding used in traditional models, and an FFEGAN model for editing facial features is designed; the model is improved based on the structure of the generative adversarial network, a feature information mapping network is added to the generator part, and the generator can effectively inject deep feature information into the generation process multiple times during training, improving the control of the attribute feature label generator. In the discriminator part of the network, by setting up an adversarial discrimination branch and a classification discrimination branch, the discriminator can better perform the discrimination task, and can also better guide the training of the generator using the classification loss, and the final experimental results show that the FFEGAN model effectively improves the control effect of the features and attributes, and solves the problem of insufficient control of feature attributes in the conversion process of the current method, and verifies that the FFEGAN model is superior to other existing methods in feature attribute conversion capability.

[0083] 2、The present application adds an attention mask generation network to the FFEGAN model, which can make the model better focus on the feature part that needs to be converted during image generation, and improve the retention level of the generated image to the original image information; in addition, due to the design defects of the reorganization loss itself, it will cause inhibition to the generation effect, the present application proposes to use a pre-trained loss network to calculate the context loss, thereby effectively solving this problem, and the final experimental results show that the improved method solves the problem of easy loss of details and original identity information in the image conversion process, effectively improves the conversion capability of the model, and verifies that the improved method is superior to other models and methods in conversion effect. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0085] Figure 1 is a feature conversion image generation process schematic diagram of the FFEGAN model of the present application;

[0086] Figure 2 is an image cycle verification process schematic diagram of the FFEGAN model of the present application;

[0087] Figure 3 is a structure diagram of the generator G of the FFEGAN model of the present application;

[0088] Figure 4 is a principle schematic diagram of the residual block of the present application;

[0089] Figure 5 is the structural diagram of the FFEGAN model discriminator D of the present application;

[0090] Figure 6 is the structural diagram of the feature information mapping network of the present application;

[0091] Figure 7 is a comparative schematic diagram of the Leaky ReLU activation function and other activation functions used in the present application

[0092] Figure 8 is a schematic diagram of the calculation principle of the context loss of the present application;

[0093] Figure 9 is a schematic diagram of part of the pictures in the CelebA dataset and the CelebA-HQ dataset used in the embodiment of the present application;

[0094] Figure 10 is a comparison diagram of the image conversion effect of the FFEGAN model and the control group model in the embodiment of the present application. DETAILED DESCRIPTION

[0095] In order to facilitate those skilled in the art to understand and implement the present application, the steps of the method of the present application are described in detail below. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or modifications to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0096] EMBODIMENT

[0097] The present application provides a face feature conversion method based on feature information mapping and attention mechanism, comprising the following steps:

[0098] Step 1, constructing a face feature editing model FFEGAN;

[0099] The face feature editing model FFEGAN includes a generator, a discriminator and a loss network;

[0100] The generator is composed of a feature information mapping network, an image generation network and an attention mask generation network;

[0101] Step 2, taking the label and original image of the target domain to be converted as the input of the face feature editing model FFEGAN

[0102] The FFEGAN model outputs a generated image that converts the feature from the original domain to the target domain;

[0103] The FFEGAN model proposed in this embodiment can be regarded as two parts, as shown in the figure Figure 1 The first part is to convert the input picture from the original domain to which the input picture belongs to the target domain. As shown in the figure Figure 2 The second part is to convert the image generated by the generator in the previous part back to the original domain, which is also the embodiment of the cycle verification idea.

[0104] Specifically, first, the target domain label is converted into deep feature information data by the feature information mapping network in the generator, and then added to the image generation network in the process of converting the original image into a generated image to realize the control of the conversion. Finally, the generated image enters the discriminator for evaluation, evaluates the authenticity of the generated image, and classifies the domain of the generated image. The evaluation of the generated image by the discriminator will help the generator learn how to generate better images.

[0105] In addition, after the feature conversion is completed, the image will also be added to the generator together with the domain label representing the original domain, and an original domain image converted twice will be obtained by re-generation; by comparing the differences between the original image and the image converted twice, the generator will learn how to preserve the original information in the conversion process, which will also improve the quality of the images generated by the generator and the conversion effect.

[0106] In step 1, as shown in the figure Figure 3 The generator G is composed of three parts: an image generation network G gen that generates target images through an encoder-decoder structure, an attention mask generation network G mask that generates attention masks using a multi-head self-attention mechanism, and a feature information mapping network M that converts one-hot encoding into high-dimensional latent information.

[0107] Assuming that the original face image is x, the feature label assigned to the original domain is c 0 , the feature label assigned to the target domain is c t , the generator takes the original face image x and the target domain label c t as input to generate a target image y * , and make its feature label match c t , while preserving other original facial information and attributes, including edge details, identity information, etc.

[0108] In addition, the target domain label c t will be generated in a random manner in the generator, which can prevent the generator from forming a dependence on the input label distribution during image generation, allowing better learning. During model training, the input feature label c tIt will be mapped by the feature information mapping network M into a high-dimensional latent code k containing implicit feature information. t This vector will be injected into the image generation network G multiple times. gen The image generation network G enhances the control of the generated images by adding labels. gen The mathematical logic expression is:

[0109] G gen (x,k t )=y (1)

[0110] The mathematical logic expression for generator G is:

[0111] G(x,c t ) = G gen (x,M(c t ))=y (2)

[0112] In formulas (1) and (2), x is the original image input to the generator, y is the new image generated by the generator, and M is the feature information mapping network.

[0113] When the generator reaches the decoder section, during the upsampling of information from different network layers, the high-dimensional latent encoding k, which contains implicit feature information, is... t It will be injected multiple times into the image generation network G gen Different levels are used to enhance the control of tags at different levels of the decoder.

[0114] The image generation network G gen The encoder part consists of three downsampling layers and the same number of bottleneck layers. The downsampling layers are used to perform convolution operations on the input data to extract the feature information contained in the image. After each convolution operation, an adaptive instance normalization method is used for normalization processing.

[0115] The image generation network G gen The decoder and encoder are structurally symmetrical. The decoder consists of three bottleneck layers and the same number of upsampling layers. The upsampling layers are used to perform transposed convolution operations, and the bottleneck layers use residual blocks for skip connections.

[0116] Specifically, in order to generate a believable target image, in the image generation network G... gen The bottleneck layers of the encoder and decoder sections employ shortcut connections, also known as residual blocks proposed by ResNet. This design allows the FFEGAN model to more effectively control the characteristics of the generated samples. The image generation network G in the FFEGAN model... genInstead of taking the target image as the output of the network, the network is required to output a face texture mask that matches the feature of the converted target domain as much as possible. In general, when an original face image x is input, the original domain is given a feature label c 0 , while the target domain is given a target label c t . The main task of the generator is to accept the original face image x and the target domain label c t as input, and further generate a target image y * with a feature label matching c t , while preserving other original information on the original image x that is irrelevant to the conversion work, including image material details, edge features, and other attributes.

[0117] The purpose of adding the residual block to the network is to solve the problems of gradient disappearance and gradient explosion, and to increase the depth of the network so that deep features can be better captured. In a deep neural network, as the number of network layers increases, the problems of gradient disappearance and gradient explosion become more and more serious. The residual block uses a skip connection to make the gradient directly backpropagate to the shallower network layer. In addition, the residual block can make the network better learn the features of the data by preserving the original input information.

[0118] As shown in Figure 4 , the working principle of the residual block in the image generation network G gen is represented by the formula as follows:

[0119] y = σ (F (x, W) + x) (3)

[0120] In formula (3), σ (·) represents an activation function, F (·) represents a residual function, and W represents all weights in the residual block;

[0121] For a deeper layer L, its relationship with the previous layer l is as follows:

[0122]

[0123] The adaptive instance normalization method is used for normalization processing after each convolution operation. The calculation process of adaptive instance normalization is as follows:

[0124] AdaIN (X, k t ) = (1 + f (k t )1) (IN (X)) + f (k t )2 (5)

[0125] In formula (5), X is the feature distribution of the image; let the dense layer be represented by a function f, then f (kt )1 and f(k t )2 are the two parameters generated by the slicing operation.

[0126] Furthermore, the attention mask generation network G mask Architecture and Image Generation Network G gen The architecture is similar, and the attention mask generation network G... mask After the last layer of the bottleneck layer, a computational operation is added for the multi-head self-attention mechanism, where the number of heads in the multi-head self-attention mechanism is set to 3. The attention mask generation network G mask In the upsampling layer, instance normalization is used to normalize the features in the attention mask generation network G. mask The last layer of the bottleneck layer has a convolutional kernel count of 1, and the attention mask generator network G... mask The original image x is taken as input, and the final output is an attention mask AttM with 1 channel and the same resolution as the original face image x. The mask AttM contains information about the relevant feature positions of the original image and the weight information of different regions in the conversion process.

[0127] Specifically, the addition of attention masks enables the model to better locate features, helping it to be less affected by other minor and irrelevant parts during the transformation of target features. In addition, attention masks also help the generator retain more of the original image's information during the transformation process.

[0128] In the attention mask generation network G mask Based on the correct generation of the attention mask AttM, the image generation network G gen A successful transformation can be completed by only transforming the relevant regions marked by the mask, while leaving other regions unchanged. The generator G outputs the image y. * The process can be expressed by the following formula:

[0129] y * =(1-AttM)·x+AttM·y' (6)

[0130] In formula (6), x is the original image input into the generator, y' is the relevant region marked by the mask, and AttM is the attention mask;

[0131] The overall working principle of generator G can be expressed as follows:

[0132] G(x,c t ) = G mask (x)·G gen (x,M(c t))+(1-G mask (x))·x (7).

[0133] Furthermore, in step 1, such as Figure 5 As shown, the discriminator includes two discriminant branches, namely, an adversarial discriminant branch D. adv And the classification and discrimination branch D cls The adversarial discriminant branch D adv Employing a fully convolutional approach, after the input image undergoes a convolution operation, its output is projected onto an N-order projection matrix containing discriminative information. Each value in this N-order projection matrix corresponds to the discriminator's evaluation of a specific region of the image; the classification discriminant branch D... cls This is used to output a probability distribution describing the features of the input image, which is used to assist the FFEGAN face feature editing model in learning various feature attributes and classifying them into the correct feature labels;

[0134] The overall structure of the discriminator is expressed by the following formula:

[0135] D = {D adv D cls} (8).

[0136] Furthermore, the Feature Information Mapping Network (FIM) can transform specific, discrete attribute labels into abstract, high-dimensional feature control information. As training progresses, the FIM learns to better perform various affine and nonlinear mappings on discrete attribute feature labels, outputting high-dimensional latent codes carrying refined feature information. Ideally, the FIM will fully learn the mapping methods, and the output information will contain all the semantic and feature information inherent in the label. By transforming numerically singular and discontinuous one-hot encodings into high-dimensional latent codes, the FIM helps the model better find and understand the relationship between control information and image features, which also helps refine and improve the control of feature information over attribute editing.

[0137] The function of the feature information mapping network is expressed by the following formula:

[0138] k t =M(c t (9)

[0139] In formula (9), M is the feature information mapping network; c t k represents discrete feature attribute labels. t It is a high-dimensional latent encoding that implicitly contains feature information;

[0140] like Figure 6As shown, the feature information mapping network is designed as a feedforward neural network. Its input is an n-dimensional one-hot code. After entering the feature information mapping network, the one-hot code needs to pass through nine dense layers, and each dense layer is followed by an activation layer. When the one-hot code passes through the dense layers, the neurons there will perform an affine transformation on the one-hot code. After passing through the activation layers, the one-hot code will undergo a nonlinear transformation. After passing through nine dense layers and nine activation layers, the feature information mapping network will output a high-dimensional latent code. The affine transformation operation of the one-hot code in the dense layers can be expressed by the following formula:

[0141]

[0142] In formula (10), z is the final output value of the affine transformation; the weight between any neuron in the k-th layer and any neuron in the (k+1)-th layer is represented by ω. k Indicates that m is the maximum number of layers in the feature information mapping network; c is an additional bias that offsets the affine transformation offset during the calculation process.

[0143] To map n-dimensional discrete feature attribute labels to 512 dimensions through affine transformation, the input layer and the subsequent seven dense layers of the feature information mapping network consist of 512 neurons, and the output layer of the feature information mapping network consists of N neurons, which will keep the final output information dimension of the network N.

[0144] like Figure 7 As shown, in deep learning research and applications, Sigmoid, Tanh, ReLU, and Leaky ReLU functions are widely used as activation functions. However, compared with Sigmoid, Tanh, and ReLU functions, the Leaky ReLU activation function exhibits unique advantages. The Leaky ReLU function is a typical piecewise function. When the input value is greater than 0, the input is directly mapped to the output. When the input value is less than 0, the output is not completely represented as 0, but multiplied by a very small slope α (usually -0.01) and mapped to a very small negative number. Compared to Sigmoid and Tanh functions, the Leaky ReLU function has a derivative function, which makes backpropagation possible. Its higher computational efficiency also allows the network to converge quickly. Compared to the ReLU activation function, the Leaky ReLU function effectively solves the neuron death problem caused by the ReLU function when the input value is less than 0. Furthermore, since the derivative of the Leaky ReLU function is always 1 in the region where the input value is greater than 0, it means that the gradient will always remain at 1. This characteristic makes the Leaky ReLU function more efficient when performing gradient descent and backpropagation, effectively avoiding the problems of gradient explosion and gradient vanishing.

[0145] Therefore, in this embodiment, the activation layer adopts a leaky linear rectifier unit Leaky ReLU as an activation function, and the mathematical expression of the LeakyReLU activation function is:

[0146]

[0147] Further, in this embodiment, the image generation network G gen The context loss is calculated by using a VGG19 network pre-trained in the image classification field as a loss network, as shown in Figure 8 The generated image and the original image are input into the loss network VGG19, and the features of the image are extracted in the last activation layer

[0148] ReLU5_4 before the dense layer, and the context loss is calculated. The feature information extracted in this layer can best reflect the potential deep features of the image.

[0149] In the generative adversarial network, the loss function plays a crucial role, which directly affects the training process of the generator and the discriminator.

[0150] Specifically, in the training process of the FFEGAN model, in order to ensure that the generator G can maximize the preservation of other information of the original image x while converting the original features to the target features when generating a new image y with target features, the reorganization loss is applied to the generator G, and the definition of the reorganization loss is represented as:

[0151] L re =||x-G(G(x,c t ),c o )||1 (12)

[0152] In formula (12), x is the original image input into the FFEGAN model; y is the generated image output by the FFEGAN model; c o is the label of the original domain corresponding to the original image;

[0153] For the two different discriminant branches of the discriminator D, the adversarial discriminant branch D adv and the classification discriminant branch D cls , in the process of model training, due to the different purposes, the goal of the generator G is to continuously reduce and reduce the adversarial loss L adv , while the goal of the discriminator D is to continuously increase and increase the adversarial loss; the corresponding loss functions are completely different;

[0154] The loss function of the adversarial discriminant branch D adv is represented as:

[0155]

[0156] In formula (13), L adv is the adversarial loss; x is the original image input into the FFEGAN model, D adv (x) is the probability distribution of the image output by the adversarial discriminant branch of the discriminator; c t is the target domain label of the conversion;

[0157] The loss function of the classification discriminant branch D cls is represented as:

[0158] ① When training the discriminator D, the function of the classification discriminant branch D cls is to evaluate the attribute category of the real image with the classification loss, so the loss function is represented as:

[0159]

[0160] In formula (14), D cls (c o |x) is the confidence probability distribution of the classification discriminant branch on whether the input image is the output feature attribute; x is the original image input into the FFEGAN model, c o is the original domain label of the feature attribute of the original image;

[0161] ② When training the generator G, the function of the classification discriminant branch D cls is to help the generator G generate an image that meets the target feature attribute label c t , so the loss function is represented as:

[0162]

[0163] In formula (15), D cls (c t |G(x, c t )) is the confidence probability distribution of the classification discriminant branch on whether the input image belongs to the target feature attribute; x is the original image input into the FFEGAN model, c t represents the target domain of the conversion task, i.e., the labeled label of the target feature attribute.

[0164] Further, the context loss, for the original input image x and the generated image y, regards them as the set of all features contained in the respective images, denoted as: X1={x i}, Y1={y j}; In calculation, X1 and Y1 need to be the same size as a set, i.e., |Y1|=|X1|, when the sizes of the two sets are different, additional features need to be added from the larger set;

[0165] First, the cosine distance of the features needs to be normalized, and the principle of the operation is represented as:

[0166]

[0167] In formula (16), d ij is the cosine distance from x i to y j ; l is an arbitrary positive integer not equal to j; ∈ is the offset of the denominator, ∈ = 1e-5;

[0168] Then, the normalized distance needs to be converted into the similarity between the features, and this conversion process is completed through the exponential operation; at the same time, in order to make the similarity between different features have a unified scale when measuring, the similarity is defined as the quotient of the similarity between the original feature and the current feature and the total similarity between the original feature and all other features, and the principle of this process is represented by formula:

[0169]

[0170] In formula (17), h is a bandwidth parameter, h > 0; CX ij is the similarity between features i and j;

[0171] When formula (17) is applied to the full image, the definition of the similarity between two sample images is represented by formula:

[0172]

[0173] Suppose the FFEGAN model uses VGG19 as the loss network, extracts the feature set of the input image through the VGG19 loss network, and then applies the feature set to the context loss function, and the definition of the context loss function is represented by formula:

[0174] L CX (x, y, k) = -log(CX(Φ k (x), Φ k (y))) (19)

[0175] In formula (19), Φ k (x) and Φ k (y) are the feature sets extracted by the kth layer of the loss network Φ for images x and y, respectively;

[0176] Further applying formula (19) to the FFEGAN model, the context loss function corresponding to the FFEGAN model is represented as:

[0177] L CX(G(x), x, k) = -log(Cx(Φ k (G(x)), Φ k (x))) (20)

[0178] According to the context loss function provided by formula (20), it can be understood that the generator G needs to minimize the context loss L CX in the process of generating images, which will help to improve the similarity between the generated images and the original images at the feature set level; therefore, the total loss function of the generator G in the FFEGAN model can be obtained as:

[0179]

[0180] In formula (21), L G represents the total loss of the generator G; L adv is the adversarial loss of the adversarial discrimination branch of the discriminator; is the classification loss of the classification discrimination branch of the discriminator; L re is the reorganization loss for maintaining the original information of the image; L cx is the context loss; λ cls , λ re and λ cx are weight coefficients of the classification loss, the reorganization loss and the context loss, which will affect the balance between maintaining the original information and feature conversion of the generated image;

[0181] The total loss function of the discriminator D of the FFEGAN model is:

[0182]

[0183] In formula (22), L D is the total loss of the discriminator; L adv is the adversarial loss of the adversarial discrimination branch of the discriminator; is the classification loss of the classification discrimination branch of the discriminator; λ cls is the weight coefficient of the classification loss, which will affect the balance of the discriminator before the adversarial generator and the correct classification label.

[0184] In order to evaluate whether the face feature conversion method of the present application is feasible in practice, and also to evaluate the performance and effect of the proposed FFEGAN model, two internationally recognized public face databases are selected for experiments in this embodiment, namely the CelebA

[45] (Large-scale CelebFaces Attributes (CelebA) Dataset) dataset and the CelebA-HQ

[46] (CelebFaces Attributes (CelebA-HQ) Database High Quality) dataset.

[0185] Part of the images of the CelebA dataset and the CelebA-HQ dataset are shown in Figure 9 The CelebA-HQ is a high-resolution version of the CelebA created by T. Karras et al., which contains 30,000 high-resolution images. Compared with the low-resolution CelebA dataset, the CelebA-HQ provides more abundant and realistic facial details. In theory, the image quality of the CelebA-HQ dataset is relatively higher, and it is more suitable for more complex tasks such as image generation, editing and attribute conversion.

[0186] The following is the specific evaluation process:

[0187] I. Establish evaluation indicators

[0188] In this embodiment, the Fréchet Inception distance (Fréchet Inception Distance, FID) score and the Kernel Inception distance (Kernel Inception Distance, KID) score are used. The calculation method of the FID distance score is:

[0189]

[0190] In formula (23), μ X and Σ X represent the mean and covariance of the training data, respectively; μ Y and Σ Y represent the mean and covariance of the generated image (to be tested data), respectively; Tr represents the trace of the matrix (i.e. the sum of the eigenvalues of the matrix); and ||·|| represents the two-norm of the vector.

[0191] The specific calculation steps are first to extract the feature vector from the network, then to calculate the mean vector and covariance matrix of the two distributions to be compared, and finally to calculate the Fréchet distance between the two matrix distributions to obtain the FID score.

[0192] The calculation method of KID distance score is:

[0193] KID = MMD(f real ,f fake ) 2 (24)

[0194] In formula (24), MMD represents the maximum mean difference, f real and f fake are the mean values of the features extracted from the real images and the generated images respectively. The mean difference between the two sets of features is calculated, the maximum mean difference is defined as the maximum Euclidean distance between the two means, and finally the value of the maximum mean difference is squared to obtain the final KID.

[0195] II. Experimental environment

[0196] The operating system of the experimental machine is Windows 1122631.329623H2 version, and the programming language used is Python, version python 3.9. For matrix operations with large dimensions, mathematical function libraries in the reference package numpy supported by Python are used for calculation. The implementation of the method and model proposed in this paper, as well as the implementation of the comparative method and model, are all based on the open source framework Pytorch, and the version of Pytorch is Pytorch stable (1.11.0). All experiments are carried out in the Pycharm integrated development platform environment, and the version number of Pycharm is 2021.3.3. For the specific hardware configuration of the experimental machine, the CPU used in the experiment is Intel(R) Core(TM) i9-10900K CPU @ 3.70 GHz, the main memory is 64 GB, the GPU is NVIDIA Quadro RTX4000, and the video memory is 16 GB.

[0197] III. Experimental parameters

[0198] Due to the limitation of hardware devices such as computing power and GPU memory, the resolution of the image in the embodiment is set to 256*256, which can preserve the information of the image as much as possible under the premise of ensuring the normal operation of the experiment. In the experiment, all batch sizes are set to 16, and the number of iterations is set to 200,000 times. The selected optimizer is the Adam optimizer, which combines the advantages of AdaGrad and RMSProp two optimization algorithms, and can better update the parameters. In the algorithm parameter design, the key parameters of the Adam optimizer are set to 0.5 and 0.999, respectively. The two parameters are the decay rate for calculating the exponential moving average of the gradient and the decay rate for calculating the exponential moving average of the gradient square. At the beginning of the experiment, the initial learning rate of the FFEGAN model is set to 1.0*10 -4 In the first half of the experiment, that is, in the first ten thousand iterations of the FFEGAN model, in order to enable the model to learn better, the learning rate of the model is set to the initial learning rate; in all subsequent iterations, the learning rate will continue to decrease, and every time the number of iterations increases by 1000, the learning rate will decrease by a multiple of the current learning rate and the current iteration number. The values of the classification loss weight λ cls and the cycle consistency loss weight λ re are set to 1 and 10, respectively.

[0199] Four, experimental steps

[0200] The FFEGAN model provided by the method of the application is composed of a generator and a discriminator and two branches thereof, and a cycle is completed in the model by sharing the generator to perform cycle consistency verification. In the first stage of the cycle, the generator will generate an image that meets the target feature attribute label using the original image and the target domain label as input. In the second stage of the cycle, the generated image and the source domain feature attribute label are used as input, and the generator generates a reorganized image that meets the original feature attribute.

[0201] The specific details of the cycle are as follows:

[0202] Step 1: The parameters of the generator G remain unchanged, and only the discriminator D is updated. The original image and the target domain label are input into the generator G to generate an image sample. The real sample is selected from the data set to judge its true or false and category by the discriminator D. The parameters of the discriminator D are adjusted and updated.

[0203] Step 2: The parameters of the discriminator D remain unchanged, and only the generator G is updated. The original image and the target domain label are input into the generator G to generate an image sample, and the sample is judged by the discriminator D to determine its true or false and category. The parameters of the generator G are adjusted and updated.

[0204] Third step: the generated fake sample image and the source domain feature attribute label are input into the generator, the difference between the reorganized image and the original image in the first step is calculated, and the parameters of the generator G are adjusted and updated.

[0205] V. Experimental design

[0206] The experiment in this embodiment is divided into two parts in total. The first part is an ablation experiment on the FFEGAN model, which verifies the influence of the proposed method on the overall performance by continuously reducing the model content of the FFEGAN. The second part is a horizontal comparison experiment with advanced methods in the current field.

[0207] It should be noted that in order to fully verify the performance and effect of the method and model proposed in this paper, this embodiment will conduct comparative experiments on other mainstream and most representative image-to-image translation frameworks. The control group selects three methods, CycleGAN, ComboGAN and StarGAN, as the comparison. The CycleGAN, ComboGAN and StarGAN models in the control group all use the default configuration provided by the authors for training.

[0208] Ablation experiment

[0209] Since the original GAN must be trained using paired data, the ablation experiment will use CycleGAN as the baseline. The improvements of the FFEGAN model proposed in this embodiment to the generative adversarial network structure are as follows: 1) a feature information mapping network is added to the generator to help the generator train; 2) the discrimination function of the discriminator is designed into two branches, which respectively perform the true and false discrimination task and the category discrimination task; 3) the reorganization loss is applied to generate the image in reverse to help training.

[0210] Therefore, the ablation experiment will focus on the above three modification points. In the experiment, several categories of attribute features that are relatively easy to observe are selected, such as hair color, gender, wearing glasses and baldness, etc. Among them, the specific hair color has four sub-attributes of black, gold, brown and gray, and different hair colors can be converted to each other. As for the binary attributes other than hair color, the target domain of the conversion is the opposite attribute, such as male and female, wearing glasses and not wearing glasses, bald and not bald, etc. The FID score and KID score are used to quantitatively evaluate the results of the ablation experiment, and the lower the FID score and KID score, the better the image generation effect. The results of the ablation experiment are shown in Table 1 below:

[0211] Table 1, ablation experiment results

[0212]

[0213] In the table, M represents a feature mapping network module, and D represents a double-branch discriminator module.

[0214] The quantitative analysis scores shown in the table are the arithmetic means of the scores obtained after conversion of each attribute. Since CycleGAN has already embodied the idea of cycle verification in the model through the application of cycle consistency loss, the difference between the reconstruction loss and it is more reflected in the number of generators, so the separate ablation experiment for the third improvement has no significant meaning.

[0215] From the results of the ablation experiment, it can be seen that after adding the feature information mapping network, the conversion effect of the model on the image and the feature has been significantly improved. The separate addition of the classification discriminant branch has limited improvement on the model, because the loss of this branch has not strong control on the generator during the training process, even if the feature attribute label has completed the mapping of the deep feature information, but such feature information is difficult to fully and effectively control the image generation of the generator. It needs to be particularly noted that the FFEGAN model is significantly different from CycleGAN in the implementation method of the cycle idea, and CycleGAN needs an additional generator to complete the cycle of the image, while the method only needs one generator.

[0216] Comparative experiment

[0217] The attribute feature conversion performed in the comparative experiment is consistent with the feature attribute selected in the ablation experiment; the results of the comparative experiment are shown in Table 2 as follows:

[0218] Table 2, results of the comparative experiment

[0219]

[0220] From the results of the comparative experiment, it can be seen that the FID score and the KID score of the method of the present application are lower than those of the other three methods of the control group, which also means that the method proposed in the present application has higher image quality than the existing methods, and can better maintain the original identity, posture and other information of the image when performing the face feature editing task.

[0221] In this embodiment, the comparative experiment is also qualitatively analyzed by comparing the conversion effect, and the conversion effect is as shown in Figure 10 .

[0222] From Figure 10As can be seen, CycleGAN has obvious distortion for complex feature attribute transformation; ComboGAN performs well on the hair color project but poorly on the gender project, because the control force of the ComboGAN model is not refined enough. StarGAN is a model that performs well in the control group, and only has redundant hair strands and tips in some projects, which is caused by the insufficient cycle consistency loss used by StarGAN. The FFEGAN model is the best-performing model in the experiment, which guarantees the conversion effect while retaining sufficient original identity information. In addition, the method of the present application is superior to other methods in preserving original skin details, edges and textures, which means that the method of the present application more effectively reduces the loss of original information in the input picture.

[0223] The results of the qualitative analysis show that the pictures generated by the FFEGAN model provided by the method of the present application are more vivid and realistic than those generated by other existing methods, which also confirms the conclusion in the quantitative analysis that the images generated by the method of the present application are of higher quality than those generated by other existing methods.

[0224] The above is only a preferred embodiment of the present application, and is not intended to limit the present application in other forms. Any person skilled in the art can modify or change the above disclosed technical content to obtain equivalent embodiments. However, any simple modification, equivalent change and modification made on the basis of the technical essence of the present application without departing from the technical solution of the present application shall still fall within the protection scope of the present application.

Claims

1. A face feature conversion method based on feature information mapping and attention mechanism, characterized in that, The method comprises the following steps: Step 1, constructing a face feature editing model FFEGAN; The face feature editing model FFEGAN comprises a generator, a discriminator and a loss network; The generator is composed of a feature information mapping network, an image generation network and an attention mask generation network; The generator G is composed of an image generation network G generating a target image through an encoder-decoder structure gen , an attention mask generation network G generating an attention mask using a multi-head self-attention mechanism mask , and a feature information mapping network M converting one-hot encoding into high-dimensional latent information; Assuming that the original face image is x, the feature label given to the original domain is c 0 , the feature label given to the target domain is c t , the generator takes the original face image x and the target domain label c t as input, generates a target image y, and makes its feature label match c t , while retaining the original information and attributes of the face; The function of the feature information mapping network is expressed by a formula as follows: k t = M(c t ) (9) In formula (9), M is a feature information mapping network; c t is a discrete feature attribute label, k t is a high-dimensional latent code that implicitly contains feature information; The input of the feature information mapping network is an n-dimensional one-hot encoding, which needs to pass through nine dense layers and an activation layer after each dense layer; the neurons in the dense layer will perform affine transformation on the one-hot encoding, and the one-hot encoding will be nonlinearly transformed after passing through the activation layer; after passing through the nine dense layers and the nine activation layers, the feature information mapping network will output a high-dimensional latent encoding; the operation of affine transformation of the one-hot encoding in the dense layer is expressed by a formula as follows: In formula (10), z is the final output value of the affine transformation; The weight between any neuron of the kth layer and any neuron of the k+1th layer is denoted as ω k m is the maximum number of layers of the feature information mapping network; and c is an additional bias for offsetting the affine transformation bias in the calculation process. The input layer of the feature information mapping network and the seven subsequent dense layers are composed of 512 neurons, and the output layer of the feature information mapping network is composed of N neurons; The activation layer adopts a Leaky ReLU as an activation function, and the mathematical expression of the Leaky ReLU activation function is as follows: Step 2, taking the label and the original image of the target domain to be converted as the input of the face feature editing model FFEGAN, and outputting a generated image that converts the feature from the original domain to the target domain; First, the label of the target domain is converted into deep feature information data by the feature information mapping network in the generator, and then added to the image generation network in the process of converting the original image into a generated image to realize the control of the conversion; finally, the generated image enters the discriminator for evaluation, evaluates the authenticity of the generated image, and classifies the generated image by domain. 2.The face feature conversion method based on feature information mapping and attention mechanism according to claim 1, characterized in that, In step 1, the target domain label c t In the generator, the target domain label c t will be mapped to high-dimensional latent encoding k t with implicit feature information by feature information mapping network M gen , which will be injected multiple times into image generation network G gen , the mathematical logic expression of which is: G gen (x, k t ) = y (1) The mathematical logic expression of the generator G is as follows: G(x, c t ) = G gen (x, M(c t )) = y (2) In formulas (1) and (2), x is the original image input into the generator, y is the new image generated by the generator; M is the feature information mapping network; The generation work of the generator runs to the decoder part, when upsampling the information of different network levels, the high-dimensional potential code k of the feature information is implied t will be injected into the image generation network G multiple times gen different levels to strengthen the control of the label at different levels of the decoder. 3.The face feature conversion method based on feature information mapping and attention mechanism according to claim 2, characterized in that, The image generation network G gen The encoder part is composed of three layers of down-sampling layers and the same number of bottleneck layers, the down-sampling layers are used for convolution operation on the input data to extract the feature information contained in the image, and normalization processing is performed after each convolution operation using an adaptive instance normalization method; The image generation network G gen The decoder part is symmetrical in structure with the encoder part, and is composed of three bottleneck layers and the same number of up-sampling layers for performing transpose convolution operations, and the bottleneck layers are connected by jump connection with residual blocks, which work in the image generation network G gen The working principle of the residual block in the image generation network G in formula is: y = sigma (F (x, W) + x) (3) In formula (3), sigma (·) represents an activation function, F (·) represents a residual function, and W represents all weights in the residual block; For a deeper layer L, the relationship with the previous layer l is as follows: In formula (4), x l is a feature vector obtained from a previous layer, and is also the original input data of the Lth layer, and i is a layer index requiring addition operation of the original feature and the nonlinear transformation result. After each convolution operation, an adaptive instance normalization method is used for normalization processing, and the calculation process of the adaptive instance normalization is as follows: AdaIN(X, k t ) = (1 + f(k t )1)(IN(X)) + f(k t )2 (5) In equation (5), X is the feature distribution of the image; let the dense layer be represented by a function f, then f(k t )1and f(k t )2are two parameters generated by the slicing operation, and IN(X) means instance normalization on the feature distribution X of the input image, that is, normalization on each channel of each sample separately. In equation (5), X is the feature distribution of the image; let the dense layer be represented by a function f, then f(k t )1and f(k t )2are two parameters generated by the slicing operation, and IN(X) means instance normalization on the feature distribution X of the input image, that is, normalization on each channel of each sample separately.

4. The face feature conversion method based on feature information mapping and attention mechanism according to claim 3, characterized in that, The attention mask generation network G mask The architecture of the image generation network G gen The architecture of the attention mask generation network G mask After the last layer of the bottleneck layer, a calculation operation for multi-head self-attention mechanism calculation is added, the number of heads of the multi-head self-attention mechanism is set to 3, and the attention mask generation network G mask The up-sampling layer part of the attention mask generation network G mask The last layer of the bottleneck layer, the number of convolution kernels is set to 1, and the attention mask generation network G mask The original image x is taken as input, and finally an attention mask AttM with one channel and the same resolution as the original face image x is output, the mask AttM contains information of relevant feature positions of the original image and weight information of different regions in the conversion work. The attention mask generation network G mask Based on the correct generation of the attention mask AttM, the image generation network G gen A successful conversion can be completed by only converting the relevant regions marked by the mask, while other regions remain unchanged, and the generator G outputs the image y * The process is expressed by the formula: y * = (1 - AttM) - x + AttM - y' (6) In formula (6), x is the original image input into the generator, y' is the relevant region marked by the mask, and AttM is the attention mask; The working principle of the generator G as a whole is expressed as follows: G(x, c t ) = G mask (x) · G gen (x, M(c t )) + (1 - G mask (x)) · x (7). 5.The face feature conversion method based on feature information mapping and attention mechanism according to claim 1, characterized in that, In step 1, the discriminator includes two discriminant branches, which are respectively an adversarial discriminant branch D adv and a classification discriminant branch D cls ; the adversarial discriminant branch D adv adopts a full convolution form, and after the input image is subjected to a convolution operation, the output result will continue to be projected into an N-order projection matrix containing discriminant information through a convolution operation, each value in the N-order projection matrix containing the evaluation result of the discriminator on a certain block region of the image; the classification discriminant branch D cls is used to output a probability distribution describing the features of the input image, which is used to assist the face feature editing model FFEGAN in learning various feature attributes and classifying them into correct feature labels. The overall structure of the discriminator is expressed by a formula as follows: D = {D adv ,D cls} (8). 6.The face feature conversion method based on feature information mapping and attention mechanism according to claim 1, characterized in that, The image generation network G gen The context loss is calculated by using a loss network pre-trained in the field of image classification, the generated image and the original image are input into the loss network, the feature distribution of the image is extracted in the last activation layer before the dense layer, and the context loss is calculated, and the feature information extracted in the layer can best reflect the potential deep features of the image.

7. The face feature conversion method based on feature information mapping and attention mechanism according to claim 1, characterized in that, In the training process of the FFEGAN model, in order to ensure that the generator G can maximize the preservation of other information of the original image x while converting the original feature to the target feature when generating a new image y with the target feature, a reorganization loss is applied to the generator, and the definition of the reorganization loss is expressed as follows: L re = ||x - G(G(x, c t ), c o )||1 (12) In formula (12), x is the original image input into the FFEGAN model; y is the generated image output by the FFEGAN model; c o is the label of the original domain corresponding to the original image. For two different discriminant branches of the discriminator, the adversarial discriminant branch D adv and the classification discriminant branch D cls In the process of model training, due to the difference of their purposes, the goal of the generator G is to continuously reduce and decrease the adversarial loss L adv While the goal of the discriminator D is to continuously increase and increase the adversarial loss; The corresponding loss functions are also completely different; The adversarial discriminative branch D adv The loss function of the adversarial discriminative branch D is represented as: In formula (13), L adv is the adversarial loss; x is the original image input into the FFEGAN model; D adv (x) is the probability distribution of the image output by the adversarial discrimination branch of the discriminator; c t is the target domain label of the conversion; The classification discriminant branch D cls The loss function of the classification discriminant branch D is represented as: ① When training the discriminator D, the classification discriminative branch D cls The function of the classification discriminative branch D is to evaluate the attribute category of the real image with a classification loss, and therefore the loss function is represented as: In formula (14), D cls (c o |x) is a confidence probability distribution of the classification discriminant branch on whether the input image is the output feature attribute; x is the original image input into the FFEGAN model, c o is the original domain label of the original image feature attribute; ② When training the generator G, the classification discriminant branch D cls helps the generator G to generate images that meet the target domain label c t , so the loss function is represented as: In formula (15), D cls (c t |G(x,c t )) is the confidence probability distribution of whether the input image belongs to the target feature attribute by the classification discriminant branch; x is the original image input into the FFEGAN model, and c t is the target domain label of the conversion task, that is, the labeled label of the target feature attribute. 8.The method of claim 6, wherein, The context loss, for the original input image x and the generated image y, will treat them as a set of all the features contained in the respective images, denoted as: X1 = {x i}, Y1 = {y j}; In the calculation, the size of X1 and Y1 as a set needs to be the same, i.e., |Y1| = |X1|, when the size of the two sets is different, additional features need to be added from the larger set; First, the cosine distance of the features needs to be normalized, and the principle of the operation is represented as: In formula (16), d ij is the cosine distance from x i to y j ; l is an arbitrary positive integer not equal to j; ∈ is the offset of the denominator, ∈ = 1e-5; Then, the normalized distance needs to be converted into the similarity between the features, and the conversion process is completed through the exponential operation; at the same time, in order to make the similarity between different features have a unified scale in the measurement, the similarity is defined as the quotient of the similarity between the original feature and the current feature and the total similarity between the original feature and all other features, and the principle of the process is represented by the formula: In Equation (17), h is a bandwidth parameter, h > 0; CX ij is the similarity between feature i and feature j; When formula (17) is applied to the full image, the definition of the similarity between two sample images is represented by the formula: Assuming that the FFEGAN model uses VGG19 as the loss network, the feature set of the input image is extracted through the VGG19 loss network, and then the feature set is applied to the context loss function, and the definition of the context loss function is represented by the formula: L CX (x,y,k) = -log(Cx(Φ k (x),Φ k (y))) (19) In Equation (19), Φ k (x) and Φ k (y) are the feature sets extracted by the k-th layer of the loss network Φ for images x and y, respectively. Further applying formula (19) to the FFEGAN model, the context loss function corresponding to the FFEGAN model is represented as: L CX (G(x), x, k) = -log(Cx(Φ k (G(x)), Φ k (x))) (20) From the context loss function provided by formula (20), it can be understood that the generator G needs to minimize the context loss L CX in the process of generating images, which will help to improve the similarity between the generated images and the original images at the feature set level; therefore, the total loss function of the generator G in the FFEGAN model is obtained as follows: In formula (21), L G represents the total loss of the generator G; L adv is the adversarial loss of the adversarial discrimination branch of the discriminator; is the classification loss of the classification discrimination branch of the discriminator; L re is the reorganization loss for maintaining the original information of the image; L cx is the context loss; λ cls , λ re and λ cx are weight coefficients of the classification loss, the reorganization loss and the context loss, and the three weight coefficients will affect the balance between maintaining the original information and feature conversion of the generated image. The total loss function of the FFEGAN model discriminator D is: In formula (22), L D is the total loss of the discriminator; L adv is the adversarial loss of the adversarial discrimination branch of the discriminator; is the classification loss of the classification discrimination branch of the discriminator; λ cls is the weight coefficient of the classification loss, which affects the balance between the adversarial generator and the correct classification label before the discriminator.

Images