Face feature conversion method based on feature information mapping and attention mechanism

Abstract

The invention discloses a face feature conversion method based on feature information mapping and an attention mechanism. The method comprises the following steps: 1) constructing a face feature editing model; the face feature editing model comprises a generator, a discriminator and a loss network; 2) taking a to-be-converted target domain label and an original image as input of a model, and outputting a converted generated image; firstly, a target domain label is converted into data of depth feature information by a feature information mapping network in a generator, then the data is added into an image generation network to realize conversion in the process of converting an original image into a generated image, and finally the generated image enters a discriminator to be evaluated, and the true degree of the generated image is evaluated. According to the method, the structure of the generator and the structure of the discriminator are modified, and a loop structure in the model is constructed, so that correct generation of the image is facilitated; and by adding an attention mask generation network and context loss, the performance of retaining the original information of the image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial feature conversion, and in particular relates to a facial feature conversion method based on feature information mapping and attention mechanism. Background Art

[0002] In recent years, the technology in the fields of computer graphics and computer vision is developing rapidly, which has also led to the progress of related fields of facial feature modification, editing and conversion. With the birth of generative adversarial networks, content-generating artificial intelligence has become a hot topic of discussion and an important research direction. The editing and modification task of facial features is a multi-domain to multi-domain conversion task. In real life, it can optimize distorted images and low-quality images, better highlight facial features, and help facial images to more effectively cope with detection and analysis; or polish completed images, reduce the difficulty and workload of film and television practitioners, and help art creators and enthusiasts better realize the display of inspiration and the creation of works of art. However, the current related technologies still have some problems in this field: (1) The existing feature conversion methods based on multi-domain image conversion still have weak control over discrete feature labels. The effective control of labels on features and attributes needs to be further strengthened, and the control accuracy of labels on feature generation also needs to be further improved. (2) The existing methods are prone to losing detailed features in the process of feature conversion and attribute editing. The commonly used cycle consistency loss reduces the effect of converting image features and attributes to a certain extent in some applications. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and provide a facial feature conversion method based on feature information mapping and attention mechanism. By modifying the structure of the generator and the structure of the discriminator, a loop structure inside the model is constructed, which not only helps to correctly generate the image; but also improves the performance of retaining the original information of the image by adding an attention mask generation network and context loss.

[0004] A facial feature conversion method based on feature information mapping and attention mechanism includes the following steps:

[0005] Step 1: Build the facial feature editing model FFEGAN;

[0006] The facial feature editing model FFEGAN includes a generator, a discriminator and a loss network;

[0007] The generator consists of three parts: feature information mapping network, image generation network and attention mask generation network;

[0008] Step 2: Take the label of the target domain to be converted and the original image as the input of the face feature editing model FFEGAN, and output a generated image that converts the features from the original domain to the target domain;

[0009] First, the target domain label is converted into deep feature information data by the feature information mapping network in the generator, and then added to the image generation network to control the conversion during the conversion of the original image to the generated image. Finally, the generated image enters the discriminator for evaluation to evaluate the authenticity of the generated image and classify the domain of the generated image.

[0010] Furthermore, in step 1, the generator G is an image generation network G that generates a target image through an encoder-decoder structure. gen , an attention mask generation network G that uses a multi-head self-attention mechanism to generate attention masks mask It consists of three parts: a feature information mapping network M that converts one-hot encoding into high-dimensional potential information;

[0011] Assume that the original face image is x and the feature label assigned to the original domain is c 0 , the target domain is assigned a feature label of c t , the generator is based on the original face image x and the target domain label c t As input, generate a target image y and make its feature label the same as c t Matching while preserving the original facial information and attributes;

[0012] The target domain label c t In the generator, it will be generated randomly. During the model training process, the input target domain label c t It will be mapped by the feature information mapping network M into a vector k containing feature attribute information t , this vector will be injected into the image generation network G multiple times gen To strengthen the control of the label on the generation, the image generation network G gen The mathematical logic expression is:

[0013] G gen (x,k t )=y (1)

[0014] The mathematical logic expression of the generator G is:

[0015] G(x,c t )=G gen (x,M(c t ))=y (2)

[0016] In formulas (1) and (2), x is the original image input to the generator, y is the new image generated by the generator, and M is the feature information mapping network.

[0017] When the generator's generation work runs to the decoder part, when upsampling the information at different network levels, the vector k containing the feature attribute information t Will be injected into the image generation network G multiple times gen Different levels are used to strengthen the control of tags at different levels of the decoder.

[0018] Furthermore, the image generation network G gen The encoder part of is composed of three downsampling layers and the same number of bottleneck layers. The downsampling layers are used to perform convolution operations on the input data to extract the feature information contained in the image. After each convolution operation, the adaptive instance normalization method is used for normalization.

[0019] The image generation network G gen The decoder part of the proposed method is symmetrical with the encoder part in structure. The decoder part consists of three bottleneck layers and the same number of upsampling layers. The upsampling layers are used to perform transposed convolution operations. The bottleneck layers use residual blocks for skip connections. The residual blocks are used in the image generation network G. gen The working principle is expressed in the formula:

[0020] y=σ(F(x,W)+x)(3)

[0021] In formula (3), σ(·) represents the activation function, F(·) represents the residual function, and W represents all weights in the residual block;

[0022] For a deeper layer L, its relationship with the previous layer is:

[0023]

[0024] After each convolution operation, the adaptive instance normalization method is used for normalization. The calculation process of the adaptive instance normalization is:

[0025] AdaIN(X, k t )=(1+f(k t )1)(IN(X))+f(k t )2 (5)

[0026] In formula (5), X is the feature distribution of the image; let the dense layer be represented by function f, then f(k t )1 and f(k t )2 are the two parameters generated by the slice operation.

[0027] Furthermore, the attention mask generation network G mask Architecture and Image Generation Network G gen The architecture is similar, the attention mask generation network G mask After the last layer of the bottleneck layer, a calculation operation for multi-head self-attention mechanism calculation is added. The number of heads of the multi-head self-attention mechanism is set to 3. The attention mask generation network G mask In the upsampling layer of the attention mask generation network G, the instance normalization method is used to normalize the features. mask In the last bottleneck layer, the number of convolution kernels is set to 1, and the attention mask generation network G mask The original image x is taken as input, and finally an attention mask AttM with a channel number of 1 and a resolution the same as that of the original face image x is output. The mask AttM contains information about the relevant feature positions of the original image and weight information of different regions in the conversion work;

[0028] In the attention mask generation network G mask Based on the correct generation of the attention mask AttM, the image generation network G gen A successful conversion can be completed by converting the relevant areas marked by the mask while keeping the other areas as they are. The generator G outputs the image y * The process is expressed by the formula:

[0029] y * =(1-AttM)·x+AttM·y′ (6)

[0030] In formula (6), x is the original image input to the generator, y′ is the relevant area marked by the mask, and AttM is the attention mask;

[0031] The overall working principle of the generator G is expressed as:

[0032] G(x,c t )=G mask (x)·G gen (x,F(c t ))+(1-G mask (x))·x (7).

[0033] Furthermore, in step 1, the discriminator includes two discriminant branches, and the two discriminant branches are adversarial discriminant branches D adv And classification and discrimination branch D cls ; The adversarial discriminant branch D advThe full convolution form is adopted. After the input image is convolved, its output result will continue to be projected into an N-order projection matrix containing discriminant information through the convolution operation. Each value in the N-order projection matrix corresponds to the evaluation result of the discriminator on a certain area of ​​the image. The classification discrimination branch D cls Used to output a probability distribution describing the features of the input image, wherein the probability distribution is used to assist the face feature editing model FFEGAN to learn various feature attributes and classify them into correct feature labels;

[0034] The overall structure of the discriminator is expressed as follows:

[0035] D={D adv , D cls} (8).

[0036] Furthermore, the function of the feature information mapping network is expressed by the formula:

[0037] k t =M(c t ) (9)

[0038] In formula (9), M is the feature information mapping network; c t is the target domain label, k t It is a high-dimensional latent code that contains implicit feature information;

[0039] The input of the feature information mapping network is an n-dimensional one-hot encoding. After entering the feature information mapping network, the one-hot encoding needs to pass through nine dense layers, and each dense layer is followed by an activation layer. When the one-hot encoding passes through the dense layer, the neurons therein will perform an affine transformation on the one-hot encoding, and the one-hot encoding will undergo a nonlinear transformation after passing through the activation layer. After passing through nine dense layers and nine activation layers, the feature information mapping network will output a high-dimensional potential code. The operation of the one-hot encoding performing an affine transformation in the dense layer is expressed by the formula:

[0040]

[0041] In formula (10), z is the final output value of the affine transformation; the weight between any neuron in the kth layer and any neuron in the k+1th layer is ω i Indicates; m is the maximum number of layers of the feature information mapping network; c is an additional bias to offset the affine transformation offset during the calculation process;

[0042] The input layer and the subsequent seven dense layers of the feature information mapping network are composed of 512 neurons, and the output layer of the feature information mapping network is composed of N neurons;

[0043] The activation layer uses the leaky linear rectifier unit Leaky ReLU as the activation function. The mathematical expression of the Leaky ReLU activation function is:

[0044]

[0045] Furthermore, the image generation network G gen The context loss is calculated by using a loss network pre-trained in the field of image classification. The generated image and the original image are input into the loss network. The feature distribution of the image is extracted in the last activation layer before the dense layer and the context loss is calculated. The feature information extracted in this layer can best reflect the potential deep features of the image.

[0046] Furthermore, during the training process of the FFEGAN model, in order to ensure that the generator G can retain other information of the original image x to the greatest extent while converting the original features to the target features when generating a new image y with the target features, the recombination loss is applied to the generator. The definition of the recombination loss is expressed as:

[0047] L re =||xG(G(x,c t ), c o )||1 (12)

[0048] In formula (12), x is the original image input to the FFEGAN model; y is the generated image output by the FFEGAN model; c o is the label of the original domain corresponding to the original image;

[0049] For the two different discriminant branches of the discriminator - adversarial discriminant branch D adv And classification and discrimination branch D cls During the model training process, due to its different purposes, the goal of the generator G is to continuously reduce and minimize the adversarial loss L adv , while the goal of the discriminator D is to continuously improve and increase the adversarial loss; the corresponding loss functions are also completely different;

[0050] The adversarial discriminant branch D adv The loss function is expressed as:

[0051]

[0052] In formula (13), L adv is the adversarial loss; x is the original image input to the FFEGAN model, D adv (x) is the probability distribution of the image output by the adversarial discriminant branch of the discriminator; c t is the target domain label of the transformation;

[0053] The classification and discrimination branch D cls The loss function is expressed as:

[0054] ① When training the discriminator D, the classification and discrimination branch D cls The function is to use classification loss to evaluate the attribute category of the real image, so the loss function is expressed as:

[0055]

[0056] In formula (14), D cls (c o |x) is the confidence probability distribution of the classification discriminant branch on whether the input image is the output feature attribute; x is the original image input to the FFEGAN model, c o is the original domain label of the original image feature attribute;

[0057] ② When training the generator G, the classification and discrimination branch D cls The function is to help the generator G generate labels c that meet the target domain t The image is, so the loss function is expressed as:

[0058]

[0059] In formula (15), D cls (c t |G(x,c t )) is the confidence probability distribution of the classification discriminant branch on whether the input image belongs to the target feature attribute; x is the original image input to the FFEGAN model, c t It represents the target domain of the conversion task, that is, the label of the target feature attribute.

[0060] Furthermore, the context loss, for the original input image x and the generated image y, regards them as the set of all features contained in their respective images, denoted as: X1 = {x i},Y1={y j}; When calculating, the size of X1 and Y1 as sets needs to be the same, that is, |Y1| = |X1|. When the sizes of the two sets are different, additional features need to be added from the larger set;

[0061] First, the cosine distance of the feature needs to be normalized. The principle of this operation is expressed as follows:

[0062]

[0063] In formula (16), d ij For x i to y jThe cosine distance; l is any positive integer not equal to j; ∈ is the offset of the denominator, ∈=1e-5;

[0064] Then, the normalized distance needs to be converted into the similarity between features. This conversion process is completed through exponential operation. At the same time, in order to make the similarity between different features have a unified scale when measuring, the similarity is defined as the quotient of the similarity between the original feature and the current feature and the total similarity between the original feature and all other features. The principle of this process is expressed by the formula:

[0065]

[0066] In formula (17), h is a bandwidth parameter, h>0; CX ij is the similarity between features i and j;

[0067] When formula (17) is applied to the full image, the definition of the similarity between two sample images is expressed as:

[0068]

[0069] Assuming that the FFEGAN model uses VGG19 as the loss network, the feature set of the input image is extracted through the VGG19 loss network, and then the feature set is applied to the context loss function. The definition of the context loss function is expressed by the formula:

[0070] L CX (x, y, k) = -log(CX(Φ k (x), Φ k (y))) (19)

[0071] In formula (19), Φ k (x) and Φ k (y) are the feature sets extracted from images x and y in the kth layer of the loss network Φ;

[0072] Further applying formula (19) to the FFEGAN model, the context loss function corresponding to the FFEGAN model is expressed as:

[0073] L CX (G(x), x, k) = -log(CX(Φ k (G(x)),Φ k (x))) (20)

[0074] According to the context loss function provided by formula (20), it can be understood that the generator G needs to minimize the context loss L in the process of generating images. CX, which will help improve the similarity between the generated image and the original image at the feature set level; therefore, the total loss function of the generator G in the FFEGAN model can be obtained as:

[0075]

[0076] In formula (21), L G represents the total loss of the generator G; L adv Adversarial loss for the discriminator against the discriminative branch; is the classification loss of the discriminator classification branch; L re L is the reorganization loss used to maintain the original information of the image; cx is the context loss; cls ,λ re and λ cx are the weight coefficients of classification loss, recombination loss and context loss. These three weight coefficients will affect the balance between maintaining the original information and feature conversion in the generated image;

[0077] The total loss function of the FFEGAN model discriminator D is:

[0078]

[0079] In formula (22), L D is the total loss of the discriminator; L adv Adversarial loss for the discriminator against the discriminative branch; is the classification loss of the discriminator classification branch; cls is the weight coefficient of the classification loss, which affects the balance between the discriminator and the adversarial generator and the correct classification label.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] 1. In order to effectively solve the problem of weak control of the model by the unique hot encoding used in the traditional model, the method of the present invention designs a FFEGAN model for editing facial features; this model is an improvement on the structure of the generative adversarial network. By adding a feature information mapping network to the generator part, the generator can effectively inject deep feature information into the generation process multiple times during the training process, thereby improving the control of the attribute feature label generator. In the discriminator part of the network, by setting up adversarial discriminant branches and classification discriminant branches, the discriminator can better perform the discrimination task, and can also use classification loss to better guide the training of the generator. The final experimental results show that the FFEGAN model effectively improves the control effect of features and attributes, and solves the problem of insufficient control of feature attributes in the conversion process of the current method, and verifies that the FFEGAN model is superior to other existing methods in feature attribute conversion capabilities.

[0082] 2. By adding an attention mask generation network to the FFEGAN model, the present invention enables the model to better focus on the feature parts that need to be converted during the process of generating pictures, thereby improving the level of retention of the original image information in the generated image; in addition, due to the design defects of the reconstruction loss itself, which will lead to the inhibition of the generation effect, the present invention proposes to use a pre-trained loss network to calculate the context loss, thereby effectively solving this problem. The final experimental results show that the improved method has well solved the problem of easy loss of details and original identity information during image conversion, effectively improving the conversion ability of the model, and verifying that the improved method is superior to other models and methods in terms of conversion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 It is a schematic diagram of the feature conversion image generation process of the FFEGAN model of the present invention;

[0085] Figure 2 Schematic diagram of the image cycle verification process of the FFEGAN model of the present invention;

[0086] Figure 3 is a structural diagram of the FFEGAN model generator G of the present invention;

[0087] Figure 4 Schematic diagram of the residual block of the present invention;

[0088] Figure 5 It is a structural diagram of the discriminator D of the FFEGAN model of the present invention;

[0089] Figure 6 It is a structural diagram of the characteristic information mapping network of the present invention;

[0090] Figure 7 Schematic diagram of the comparison between the Leaky ReLU activation function used in the present invention and other activation functions'

[0091] Figure 8 Schematic diagram of the calculation principle of context loss of the present invention;

[0092] Fig. 9 Schematic diagram of some images in the CelebA dataset and the CelebA-HQ dataset used in the embodiments of the present invention;

[0093] Fig.10 It is a comparison chart of the image conversion effects of the FFEGAN model in the embodiment of the present invention and the control group model. DETAILED DESCRIPTION

[0094] In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the following detailed description is given to each step of the method proposed by the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0095] Example

[0096] The present invention provides a facial feature conversion method based on feature information mapping and attention mechanism, comprising the following steps:

[0097] Step 1: Build the facial feature editing model FFEGAN;

[0098] The facial feature editing model FFEGAN includes a generator, a discriminator and a loss network;

[0099] The generator consists of three parts: feature information mapping network, image generation network and attention mask generation network;

[0100] Step 2: Take the label of the target domain to be converted and the original image as the input of the face feature editing model FFEGAN, and output a generated image that converts the features from the original domain to the target domain;

[0101] The FFEGAN model proposed in this embodiment can be viewed as two parts, such as Figure 1As shown in , the first part is the part that converts the input image from the original domain to the target domain. Figure 2 As shown, the second part is to convert the image generated by the generator in the previous part back to the original domain, which is also the embodiment of the idea of ​​​​cyclic verification.

[0102] Specifically, the target domain label is first converted into deep feature information data by the feature information mapping network in the generator, and then added to the image generation network in the process of converting the original image to the generated image to control the conversion. Finally, the generated image enters the discriminator for evaluation to evaluate the authenticity of the generated image and classify the generated image into domains. The discriminator's evaluation of the generated image will help the generator learn how to generate better images.

[0103] In addition, after the feature conversion is completed, the image will be added to the generator together with the domain label representing the original domain, and an original domain image after two conversions will be obtained through regeneration; by comparing the difference between the original image and the image after two conversions, the generator will learn how to retain the original information during the conversion process, which will also improve the quality of the images generated by the generator and the conversion effect.

[0104] In step 1, if Figure 3 As shown, the generator G is composed of an image generation network G that generates a target image through an encoder-decoder structure. gen , an attention mask generation network G that uses a multi-head self-attention mechanism to generate attention masks mask It consists of three parts: a feature information mapping network M that converts one-hot encoding into high-dimensional potential information;

[0105] Assume that the original face image is x and the feature label assigned to the original domain is c 0 , the target domain is assigned a feature label of c t , the generator is based on the original face image x and the target domain label c t As input, generate a target image y * , and make its feature label the same as c t Matching while retaining other original facial information and attributes, including edge details, identity information, etc.

[0106] In addition, the target domain label c t In the generator, it will be generated randomly. This design allows the generator to generate images without relying on the input label distribution, which can better learn. During the model training process, the input feature label c t It will be mapped by the feature information mapping network M into a vector k containing feature attribute information t, this vector will be injected into the image generation network G multiple times gen To strengthen the control of the label on the generation, the image generation network G gen The mathematical logic expression is:

[0107] G gen (x,k t )=y (1)

[0108] The mathematical logic expression of the generator G is:

[0109] G(x,c t )=G gen (x,M(c t ))=y (2)

[0110] In formulas (1) and (2), x is the original image input to the generator, y is the new image generated by the generator, and M is the feature information mapping network.

[0111] When the generator's generation work runs to the decoder part, when upsampling the information at different network levels, the vector k containing the feature attribute information t Will be injected into the image generation network G multiple times gen Different levels are used to strengthen the control of tags at different levels of the decoder.

[0112] The image generation network G gen The encoder part of is composed of three downsampling layers and the same number of bottleneck layers. The downsampling layers are used to perform convolution operations on the input data to extract the feature information contained in the image. After each convolution operation, the adaptive instance normalization method is used for normalization.

[0113] The image generation network G gen The decoder part is structurally symmetrical to the encoder part. The decoder part consists of three bottleneck layers and the same number of upsampling layers. The upsampling layers are used to perform transposed convolution operations, and the bottleneck layers use residual blocks for jump connections.

[0114] Specifically, in order to generate a credible target image, in the image generation network G gen The bottleneck layer of the encoder and decoder uses a skip connection, which is the residual block proposed by ResNet. This design enables the FFEGAN model to more effectively control the characteristics of the generated samples; the image generation network Ggen in the FFEGAN model will not use the target image as the output of the network, but will do its best to output a facial texture mask that matches the characteristics of the conversion target domain. In general, when an original face image x is input, the original domain is assigned a feature label c0 , and the target domain is assigned the target label c t The main task of the generator is to accept the original face image x and the target domain label c t As input, a target image y is further generated by synthesizing the generated attention mask AttM and the facial region texture mask y′. * , whose feature labels are the same as c t To match, during the conversion, other original information on the original image x that is not related to the conversion work needs to be retained, including the image's material details, edge features and other attributes.

[0115] The purpose of adding residual blocks to the network is to solve the problem of gradient vanishing and gradient exploding, while also increasing the depth of the network so that deep features can be better captured. In deep neural networks, as the number of network layers increases, the problems of gradient vanishing and gradient exploding become more and more serious. Residual blocks use skip connections so that gradients can be directly back-propagated to shallower network layers. In addition, residual blocks can also enable the network to better learn the characteristics of the data by retaining the information of the original input.

[0116] like Figure 4 As shown, the residual block is in the image generation network G gen The working principle is expressed in the formula:

[0117] y=σ(F(x,W)+x) (3)

[0118] In formula (3), σ(·) represents the activation function, F(·) represents the residual function, and W represents all weights in the residual block;

[0119] For a deeper layer L, its relationship with the previous layer is:

[0120]

[0121] After each convolution operation, the adaptive instance normalization method is used for normalization. The calculation process of the adaptive instance normalization is:

[0122] AdaIN(X, k t )=(1+f(k t )1)(IN(X))+f(k t )2 (5)

[0123] In formula (5), X is the feature distribution of the image; let the dense layer be represented by function f, then f(k t )1 and f(k t )2 are the two parameters generated by the slice operation.

[0124] Furthermore, the attention mask generation network G mask Architecture and Image Generation Network G gen The architecture is similar, the attention mask generation network G mask After the last layer of the bottleneck layer, a calculation operation for multi-head self-attention mechanism calculation is added. The number of heads of the multi-head self-attention mechanism is set to 3. The attention mask generation network G mask In the upsampling layer of the attention mask generation network G, the instance normalization method is used to normalize the features. mask In the last bottleneck layer, the number of convolution kernels is set to 1, and the attention mask generation network G mask The original image x is taken as input, and finally an attention mask AttM with a channel number of 1 and a resolution the same as that of the original face image x is output. The mask AttM contains information about the relevant feature positions of the original image and weight information of different regions in the conversion work;

[0125] Specifically, the addition of the attention mask will enable the model to better locate features and help the model not be easily affected by other minor and irrelevant parts during the conversion of target features. In addition, the addition of the attention mask can also help the generator retain more original information of the original image during the conversion process.

[0126] In the attention mask generation network G mask Based on the correct generation of the attention mask AttM, the image generation network G gen A successful conversion can be completed by converting the relevant areas marked by the mask while keeping the other areas as they are. The generator G outputs the image y * The process is expressed by the formula:

[0127] y * =(1-AttM)·x+AttM·y′ (6)

[0128] In formula (6), x is the original image input to the generator, y′ is the relevant area marked by the mask, and AttM is the attention mask;

[0129] The overall working principle of the generator G is expressed as:

[0130] G(x,c t )=G mask (x)·G gen (x,F(c t ))+(1-G mask (x))·x (7).

[0131] Further, in step 1, if Figure 5 As shown, the discriminator includes two discriminant branches, and the two discriminant branches are adversarial discriminant branches D adv And classification and discrimination branch D cls ; The adversarial discriminant branch D adv The full convolution form is adopted. After the input image is convolved, its output result will continue to be projected into an N-order projection matrix containing discriminant information through the convolution operation. Each value in the N-order projection matrix corresponds to the evaluation result of the discriminator on a certain area of ​​the image. The classification discrimination branch D cls Used to output a probability distribution describing the features of the input image, wherein the probability distribution is used to assist the face feature editing model FFEGAN to learn various feature attributes and classify them into correct feature labels;

[0132] The overall structure of the discriminator is expressed as follows:

[0133] D={D adv , D cls} (8).

[0134] Furthermore, the feature information mapping network can transform and map specific, discrete attribute labels into abstract high-dimensional feature control information. As the training process continues, the feature information mapping network will continue to learn how to better perform various affine mappings and nonlinear mappings on discrete attribute feature labels, and output high-dimensional latent codes that carry fine feature information. Ideally, the feature information mapping network will fully learn the mapping method, and the output information will contain all the semantic and feature information carried by the label. By converting the single and discontinuous one-hot encoding into a high-dimensional latent encoding, the feature information mapping network can help the model better find and understand the relationship between control information and image features, which will also help to refine and improve the control of feature information over attribute editing.

[0135] The function of the feature information mapping network is expressed as follows:

[0136] k t =M(c t ) (9)

[0137] In formula (9), M is the feature information mapping network; c t is a discrete feature attribute label, k t It is a high-dimensional latent code that contains implicit feature information;

[0138] like Figure 6As shown, the structure of the feature information mapping network is designed as a feedforward neural network, whose input is an n-dimensional one-hot encoding. After entering the feature information mapping network, the one-hot encoding needs to pass through nine dense layers, and each dense layer is followed by an activation layer; when the one-hot encoding passes through the dense layer, the neurons therein will perform affine transformation on the one-hot encoding, and the one-hot encoding will perform nonlinear transformation after the activation layer. After passing through nine dense layers and nine activation layers, the feature information mapping network will output a high-dimensional potential code. The operation of the one-hot encoding performing affine transformation in the dense layer is expressed by the formula:

[0139]

[0140] In formula (10), z is the final output value of the affine transformation; the weight between any neuron in the kth layer and any neuron in the k+1th layer is ω i Indicates; m is the maximum number of layers of the feature information mapping network; c is an additional bias to offset the affine transformation offset during the calculation process;

[0141] In order to map the n-dimensional discrete feature attribute labels to 512 dimensions through affine transformation, the input layer of the feature information mapping network and the seven dense layers thereafter are composed of 512 neurons, and the output layer of the feature information mapping network is composed of N neurons, which will keep the information dimension of the final output of the network to be N;

[0142] like Figure 7 As shown in the figure, in the research and application of deep learning, Sigmoid, Tanh, ReLU and Leaky ReLU functions are widely used as activation functions. However, compared with Sigmoid, Tanh and ReLU functions, Leaky ReLU activation function shows its unique advantages. Leaky ReLU function is a typical piecewise function. When the input value is greater than 0, the input will be directly mapped to the output. When the input value is less than 0, the output will not be completely expressed as 0, but multiplied by a very small slope α (usually -0.01) and mapped to a very small negative number. Compared with Sigmoid and Tanh functions, Leaky ReLU function has a derivative function, which makes back propagation possible. The advantage of higher computational efficiency also allows the network to converge quickly. Compared with ReLU activation function, Leaky ReLU function effectively solves the problem of neuron death caused by ReLU function when the input value is less than 0. In addition, since the derivative of Leaky ReLU function is always 1 in the region where the input value is greater than 0, it means that the gradient will always be kept at 1. This feature makes the Leaky ReLU function more efficient when the model performs gradient descent and back propagation, effectively avoiding the problems of gradient explosion and gradient disappearance.

[0143] Therefore, in this embodiment, the activation layer uses the leaky linear rectifier unit Leaky ReLU as the activation function, and the mathematical expression of the LeakyReLU activation function is:

[0144]

[0145] Furthermore, in this embodiment, the image generation network G gen The context loss is calculated by using a VGG19 network pre-trained in the field of image classification as the loss network, such as Figure 8 As shown, the generated image and the original image are input into the loss network VGG19, and the feature distribution of the image is extracted and the context loss is calculated in the last activation layer ReLU5_4 before the dense layer. The feature information extracted in this layer can best reflect the potential deep features of the image.

[0146] In generative adversarial networks, the loss function plays a crucial role, which directly affects the training process of the generator and the discriminator.

[0147] Specifically, during the training process of the FFEGAN model, in order to ensure that the generator G can retain other information of the original image x to the greatest extent while converting the original features to the target features when generating a new image y with the target features, the recombination loss is applied to the generator G. The definition of the recombination loss is expressed as:

[0148] L re =||xG(G(x,c t ), c o )||1 (12)

[0149] In formula (12), x is the original image input to the FFEGAN model; y is the generated image output by the FFEGAN model; c o is the label of the original domain corresponding to the original image;

[0150] For the two different discriminant branches of the discriminator D - adversarial discriminant branch D adv And classification and discrimination branch D cls During the model training process, due to its different purposes, the goal of the generator G is to continuously reduce and minimize the adversarial loss L adv , while the goal of the discriminator D is to continuously improve and increase the adversarial loss; the corresponding loss functions are also completely different;

[0151] The adversarial discriminant branch D adv The loss function is expressed as:

[0152]

[0153] In formula (13), L adv is the adversarial loss; x is the original image input to the FFEGAN model, D adv (x) is the probability distribution of the image output by the adversarial discriminant branch of the discriminator; c t is the target domain label of the transformation;

[0154] The classification and discrimination branch D cls The loss function is expressed as:

[0155] ① When training the discriminator D, the classification and discrimination branch D cls The function is to use classification loss to evaluate the attribute category of the real image, so the loss function is expressed as:

[0156]

[0157] In formula (14), D cls (c o |x) is the confidence probability distribution of the classification discriminant branch on whether the input image is the output feature attribute; x is the original image input to the FFEGAN model, c o is the original domain label of the original image feature attribute;

[0158] ② When training the generator G, the classification and discrimination branch D cls The function is to help the generator G generate the target feature attribute label c t The image is, so the loss function is expressed as:

[0159]

[0160] In formula (15), D cls (c t |G(x,c t )) is the confidence probability distribution of the classification discriminant branch on whether the input image belongs to the target feature attribute; x is the original image input to the FFEGAN model, c t It represents the target domain of the conversion task, that is, the label of the target feature attribute.

[0161] Furthermore, the context loss, for the original input image x and the generated image y, regards them as the set of all features contained in their respective images, denoted as: X1 = {x i}, Y1={y j}; When calculating, the size of X1 and Y1 as sets needs to be the same, that is, |Y1| = |X1|. When the sizes of the two sets are different, additional features need to be added from the larger set;

[0162] First, the cosine distance of the feature needs to be normalized. The principle of this operation is expressed as follows:

[0163]

[0164] In formula (16), d ij For x i to y j The cosine distance; l is any positive integer not equal to j; ∈ is the offset of the denominator, ∈=1e-5;

[0165] Then, the normalized distance needs to be converted into the similarity between features. This conversion process is completed through exponential operation. At the same time, in order to make the similarity between different features have a unified scale when measuring, the similarity is defined as the quotient of the similarity between the original feature and the current feature and the total similarity between the original feature and all other features. The principle of this process is expressed by the formula:

[0166]

[0167] In formula (17), h is a bandwidth parameter, h>0; CX ij is the similarity between features i and j;

[0168] When formula (17) is applied to the full image, the definition of the similarity between two sample images is expressed as:

[0169]

[0170] Assuming that the FFEGAN model uses VGG19 as the loss network, the feature set of the input image is extracted through the VGG19 loss network, and then the feature set is applied to the context loss function. The definition of the context loss function is expressed by the formula:

[0171] L CX (x, y, k) = -log(CX(Φ k (x), Φ k (y))) (19)

[0172] In formula (19), Φ k (x) and Φ k (y) are the feature sets extracted from images x and y in the kth layer of the loss network Φ;

[0173] Further applying formula (19) to the FFEGAN model, the context loss function corresponding to the FFEGAN model is expressed as:

[0174] L CX (G(x), x, k) = -log(CX(Φk (G(x)),Φ k (x))) (20)

[0175] According to the context loss function provided by formula (20), it can be understood that the generator G needs to minimize the context loss L in the process of generating images. CX , which will help improve the similarity between the generated image and the original image at the feature set level; therefore, the total loss function of the generator G in the FFEGAN model can be obtained as:

[0176]

[0177] In formula (21), L G represents the total loss of the generator G; L adv Adversarial loss for the discriminator against the discriminative branch; is the classification loss of the discriminator classification branch; L re L is the reorganization loss used to maintain the original information of the image; cx is the context loss; cls ,λ re and λ cx are the weight coefficients of classification loss, recombination loss and context loss. These three weight coefficients will affect the balance between maintaining the original information and feature conversion in the generated image;

[0178] The total loss function of the FFEGAN model discriminator D is:

[0179]

[0180] In formula (22), L D is the total loss of the discriminator; L adv Adversarial loss for the discriminator against the discriminative branch; is the classification loss of the discriminator classification branch; cls is the weight coefficient of the classification loss, which affects the balance between the discriminator and the adversarial generator and the correct classification label.

[0181] In order to evaluate whether the facial feature conversion method of the present invention is feasible in practice, and also to evaluate the performance and effect of the proposed FFEGAN model, this embodiment selects two internationally recognized public face databases, namely the CelebA

[45] (Large-scale CelebFaces Attributes (CelebA) Dataset) dataset and the CelebA-HQ

[46] (CelebFaces Attributes (CelebA-HQ) Database High Quality) dataset for experiments.

[0182] Some images of the CelebA dataset and CelebA-HQ dataset are shown in the figure below. Fig. 9 As shown in Figure 1, CelebA-HQ is a high-resolution version of CelebA created by T.Karras et al. It contains 30,000 high-resolution images, which provide richer and more realistic facial details compared to the low-resolution CelebA dataset. In theory, the image quality of the CelebA-HQ dataset is relatively higher and more suitable for more complex tasks such as image generation, editing, and attribute transfer.

[0183] The following is the specific evaluation process:

[0184] 1. Establishing evaluation indicators

[0185] In this embodiment, two indicators, Fréchet Inception Distance (FID) score and Kernel Inception Distance (KID) score, are used. The calculation method of FID distance score is:

[0186]

[0187] In formula (23), μ X and Σ X Represent the mean and covariance of the training data respectively; μ Y and Σ Y represents the mean and covariance of the generated image (test data); Tr represents the trace of the matrix (that is, the sum of the eigenvalues ​​of the matrix); |||·||| represents the bi-norm of the vector.

[0188] The specific calculation steps are to first extract the feature vector from the network, then calculate the mean vector and covariance matrix of the two distributions to be compared, and finally calculate the Fréchet distance between the two matrix distributions to get the FID score.

[0189] The KID distance score is calculated as:

[0190] KID=MMD(f real ,f fake ) 2 (twenty four)

[0191] In formula (24), MMD represents the maximum mean difference, f real and f fakeare the means of the features extracted from the real image and the generated image, respectively. The mean difference between the two sets of features is calculated, and the maximum mean difference is defined as the maximum Euclidean distance between the two means. Finally, the value of the maximum mean difference is squared to obtain the final KID.

[0192] 2. Experimental Environment

[0193] The operating system of the experimental machine is Windows1122631.329623H2 version, and the programming language used is Python, whose version is python 3.9. For matrix operations with larger dimensions, the mathematical function library in the reference package numpy supported by Python is used for calculation. The implementation of the methods and models proposed in this paper, as well as the implementation of the comparative methods and models, are all based on the open source framework Pytorch, and the version of Pytorch is Pytorch stable (1.11.0). All experiments were carried out in the Pycharm integrated development platform environment, and the version number of Pycharm is 2021.3.3. As for the specific hardware configuration of the experimental machine, the CPU used in the experiment is Intel (R) Core (TM) i9-10900K CPU @ 3.70GHz, the host memory is 64GB, the GPU is NVIDIA Quadro RTX4000, and the video memory is 16GB.

[0194] 3. Experimental parameters

[0195] Due to the limitations of hardware devices such as computing power and GPU video memory, the image resolution is set to 256*256 in this embodiment. Images of this resolution can retain the image information as much as possible while ensuring the normal operation of the experiment. During the experiment, all batch sizes were set to 16 and the number of iterations was set to 200,000. The optimizer selected for use is the Adam optimizer. This optimization method combines the advantages of the two optimization algorithms, AdaGrad and RMSProp, and can better update parameters. In terms of algorithm parameter design, the key parameters β1 of the Adam optimizer are set to 0.5 and β2 are set to 0.999. These two parameters are the decay rate of the exponential moving average used to calculate the gradient and the decay rate of the exponential moving average used to calculate the square of the gradient, respectively. At the beginning of the experiment, the initial learning rate of the FFEGAN model will be set to 1.0×10 -4 In the first half of the experiment, that is, in the first 10,000 iterations of the FFEGAN model, in order to enable the model to learn better, the learning rate of the model is set to the initial learning rate; in all subsequent iterations, the learning rate will continue to decrease, and every time the number of iterations increases by 1,000, the learning rate will decrease by the quotient of the current learning rate and the current number of iterations. The weight of the classification loss λcls and the weight λ of the cycle consistency loss re The values ​​of are set to 1 and 10 respectively.

[0196] IV. Experimental Procedure

[0197] The FFEGAN model provided by the method of the present invention consists of a generator and a discriminator and two branches thereof. A loop is completed within the model by sharing the generator to perform a loop consistency check. In the first stage of the loop, the original image and the target domain label are used as input, and the generator will generate an image that meets the target feature attribute label. In the second stage of the loop, the generated image and the source domain feature attribute label are used as input, and the generator generates a reconstructed image that meets the original feature attribute.

[0198] The specific details of the loop are as follows:

[0199] Step 1: The parameters of the generator G remain unchanged, and only the discriminator D is updated. Input the original image and the target domain label into the generator G to generate image samples. Select real samples from the data set and use the discriminator D to judge their authenticity and category. In this way, the parameters of the discriminator D are adjusted and updated.

[0200] Step 2: The parameters of the discriminator D remain unchanged, and only the generator G is updated. The original image and the target domain label are input into the generator G to generate image samples, which are then judged by the discriminator D for authenticity and category. The parameters of the generator G are adjusted and updated accordingly.

[0201] Step 3: Add the generated fake sample image and source domain feature attribute label as input to the generator, calculate the difference between the reconstructed image and the original image input in the first step, and adjust and update the parameters of the generator G accordingly.

[0202] 5. Experimental Design

[0203] The experiment in this embodiment is divided into two parts. The first part is an ablation experiment on the FFEGAN model, which verifies the impact of the proposed method on the overall performance by continuously reducing the model content of FFEGAN. The second part is a horizontal comparison experiment with the advanced methods in the current field.

[0204] It should be noted that in order to fully verify the performance and effect of the method and model proposed in this article, other mainstream and most representative image-to-image translation frameworks are used for comparative experiments in this embodiment. The control group selected three methods, CycleGAN, ComboGAN and StarGAN, for comparison. The CycleGAN, ComboGAN and StarGAN models in the control group were trained using the default configuration provided by their authors in terms of parameter configuration.

[0205] Ablation experiment

[0206] Since the original GAN ​​must be trained with paired data, the ablation experiment will use CycleGAN as the baseline. The FFEGAN model proposed in this embodiment improves the structure of the generative adversarial network in the following ways: 1) a feature information mapping network is added to the generator to help the generator train; 2) the discriminant function of the discriminator is designed into two branches, which respectively perform the tasks of true and false and type discrimination; 3) the recombination loss is applied to reverse the generated image to help training.

[0207] Therefore, the ablation experiment will focus on the three modification points mentioned above. The attributes selected during the experiment are relatively easier to observe, such as hair color, gender, wearing glasses and baldness, etc. The specific hair color has four sub-attributes: black, gold, brown, and gray. Different hair colors can be converted to each other. For binary attributes other than hair color, the target domain of the conversion is the opposite attribute, such as male and female, wearing glasses and not wearing glasses, bald and not bald, etc. The FID score and KID score are used to quantitatively evaluate the ablation experiment results. The lower the FID score and KID score, the better the image generation effect. The results of the ablation experiment are shown in Table 1 below:

[0208] Table 1. Ablation experiment results

[0209]

[0210] In the table, M represents the feature mapping network module, and D represents the dual-branch discriminator module.

[0211] The quantitative analysis scores shown in the results in the table are the arithmetic mean of the scores after the conversion of each attribute. Since CycleGAN has already reflected the idea of ​​cyclic verification in the model through the application of cycle consistency loss, the difference between the recombination loss and it is more reflected in the number of generators, so it is not significant to conduct a separate ablation experiment on the third improvement.

[0212] From the results of the ablation experiment, it can be seen that after adding the feature information mapping network, the model's conversion effect on images and features has been significantly improved. However, adding a classification discrimination branch alone has limited improvement on the model. This is because the loss of this branch does not have strong control over the generator alone during training. Even if the feature attribute label has completed the mapping of deep feature information, this feature information is difficult to fully and effectively control the image generation of the generator. It should be noted that the FFEGAN model is significantly different from CycleGAN in the implementation method of the cycle idea. CycleGAN requires an additional generator to complete the image cycle, while the method of the present invention only requires one generator.

[0213] Comparative experiment

[0214] The conversion of attribute features in the horizontal comparison experiment is consistent with the feature attributes selected in the ablation experiment; the results of the comparison experiment are shown in Table 2 below:

[0215] Table 2. Comparative experimental results

[0216]

[0217] From the results of the comparative experiment, it can be seen that the FID score and KID score of the method of the present invention are lower than those of the other three methods in the control group. This also means that the quality of the images generated by the method proposed in the present invention is higher than that of the existing methods. When performing the task of facial feature editing, it can perform the conversion with better effect, and at the same time can better maintain the original identity, posture and other information of the image.

[0218] In this embodiment, the conversion effect comparison is also used to conduct qualitative analysis of the comparative experiment, where the conversion effect is as follows: Fig.10 shown.

[0219] from Fig.10 It can be seen that for complex feature attribute transformations, CycleGAN has obvious deformation; ComboGAN performs well in the hair color project, but performs poorly in the gender project, which is because the control of the ComboGAN model is not refined enough. StarGAN is a model that performs well in the control group. Only in some projects, there are extra features such as hair and hair ends. This is caused by the lack of cycle consistency loss used by StarGAN. The FFEGAN model is the best performing model in the experiment. While ensuring the conversion effect, it also retains sufficient original identity information. In addition, the method of the present invention is superior to other methods in retaining the original skin details, edges and textures, which means that the method of the present invention more effectively reduces the loss of original information in the input image.

[0220] The results of qualitative analysis show that the images generated by the FFEGAN model provided by the method of the present invention are more vivid and realistic than those by other existing methods. This also confirms the conclusion in quantitative analysis that the quality of images generated by the method of the present invention is higher than that of other existing methods.

[0221] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A facial feature conversion method based on feature information mapping and attention mechanism, characterized in that: The following steps are involved: Step 1: Build the facial feature editing model FFEGAN; The facial feature editing model FFEGAN includes a generator, a discriminator and a loss network; The generator consists of three parts: feature information mapping network, image generation network and attention mask generation network; Step 2: Take the label of the target domain to be converted and the original image as the input of the face feature editing model FFEGAN, and output a generated image that converts the features from the original domain to the target domain; First, the target domain label is converted into deep feature information data by the feature information mapping network in the generator, and then added to the image generation network to control the conversion during the conversion of the original image to the generated image. Finally, the generated image enters the discriminator for evaluation to evaluate the authenticity of the generated image and classify the domain of the generated image.

2. According to claim 1, a facial feature conversion method based on feature information mapping and attention mechanism is characterized in that: In step 1, the generator G is an image generation network G that generates a target image through an encoder-decoder structure. gen , an attention mask generation network G that uses a multi-head self-attention mechanism to generate attention masks mask It consists of three parts: a feature information mapping network M that converts one-hot encoding into high-dimensional potential information; Assume that the original face image is x and the feature label assigned to the original domain is c 0 , the target domain is assigned a feature label of c t , the generator is based on the original face image x and the target domain label c t As input, generate a target image y and make its feature label the same as c t Matching while preserving the original facial information and attributes; The target domain label c t In the generator, it will be generated randomly. During the model training process, the input target domain label c t It will be mapped by the feature information mapping network M into a vector k containing feature attribute information t , this vector will be injected into the image generation network G multiple times gen To strengthen the control of the label on the generation, the image generation network G gen The mathematical logic expression is: G gen (x,k t )=y (1) The mathematical logic expression of the generator G is: G(x,c t )=G gen (x,M(c t ))=y (2) In formulas (1) and (2), x is the original image input to the generator, y is the new image generated by the generator, and M is the feature information mapping network. When the generator's generation work runs to the decoder part, when upsampling the information at different network levels, the vector k containing the feature attribute information t Will be injected into the image generation network G multiple times gen Different levels are used to strengthen the control of tags at different levels of the decoder.

3. According to claim 2, a facial feature conversion method based on feature information mapping and attention mechanism is characterized in that: The image generation network G gen The encoder part of is composed of three downsampling layers and the same number of bottleneck layers. The downsampling layers are used to perform convolution operations on the input data to extract the feature information contained in the image. After each convolution operation, the adaptive instance normalization method is used for normalization. The image generation network G gen The decoder part of the proposed method is symmetrical with the encoder part in structure. The decoder part consists of three bottleneck layers and the same number of upsampling layers. The upsampling layers are used to perform transposed convolution operations. The bottleneck layers use residual blocks for skip connections. The residual blocks are used in the image generation network G. gen The working principle is expressed in the formula: y=σ(F(x,W)+x) (3) In formula (3), σ(·) represents the activation function, F(·) represents the residual function, and W represents all weights in the residual block; For a deeper layer L, its relationship with the previous layer is: After each convolution operation, the adaptive instance normalization method is used for normalization. The calculation process of the adaptive instance normalization is: AdaIN(X,k t )=(1+f(k t )1)(IN(X))+f(k t )2 (5) In formula (5), X is the feature distribution of the image; let the dense layer be represented by function f, then f(k t )1 and f(k t )2 are the two parameters generated by the slice operation.

4. According to claim 2, a facial feature conversion method based on feature information mapping and attention mechanism is characterized in that: The attention mask generation network G mask Architecture and Image Generation Network G gen The architecture is similar, the attention mask generation network G mask After the last layer of the bottleneck layer, a calculation operation for multi-head self-attention mechanism calculation is added. The number of heads of the multi-head self-attention mechanism is set to 3. The attention mask generation network G mask In the upsampling layer of the attention mask generation network G, the instance normalization method is used to normalize the features. mask In the last bottleneck layer, the number of convolution kernels is set to 1, and the attention mask generation network G mask The original image x is taken as input, and finally an attention mask AttM with a channel number of 1 and a resolution the same as that of the original face image x is output. The mask AttM contains information about the relevant feature positions of the original image and weight information of different regions in the conversion work; In the attention mask generation network G mask Based on the correct generation of the attention mask AttM, the image generation network G gen A successful conversion can be completed by converting the relevant areas marked by the mask while keeping the other areas as they are. The generator G outputs the image y * The process is expressed by the formula: y * =(1-AttM)·x+AttM·y′ (6) In formula (6), x is the original image input to the generator, y′ is the relevant area marked by the mask, and AttM is the attention mask; The overall working principle of the generator G is expressed as: G(x,c t )=G mask (x)·G gen (x,F(c t ))+(1-G mask (x))·x (7)。 5. According to claim 1, a facial feature conversion method based on feature information mapping and attention mechanism is characterized in that: In step 1, the discriminator includes two discriminant branches, and the two discriminant branches are adversarial discriminant branches D and adv And classification and discrimination branch D cls ; The adversarial discriminant branch D adv The full convolution form is adopted. After the input image is convolved, its output result will continue to be projected into an N-order projection matrix containing discriminant information through the convolution operation. Each value in the N-order projection matrix corresponds to the evaluation result of the discriminator on a certain area of ​​the image. The classification discrimination branch D cls Used to output a probability distribution describing the features of the input image, wherein the probability distribution is used to assist the face feature editing model FFEGAN to learn various feature attributes and classify them into correct feature labels; The overall structure of the discriminator is expressed as follows: D={D adv ,D cls } (8)。 6. The method for facial feature conversion based on feature information mapping and attention mechanism according to claim 1, characterized in that: The function of the feature information mapping network is expressed as follows: k t =M(c t ) (9) In formula (9), M is the feature information mapping network; c t is a discrete feature attribute label, k t It is a high-dimensional latent code that contains implicit feature information; The input of the feature information mapping network is an n-dimensional one-hot encoding. After entering the feature information mapping network, the one-hot encoding needs to pass through nine dense layers, and each dense layer is followed by an activation layer. When the one-hot encoding passes through the dense layer, the neurons therein will perform affine transformation on the one-hot encoding, and the one-hot encoding will undergo nonlinear transformation after passing through the activation layer. After passing through nine dense layers and nine activation layers, the feature information mapping network will output a high-dimensional potential code. The operation of affine transformation of the one-hot encoding in the dense layer is expressed by the formula: In formula (10), z is the final output value of the affine transformation; the weight between any neuron in the kth layer and any neuron in the k+1th layer is ω i Indicates; m is the maximum number of layers of the feature information mapping network; c is an additional bias to offset the affine transformation offset during the calculation process; The input layer and the subsequent seven dense layers of the feature information mapping network are composed of 512 neurons, and the output layer of the feature information mapping network is composed of N neurons; The activation layer uses the leaky linear rectifier unit Leaky ReLU as the activation function. The mathematical expression of the ReLU activation function is:

7. The method for facial feature conversion based on feature information mapping and attention mechanism according to claim 1, characterized in that: The image generation network Ggen calculates context loss by using a loss network pre-trained in the field of image classification, inputs the generated image and the original image into the loss network, extracts the feature distribution of the image in the last activation layer before the dense layer and calculates the context loss. The feature information extracted in this layer can best reflect the potential deep features of the image.

8. The method for facial feature conversion based on feature information mapping and attention mechanism according to claim 1, characterized in that: During the training process of the FFEGAN model, in order to ensure that the generator G can retain other information of the original image x to the greatest extent while converting the original features to the target features when generating a new image y with the target features, the recombination loss is applied to the generator. The definition of the recombination loss is expressed as: L re =||x-G(G(x,c t ),c o )||1 (12) In formula (12), x is the original image input to the FFEGAN model; y is the generated image output by the FFEGAN model; c o is the label of the original domain corresponding to the original image; For the two different discriminant branches of the discriminator - adversarial discriminant branch D adv And classification and discrimination branch D cls During the model training process, due to its different purposes, the goal of the generator G is to continuously reduce and minimize the adversarial loss L adv , while the goal of the discriminator D is to continuously improve and increase the adversarial loss; the corresponding loss functions are also completely different; The adversarial discriminant branch D adv The loss function is expressed as: In formula (13), L adv is the adversarial loss; x is the original image input to the FFEGAN model, D adv (x) is the probability distribution of the image output by the adversarial discriminant branch of the discriminator; c t is the target domain label of the transformation; The classification and discrimination branch D cls The loss function is expressed as: ① When training the discriminator D, the classification and discrimination branch D cls The function is to use classification loss to evaluate the attribute category of the real image, so the loss function is expressed as: In formula (14), D cls (c o |x) is the confidence probability distribution of the classification discriminant branch on whether the input image is the output feature attribute; x is the original image input to the FFEGAN model, c o is the original domain label of the original image feature attribute; ② When training the generator G, the classification and discrimination branch D cls The function is to help the generator G generate labels c that meet the target domain t The image is, so the loss function is expressed as: In formula (15), D cls (c t |G(x,c t )) is the confidence probability distribution of the classification discriminant branch on whether the input image belongs to the target feature attribute; x is the original image input to the FFEGAN model, c t is the target domain label of the conversion task, that is, the label of the target feature attribute.

9. The method for facial feature conversion based on feature information mapping and attention mechanism according to claim 7, characterized in that: The context loss, for the original input image x and the generated image y, regards them as the set of all features contained in their respective images, denoted as: X1 = {x i },Y1={y j }; When calculating, the size of X1 and Y1 as sets needs to be the same, that is, |Y1| = |X1|. When the sizes of the two sets are different, additional features need to be added from the larger set; First, the cosine distance of the feature needs to be normalized. The principle of this operation is expressed as follows: In formula (16), d ij For x i to y j The cosine distance; l is any positive integer not equal to j; ∈ is the offset of the denominator, ∈=1e-5; Then, the normalized distance needs to be converted into the similarity between features. This conversion process is completed through exponential operation. At the same time, in order to make the similarity between different features have a unified scale when measuring, the similarity is defined as the quotient of the similarity between the original feature and the current feature and the total similarity between the original feature and all other features. The principle of this process is expressed by the formula: In formula (17), h is a bandwidth parameter, h>0; CX ij is the similarity between features i and j; When formula (17) is applied to the full image, the definition of the similarity between two sample images is expressed as: Assuming that the FFEGAN model uses VGG19 as the loss network, the feature set of the input image is extracted through the VGG19 loss network, and then the feature set is applied to the context loss function. The definition of the context loss function is expressed by the formula: L CX (x, y, k)=-log(CX(Φ k (x), Φ k (y))) (19) In formula (19), Φ k (x) and Φ k (y) are the feature sets extracted from images x and y in the kth layer of the loss network Φ; Further applying formula (19) to the FFEGAN model, the context loss function corresponding to the FFEGAN model is expressed as: L CX (G(x), x, k)=-log(CX(Φ k (G(x)), Φ k (x))) (20) According to the context loss function provided by formula (20), it can be understood that the generator G needs to minimize the context loss L in the process of generating images. CX , which will help improve the similarity between the generated image and the original image at the feature set level; therefore, the total loss function of the generator G in the FFEGAN model can be obtained as: In formula (21), L G represents the total loss of the generator G; L adv Adversarial loss for the discriminator against the discriminative branch; is the classification loss of the discriminator classification branch; L re L is the reorganization loss used to maintain the original information of the image; cx is the context loss; cls ,λ re and λ cx are the weight coefficients of classification loss, recombination loss and context loss. These three weight coefficients will affect the balance between maintaining the original information and feature conversion in the generated image; The total loss function of the FFEGAN model discriminator D is: In formula (22), L D is the total loss of the discriminator; L adv Adversarial loss for the discriminator against the discriminative branch; is the classification loss of the discriminator classification branch; cls is the weight coefficient of the classification loss, which affects the balance between the discriminator and the adversarial generator and the correct classification label.

Images