Facial expression data enhancement method based on GANimationimprovement

Abstract

The invention provides a GANimationimprovement-based facial expression data enhancement method, which comprises the following steps of: acquiring a facial image data set, performing first cascade on AU sequences of facial expressions of an original image and a reference image to serve as input of a generator, acquiring a first generated image by utilizing the generator based on an attention mask and a color mask, and performing second cascade on the first generated image to serve as output of the generator; the generator carries out feature extraction by adopting a depth separable convolution module, and the generator carries out feature enhancement on extracted features of the depth separable convolution module by adopting a collaboration-based meta attention module; performing second cascade on the first generated image and the AU sequence of the facial expressions of the original image, and inputting the AU sequence into the generator to obtain a second generated image; carrying out authenticity discrimination by using a discriminator; predicting a facial expression AU sequence of the input image by using a discriminator; and updating network parameters of the generator and the discriminator by utilizing back propagation according to the joint loss function. The method improves the quality and details of the generated image, and enables the generated expression to be more real and natural.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision / deep learning, and in particular to a method for enhancing facial expression data. Background Art

[0002] Facial expressions, as an essential component of human emotional communication, have broad application value in fields such as computer vision and human-computer interaction. By recognizing and simulating facial expressions, computer systems can better understand and respond to human emotions, thereby improving user experience and human-computer interaction. Facial expression recognition technology has important applications in multiple fields, including but not limited to sentiment analysis, security monitoring, virtual reality, and intelligent customer service. For example, in sentiment analysis, by identifying the user's emotional state, the system can provide more personalized services; in security monitoring, by identifying abnormal expressions, the system can promptly detect potential security threats.

[0003] However, existing facial datasets often suffer from an imbalanced distribution of data types. In existing datasets, some expression types have a larger number of samples, while other expression types have a smaller number of samples. This uneven distribution can lead to low recognition rates of certain expressions in the trained model, affecting overall robustness and accuracy. In order to obtain diverse expression samples, data collection usually requires a lot of time and manpower. In addition, professional equipment and technical support are required, which increases the difficulty and cost of data collection. Due to the diversity of data collection environments, the quality of the collected data varies and may contain a lot of noise, blur, occlusion and other problems, which will affect the training effect of the model.

[0004] To address the above issues, researchers have proposed a variety of data enhancement techniques, such as traditional image enhancement methods (rotation, scaling, cropping, etc.) and methods based on generative adversarial networks (GANs). However, these methods still have some shortcomings. Limitations of traditional image enhancement methods: Although traditional image enhancement methods can increase data diversity, the generated images are of low quality and cannot well preserve the feature information of the original images. These methods can usually only enhance within a limited range and have difficulty generating complex and diverse expression images, especially when dealing with large expression spans.

[0005] Challenges of GAN-based data augmentation methods: Existing GAN-based data augmentation methods are prone to problems such as blurring and distortion when generating images, resulting in low quality of generated images that cannot meet the needs of practical applications. During the training process, the generator and discriminator are prone to overfitting, resulting in a lack of diversity in the generated images and difficulty generalizing to new datasets. The GAN training process is very unstable and prone to gradient vanishing or gradient exploding problems, making it difficult for the model to converge.

[0006] The invention patent with publication number CN 114399453, a method for synthesizing facial expressions based on generative adversarial networks, discloses the following steps: constructing a feature separation framework and a feature fusion framework based on generative adversarial networks, and either framework includes a mapping

[0007] The method uses a generative network consisting of a projection network and a progressive layer. It extracts an initial identity feature vector from the source image and feeds it into a feature separation framework for encoding and decoding to obtain an identity feature vector. It extracts an initial expression feature vector from the reference image and feeds it into a feature separation framework for encoding and decoding to obtain an expression feature vector. The identity feature vector and expression feature vector are sequentially fed into a generative network in a feature fusion framework using a specified progressive layer, where fusion operations are performed at different levels to obtain a facial expression image. This method can achieve high-resolution synthetic output of facial expression images. However, the quality and detail of the generated images still need to be improved. Summary of the Invention

[0008] To address the technical problems of poor quality and details of images generated by existing methods, this paper proposes a facial expression data enhancement method based on the improved GANimation. By adopting depthwise separable convolution, coordination-based meta-attention (CBAM) attention mechanism and total variation loss function in the generator, and adding a squeeze-and-excitation network (SENet) attention mechanism in the discriminator, the quality and details of the generated images are improved, making the generated expressions more realistic and natural.

[0009] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0010] A facial expression data enhancement method based on GANimation improvement, comprising the steps of:

[0011] S1: Acquire a facial image dataset, wherein each sample in the facial image dataset includes an original image and a reference image; extract an AU sequence of facial expressions of the reference image and an AU sequence of facial expressions of the original image respectively;

[0012] S2: Concatenating the AU sequences of facial expressions of the original image and the reference image for the first time as input to the generator, and the generator obtains a first generated image based on the attention mask and the color mask;

[0013] S3: Concatenate the first generated image and the AU sequence of facial expressions of the original image for a second time and input the concatenated image into the generator to obtain a second generated image;

[0014] S4: using the first generated image and the reference image as input images for the discriminator, and using the discriminator to perform authenticity discrimination; using the first generated image and the second generated image as input images for the discriminator, and using the discriminator to predict the facial expression AU sequence of the input image;

[0015] S5: Calculate the joint loss function based on the discriminator's discrimination result, the discriminator's prediction result, the second generated image, and the attention mask, and use backpropagation to update the generator and discriminator network parameters according to the joint loss function;

[0016] S6: Repeat steps S2 to S5 until the stopping condition is met, and use the generator of the network model with the optimal parameters to generate the target facial expression image.

[0017] Furthermore, the generator includes a generator input module, an attention generation module GA, a color generation module GI and a generator output module. The generator input module and the generator output module are respectively connected to the attention generation module GA and the color generation module GI. The generator input module is used to perform a cascade operation to obtain the generator input, the attention generation module GA is used to generate an attention mask A, the color generation module GI is used to generate a color mask C, and the generator output module is used to fuse the generator input, attention mask and color mask obtained by the cascade operation to obtain the generated image of the generator output.

[0018] Furthermore, the attention generation module GA and the color generation module GI share a depth-wise separable convolution module, a collaborative-based meta-attention module and an upsampling module connected in sequence. The depth-wise separable convolution module is used to extract multi-scale features based on the input image; the collaborative-based meta-attention module is used to adaptively select channels and spatially important areas to enhance key features to obtain an enhanced feature map; the upsampling module is used to gradually increase the spatial size of the enhanced feature map and restore it to the size of the input image.

[0019] Furthermore, the attention generation module GA adds a first convolution block after the upsampling module; the first convolution block generates an attention mask A through a convolution layer and constrains the output range in combination with a Sigmoid activation function;

[0020] The color generation module GI adds a second convolution block after the upsampling module. The second convolution block generates a transformation matrix of the RGB color channels as a color mask C through a convolution layer, and combines the Tanh activation function to constrain the output range.

[0021] Furthermore, the discriminator includes a discriminator input module, an authenticity discrimination module DI, an AU condition matching module Dy and a discriminator output module; the discriminator input module and the discriminator output module are respectively connected to the authenticity discrimination module and the AU condition matching module; the discriminator input module is used to receive the discriminator input image; the authenticity discrimination module DI is used to discriminate the authenticity of the image based on the discriminator input image and output the authenticity score; the AU condition matching module Dy is used to predict the AU sequence of the facial expression of the image based on the discriminator input image.

[0022] Furthermore, the authenticity discrimination module DI and the AU condition matching module Dy share the sequentially connected convolution block I, convolution block II, convolution block III and SENet module, where convolution block I, convolution block II and convolution block III are used to extract general image features, and the SENet module is used to perform feature enhancement; the authenticity discrimination module DI adds a sequentially connected convolution block IV, a global average pooling layer I and a linear layer after the SENet module, where convolution block IV is used to focus on higher-level image features, and the global average pooling layer I is used to fuse the global information of each channel of the feature map output by convolution block IV, and the linear layer outputs a classification score based on the global information; the AU condition matching module Dy adds a sequentially connected convolution block V and an output layer of the authenticity discrimination module DI after the SENet module, where convolution block V is used to extract features of facial expression-related areas, and the output layer of the authenticity discrimination module DI outputs a facial expression AU vector through a sequentially connected global average pooling layer II, a first fully connected layer and a second fully connected layer.

[0023] Furthermore, the joint loss function includes image adversarial loss, attention loss, conditional expression loss and identity loss, and the total variation loss is introduced into the attention loss to enhance the control of image smoothness.

[0024] Furthermore, the method for calculating the joint loss function is: using the authenticity discrimination module DI to discriminate the first generated image and the reference image respectively, obtain the authenticity score of the first generated image and the authenticity score of the reference image and use the image adversarial loss function to calculate the image adversarial loss; using the attention loss function to calculate the first attention loss according to the first attention mask when the generator generates the first generated image, and using the attention loss function to calculate the second attention loss according to the second attention mask when the generator generates the second generated image; using the AU condition matching module Dy to perform facial expression AU sequences on the first generated image and the second generated image respectively, obtain the predicted AU sequence of facial expressions of the reference image and the predicted AU sequence of facial expressions of the original image and use the conditional expression loss function to perform conditional expression loss calculation; using the identity loss function to calculate the identity loss based on the second generated image; finally, constructing the final joint loss function by linearly combining the image adversarial loss, attention loss, conditional expression loss, and identity loss:

[0025] L=L I +λ y L y +λ A (L A1 +L A2 )+λ idt L idt

[0026] Among them, λ A ,λ y and λ idt is a hyperparameter used to balance the relative importance of each loss term, L I is the image adversarial loss, L y is the conditional expression loss, L A1 is the first attention loss, L A2 is the second attention loss, L idt Loss for identity loss.

[0027] The attention loss function is:

[0028]

[0029] Among them, L A represents the attention loss, H, W represent the height and width of the image input by the generator, A i,j Represents the element value at position [i, j] in the attention mask A. The first term of the attention loss function is the total variation loss, λ is a hyperparameter, and I i represents the original image or the first generated image, P i represents the original image distribution or the first generated image distribution, Express expectations.

[0030] The method for carrying out fusion is: 1 out =(1-A)·C+A·I in , I in represents the generator input, I out Represents the generated image output by the generator.

[0031] The beneficial effects of the present invention are:

[0032] First, the generator uses depth-wise separable convolution when downsampling, which effectively reduces the number of parameters and improves the network's running speed;

[0033] Secondly, we add the collaborative-based meta-attention (CBAM) attention mechanism and total variational loss function to the generator to more effectively generate clear and natural expression images and reduce overfitting.

[0034] Finally, adding the Squeeze and Excite Network (SENet) attention mechanism to the discriminator can better capture the global information and local details of the image, thereby more effectively judging the distance between the generated image and the real image. This not only improves the discriminator's ability to distinguish, but also accelerates the convergence of network model training, effectively reduces the model's attention to redundant information, and improves the model's generalization ability.

[0035] The present invention improves the quality and details of the generated images, making the generated expressions more realistic and natural. In particular, it can still complete the task of facial expression generation with high quality even when the face is blocked (such as eyes, hair, etc.) or the expression span is large.

[0036] The present invention can expand a limited image data set to a data set sufficient for network training, greatly reducing the cost of data acquisition, providing data guarantee and key technical support for improving the accuracy of facial expression recognition technology, and has strong promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0039] Figure 2 Schematic diagram of the fusion operation in the generator of the present invention.

[0040] Figure 3 Schematic diagram of depth-wise separable convolution.

[0041] Figure 4 Schematic diagram of the collaborative-based meta-attention module (CBAM).

[0042] Figure 5 Schematic diagram of SENet structure. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0044] A facial expression data enhancement method based on GANimation improvement, such as Figure 1 As shown, the steps include:

[0045] S1: Obtain a face image dataset, where each sample in the face image dataset includes an original image I yr and reference image I yc ; Extract the action unit sequence y of the facial expression of the reference image respectively g (hereinafter referred to as AU sequence, ActionUnit,) and the AU sequence y of the facial expression of the original image r In this embodiment Figure 1 、 Figure 2 The faces in [1] are from the existing CelebA dataset.

[0046] S2: The original image I yr The AU sequence of facial expressions of the reference image is first cascaded as the input of the generator, and the generator obtains a first generated image based on the attention mask and the color mask. The generator adopts a depth-wise separable convolution module for feature extraction, which reduces the number of parameters and reduces the computational complexity. The extracted features of the depth-wise separable convolution module are enhanced by a collaborative meta-attention module, which reduces redundant information and improves the feature extraction capability.

[0047] The generator includes a generator input module, an attention generation module GA, a color generation module GI and a generator output module. The generator input module and the generator output module are respectively connected to the attention generation module GA and the color generation module GI. The generator input module is used to perform a first cascade operation or a second cascade to obtain the generator input. The attention generation module GA is used to generate a first attention mask or a second attention mask. The color generation module GI is used to generate a first color mask or a second color mask. The generator output module is used to fuse the generator input, the first attention mask and the first color mask obtained by the first cascade operation to obtain a first generated image or to fuse the generator input, the second attention mask and the second color mask obtained by the second cascade operation to obtain a second generated image.

[0048] The method for fusion acquisition and generation of images is:

[0049] like Figure 2 As shown, the given subscripts o and f represent the origin and end point of the operation, y f Represented as a set of N target expression AU sequence vectors, where the attention mask A = G A (I yo |y f )∈{0,…,1} H×W , color mask C=G I (I yo |y f )∈R H ×W×3 ,The generator generates these two masks, which are combined with the generator input and applied to the following fusion formula, focusing on the details of the face area while keeping the background unchanged to generate the target image.

[0050] The fusion formula is:

[0051] I out =(1-A)·C+A·I in

[0052] Among them, I in For I yo and y f The cascaded input, I out The generated image output by the generator.

[0053] The attention generation module GA and the color generation module GI share a depthwise separable convolution module, a collaborative meta-attention module (CBAM), and an upsampling module connected in sequence; wherein the depthwise separable convolution module is used to downsample the input image and extract multi-scale features; the collaborative meta-attention module (CBAM) is used to adaptively select channels and spatially important areas to enhance key features and obtain an enhanced feature map; the upsampling module is used to gradually increase the spatial size of the enhanced feature map and restore it to the size of the input image. In this embodiment, the upsampling module uses a deconvolution layer to perform two upsampling operations.

[0054] In traditional convolution operations, each convolution kernel needs to perform convolution operations on all channels of the input feature map, which leads to a huge number of model parameters and complex calculations. Figure 3 As shown in the figure, depthwise separable convolution decomposes the convolution process into two steps: depthwise convolution and pointwise convolution. In depthwise convolution, each input channel is convolved with a convolution kernel, and the outputs of all channels are concatenated to produce the final output feature map. Pointwise convolution, on the other hand, operates on a 1×1 convolution kernel, allowing the number of output channels to be freely adjusted as needed. In this way, depthwise separable convolution not only effectively reduces the number of model parameters but also flexibly adjusts the number of output channels, thereby reducing computational complexity and improving model efficiency. Furthermore, compared to conventional convolution, depthwise separable convolution exhibits stronger feature preservation during downsampling. Through depthwise convolution, depthwise separable convolution better captures the spatial information of the input feature map, preserving the spatial structure of the original features and helping to preserve local features in the image. The subsequent pointwise convolution integrates and adjusts this feature information to ensure that important features are retained as much as possible during downsampling. This step-by-step design enables depthwise separable convolution to more effectively preserve useful features during downsampling, thereby improving the model's representation and generalization capabilities, thereby enhancing model performance and accuracy. Assume that the convolution input feature map size is D K ×D K ×M, the convolution kernel size is D F ×D F ×M, the number is N. If ordinary convolution is used, the amount of calculation is D K ×D K ×M×N×D F ×D F , and the computational cost of DSC is D K ×D K ×M×D F ×D F +M×N×D K ×D K, comparing the two, we can get that the computational cost of DSC is twice that of ordinary convolution. It can be seen that the computational efficiency of DSC is much higher than that of ordinary convolution.

[0055] CBAM overcomes the problem of traditional channel-wise attention mechanisms ignoring spatial attention information, and can adaptively select important channels and spatial regions. CBAM is primarily composed of a Channel Attention Module (CAM) and a Spatial Attention Module (SAM). CAM analyzes the channel information of feature maps to adaptively select which channels are more important. It extracts global information from feature maps through global average pooling and global max pooling, and then generates channel weights. In the encoder part of the generator, CAM helps extract more important feature channels and reduce redundant information. In the decoder part of the generator, it helps the generator focus on feature channels related to the target expression. SAM analyzes the spatial information of feature maps to select areas of the image that require special attention. It weights the spatial positions of feature maps to highlight important areas. In the generator's attention generation module, SAM helps the generator focus on facial regions that require modification (such as the eyes and mouth) while ignoring background or other irrelevant areas. In the generator's color generation module, SAM helps the generator more precisely adjust the pixel values of the target expression area.

[0056] The implementation method of the collaborative meta-attention module (CBAM) is as follows: Figure 4 As shown, global average pooling and global maximum pooling are performed on each channel of the input feature map F to obtain the average pooling feature and max pooling features Two different channel-level statistics are obtained to provide multi-view information for subsequent channel weight calculation; average pooling features and max pooling features Input the shared MLP separately and apply the Sigmoid activation function and then add them to get the channel attention matrix M c (F), the channel attention matrix M c (F) is multiplied channel by channel with the input feature F to obtain the channel attention output feature map F1. The shared MLP consists of the fully connected layer W0 and the fully connected layer W1 connected in sequence.

[0057] The formula is:

[0058]

[0059] F1=F×M c (F)

[0060] Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, and σ(·) is the Sigmoid activation function.

[0061] The channel attention output feature map F1 is average pooled and max pooled along the channel dimension to obtain the average pooled feature of each spatial position over all channels. and max pooling features The average pooling feature and max pooling features Splice along the channel dimension, use the convolution operation with a convolution kernel size of 7×7 to convolve the spliced features, output a single channel feature map and apply the Sigmoid activation function to obtain the spatial attention matrix M S (F), the spatial attention matrix M S (F) is multiplied by the channel attention output feature map F1 in each spatial position to obtain the final enhanced feature map F2:

[0062]

[0063] F2=F1×M S (F)

[0064] The attention generation module GA adds a first convolution block after the upsampling module; the first convolution block generates an attention mask A∈[0,1] through a convolution layer. H×W , H and W represent the height and width of the input image respectively, and the output range is constrained by the Sigmoid activation function [0,1]. Attention mask A∈[0,1] H×W Indicates that the attention mask is normalized and focuses on the area in the image that needs to be modified (0 means that the original area of the input image is completely retained, and the corresponding pixel position is not modified, and the value of the input image is directly used; 1 means that the color mask area is fully used, and the corresponding pixel position is completely replaced by the value of the color mask).

[0065] The color generation module GI adds a second convolution block after the upsampling module. The second convolution block generates a transformation matrix of the RGB color channel as a color mask C∈R through the convolution layer. H×W×3 , combined with the Tanh activation function to constrain the output range [-1,1]. Color mask C∈R H×W×3 It provides pixel-level color transformation of the target expression area, which is combined with the attention mask to modify only the relevant areas and reduce redundant calculations.

[0066] S3: Compare the first generated image with the original image I yr The AU sequence of facial expressions is cascaded for the second time and input into the generator to obtain a second generated image.

[0067] In this embodiment the generator is applied twice, first mapping the original image I yr →First generated image I yg , used to calculate the original image I yr →Generate the facial expression in the reference image (the first generated image I yg ), then map the first generated image I yg →The second generated image Try to restore the first generated image and get the second generated image The second generated image is used for subsequent AU condition matching module to make judgments. With the original image I yr Compare and ensure that the first generated image and the original image I yr The faces in the image are the same person.

[0068] S4: Using the first generated image and the reference image I yc An image is input to the discriminator, and the discriminator is used to perform authenticity discrimination; the first generated image and the second generated image are used as the discriminator input images, and the discriminator is used to predict the facial expression AU sequence of the discriminator input image.

[0069] The discriminator includes a discriminator input module, an authenticity discrimination module DI, an AU condition matching module Dy and a discriminator output module; the discriminator input module and the discriminator output module are respectively connected to the authenticity discrimination module and the AU condition matching module; the discriminator input module is used to receive the discriminator input image, including the first generated image I yg , reference image I yc and the second generated image; the authenticity discrimination module DI is used to discriminate the authenticity of the image according to the input image and output the authenticity score; the AU condition matching module Dy is used to predict the AU sequence of the image facial expression according to the input image.

[0070] The authenticity discrimination module DI adopts the PatchGAN architecture, but adds a SENet module, which can use channel attention to strengthen key features, suppress redundant information, improve the accuracy of local detail discrimination, guide the generator to output more realistic images and enhance generalization capabilities. The authenticity discrimination module DI and the AU condition matching module Dy share the sequentially connected convolution block I, convolution block II, convolution block III and SENet module. Convolution block I, convolution block II and convolution block III are used to extract general image features, and the SENet module is used to enhance features. The authenticity discrimination module DI adds a sequentially connected convolution block IV, a global average pooling layer I and a linear layer after the SENet module. The convolution block IV is used to focus on higher-level image features. The global average pooling layer I fuses the global information of each channel of the feature map output by the convolution block IV. The linear layer outputs a classification score based on the global information. The linear layer here does not use the Softmax function. The AU condition matching module Dy adds a sequentially connected convolution block V, a global average pooling layer II and a first fully connected layer (Dropout is 0.5) and a second fully connected layer after the SENet module. The convolution block V is used to extract features of facial expression-related areas (such as eyes and mouth), focusing on the enhancement of expression-related features. The global average pooling II and two fully connected layers are used to output facial expression AU vectors.

[0071] like Figure 5 As shown, the calculation process of the SENet module is:

[0072] Squeeze operation: Since convolution is performed in a local space, it is difficult to obtain global features at the channel level, which requires the Squeeze operation to perform global average pooling on the feature map. Perform global average pooling on each channel of to generate a channel-level global descriptor z∈R C , all channels are described by a value z. C is the number of channels, H and W are the height and width of the feature map respectively, and the global description of channel c is calculated as:

[0073]

[0074] Among them, u c (i, j) represents the eigenvalue of position (i, j) in channel c, z c is an element in the global descriptor z.

[0075] Excitation operation: The global descriptor z is input into the fully connected layer, the global feature information is mapped into a weight matrix, and the weight matrix is normalized by the activation function to obtain the attention weight matrix s. The attention weight matrix s is used to multiply the feature map u channel by channel to obtain the weighted feature map x:

[0076] s=σ(W3δ(W2z))

[0077] x c =s c u c

[0078] Among them, x c is the cth channel of the weighted feature map x, s c is the cth channel of the attention weight matrix s, u c is the cth channel of the feature map, W2 and W3 are two fully connected layers, and δ(·) is the ReLU activation function. After SENet is embedded in the convolutional layer of the discriminator module, especially during the feature extraction stage, it dynamically adjusts global information to enhance the discriminator's ability to capture global image information and local details. It also uses squeeze and excitation operations to generate channel weights and apply them to the original feature map, optimizing the feature extraction process.

[0079] S5: Calculate the joint loss function based on the discriminator's discrimination result, the discriminator's prediction result, the second generated image, and the attention mask, and use backpropagation to update the generator and discriminator network parameters based on the joint loss function.

[0080] The joint loss function includes image adversarial loss, attention loss, conditional expression loss, and identity loss. Total variation loss is introduced into the attention loss to enhance the control of image smoothness. The application of total variation loss in image processing is primarily reflected in the control of image smoothness. It is a loss function that penalizes high-frequency noise and details in an image. During image generation, the model may over-focus on noise or details in the training data, resulting in an image that is not smooth enough. By penalizing the image gradient, total variation loss forces the generated image to be more spatially flat, avoiding overly complex textures and noise. This helps generate more realistic, clear, and visually pleasing images.

[0081] The method for calculating the joint loss function is: using the authenticity discrimination module DI to respectively calculate the first generated image and the reference image I yc Perform discrimination to obtain the authenticity score of the first generated image and the reference image I yc The authenticity score is calculated using the image adversarial loss function; the first attention loss L is calculated using the attention loss function according to the first attention mask when the generator generates the first generated image A1 , the second attention loss L is calculated using the attention loss function according to the second attention mask when the generator generates the second generated image A2; Use the AU condition matching module Dy to perform facial expression AU sequences on the first generated image and the second generated image respectively, obtain the predicted AU sequence of facial expressions of the reference image and the predicted AU sequence of facial expressions of the original image and use the conditional expression loss function to perform conditional expression loss calculation; use the identity loss function based on the second generated image to calculate the identity loss; finally, construct the final joint loss function by linearly combining the image adversarial loss, attention loss, conditional expression loss, and identity loss.

[0082] The image adversarial loss is:

[0083]

[0084] Among them, G is the generator, G(I yr |y g ) indicates that the generator is based on the original image I yr and the AU sequence y of facial expressions of the reference image g The first generated image I yg ;D I represents the authenticity discrimination module, D I (·) is the authenticity discrimination module D I The authenticity discrimination operation, output score, I yc is the reference image, E *～* represents the expected value, P r is the original image distribution, P c is the reference image distribution. gp is the gradient penalty coefficient, which is used to control the intensity of the gradient penalty; It represents the original image I yr and generate the image G(I yr |y g ), is a random interpolation image distribution, obtained by linear interpolation between the real image and the generated image; Denotes the calculation discriminator D I In the interpolated image The gradient on Represents the gradient penalty term, which is used to ensure that the L2 norm of the gradient is close to 1 to prevent the gradient from exploding or disappearing.

[0085] Image loss function L I It is used to make the distribution of generated images tend to the distribution of reference images. This is modified based on WGAN. Because the original GAN training is difficult and it is easy to cause gradient explosion or disappearance, the image loss L is used. IIt is to maximize the score of the generated image through the discriminator and minimize the score of the input image through the discriminator, by adding λ gp This gradient penalty coefficient makes the distribution of generated images closer to the distribution of reference images.

[0086] The attention loss includes the first attention loss L A1 and the second attention loss L A2 , the first attention loss L A1 for:

[0087]

[0088] Among them, H, W represent the length and width of the input image, λ is a hyperparameter used to balance the importance of different loss terms, and A i,j is the value of the i-th row and j-th column in the matrix A (attention mask); is the total variation loss (TV loss), which is used to measure the smoothness of the attention mask A. is the L2 penalty term. This part of the loss is used to regularize the attention mask A to prevent its value from being too large, thereby avoiding the saturation of the attention mask approaching 1.

[0089] First attention loss L A1 According to the original image I yr and the AU sequence y of facial expressions of the reference image g The attention loss calculated after inputting the generator is calculated, and the second attention loss L A1 With the first attention loss L A1 Compared to the first generated image I yg and the AU sequence y of facial expressions of the original image r The attention loss calculated after inputting the generator is calculated. The method is the same and will not be repeated here.

[0090] The conditional expression loss is:

[0091]

[0092] Where Dy is the AU condition matching module, and Dy(·) is the facial expression AU sequence prediction result output by the AU condition matching module; Represents the first generated image I yg (Right now )’s facial expression AU sequence prediction results, Represents the original image I yr The prediction results of facial expression AU sequence, By minimizing the first loss in the formula, the generator G can be optimized so that the generated image can better match the reference image. By minimizing the second loss in the formula, the restored second generated image is guaranteed. With the original image I yr The consistency of faces.

[0093] The identity loss is:

[0094]

[0095] in, Represents the second generated image It represents the generator generating image I according to the first yg and the original image facial expression AU sequence y r The generated cyclic image is then computed. The L1 norm of the cyclically generated image and the original image is calculated to measure whether the face in the first generated image is the same person as the original image. The identity loss makes the generated image output by the generator closer to the input image, preserving identity information and preventing redundant modifications.

[0096] Finally, the final joint loss function is constructed by linearly combining the image adversarial loss, attention loss, conditional expression loss, and identity loss:

[0097] L=L I +λ y L y +λ A (L A1 +L A2 )+λ idt L idt

[0098] Where: A ,λ y and λ idt is a hyperparameter that controls the behavior and performance of the model and balances the relative importance of each loss term.

[0099] S6: Repeat steps S2 to S5 until the stopping condition is met (the preset training round is reached), and use the generator of the network model with the optimal parameters to generate the target facial expression image.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A facial expression data enhancement method based on GANimation improvement, characterized in that: Including steps: S1: Acquire a facial image dataset, wherein each sample in the facial image dataset includes an original image and a reference image; extract an AU sequence of facial expressions of the reference image and an AU sequence of facial expressions of the original image respectively; S2: Concatenating the AU sequences of facial expressions of the original image and the reference image for the first time as input to the generator, and the generator obtains a first generated image based on the attention mask and the color mask; S3: Concatenate the first generated image and the AU sequence of facial expressions of the original image for a second time and input the concatenated image into the generator to obtain a second generated image; S4: Using the first generated image and the reference image as input images for the discriminator, and using the discriminator to perform authenticity judgment; The first generated image and the second generated image are used as input images of the discriminator, and the discriminator is used to predict the facial expression AU sequence of the input image; S5: Calculate the joint loss function based on the discriminator's discrimination result, the discriminator's prediction result, the second generated image, and the attention mask, and use backpropagation to update the generator and discriminator network parameters according to the joint loss function; S6: Repeat steps S2 to S5 until the stopping condition is met, and use the generator of the network model with the optimal parameters to generate the target facial expression image.

2. The facial expression data enhancement method based on GANimation improvement according to claim 1, characterized in that: The generator includes a generator input module, an attention generation module GA, a color generation module GI and a generator output module. The generator input module and the generator output module are respectively connected to the attention generation module GA and the color generation module GI. The generator input module is used to perform a cascade operation to obtain the generator input, the attention generation module GA is used to generate an attention mask A, and the color generation module GI is used to generate a color mask C. The generator output module is used to fuse the generator input, attention mask and color mask obtained by the cascade operation to obtain the generated image of the generator output.

3. The facial expression data enhancement method based on GANimation improvement according to claim 2, characterized in that: The attention generation module GA and the color generation module GI share a depth-separable convolution module, a collaboration-based meta-attention module and an upsampling module connected in sequence, and the depth-separable convolution module is used to extract multi-scale features according to the input image; The collaborative meta-attention module is used to adaptively select channels and spatially important areas to enhance key features and obtain enhanced feature maps; the upsampling module is used to gradually increase the spatial size of the enhanced feature maps and restore them to the size of the input image.

4. The method for enhancing facial expression data based on GANimation improvement according to claim 3, characterized in that: The attention generation module GA adds a first convolution block after the upsampling module; the first convolution block generates an attention mask A through a convolution layer and constrains the output range in combination with a Sigmoid activation function; The color generation module GI adds a second convolution block after the upsampling module. The second convolution block generates a transformation matrix of the RGB color channels as a color mask C through a convolution layer, and combines the Tanh activation function to constrain the output range.

5. The method for enhancing facial expression data based on GANimation improvement according to any one of claims 1 to 4, characterized in that: The discriminator includes a discriminator input module, an authenticity discrimination module DI, an AU condition matching module Dy, and a discriminator output module; the discriminator input module and the discriminator output module are respectively connected to the authenticity discrimination module and the AU condition matching module; the discriminator input module is used to receive the discriminator input image; the authenticity discrimination module DI is used to discriminate the authenticity of the image based on the discriminator input image and output an authenticity score; The AU condition matching module Dy is used to predict the AU sequence of facial expressions in the image based on the discriminator input image.

6. The method for enhancing facial expression data based on GANimation improvement according to claim 5, characterized in that: The authenticity discrimination module DI and the AU condition matching module Dy share the sequentially connected convolution block I, convolution block II, convolution block III and SENet module. The convolution block I, convolution block II and convolution block III are used to extract general image features, and the SENet module is used to perform feature enhancement. The authenticity discrimination module DI adds a sequentially connected convolution block IV, a global average pooling layer I and a linear layer after the SENet module. The convolution block IV is used to focus on higher-level image features, the global average pooling layer I is used to fuse the global information of each channel of the feature map output by the convolution block IV, and the linear layer outputs the classification score according to the global information. The AU condition matching module Dy adds a sequentially connected convolution block V and the authenticity discrimination module DI output layer after the SENet module. The convolution block V is used to extract features of facial expression-related areas. The authenticity discrimination module DI output layer outputs the facial expression AU vector through the sequentially connected global average pooling layer II, the first fully connected layer and the second fully connected layer.

7. The method for enhancing facial expression data based on GANimation improvement according to any one of claims 1 to 4 or 6, characterized in that: The joint loss function includes image adversarial loss, attention loss, conditional expression loss and identity loss. The total variation loss is introduced into the attention loss to enhance the control of image smoothness.

8. The method for enhancing facial expression data based on GANimation improvement according to claim 7, characterized in that: The method for calculating the joint loss function is: using the authenticity discrimination module DI to discriminate the first generated image and the reference image respectively, obtain the authenticity score of the first generated image and the authenticity score of the reference image and use the image adversarial loss function to calculate the image adversarial loss; using the attention loss function to calculate the first attention loss according to the first attention mask when the generator generates the first generated image, and using the attention loss function to calculate the second attention loss according to the second attention mask when the generator generates the second generated image; using the AU condition matching module Dy to perform facial expression AU sequences on the first generated image and the second generated image respectively, obtain the predicted AU sequence of facial expression of the reference image and the predicted AU sequence of facial expression of the original image and use the conditional expression loss function to perform conditional expression loss calculation; using the identity loss function to calculate the identity loss based on the second generated image; finally, the final joint loss function is constructed by linearly combining the image adversarial loss, attention loss, conditional expression loss, and identity loss: L=L I +λ y L y +λ A (L A1 +L A2 )+λ idt L idt Among them, λ A ,λ y and λ idt is a hyperparameter used to balance the relative importance of each loss term, L I is the image adversarial loss, L y is the conditional expression loss, L A1 is the first attention loss, L A2 is the second attention loss, L idt Loss for identity loss.

9. The method for enhancing facial expression data based on GANimation improvement according to claim 8, characterized in that: The attention loss function is: Among them, L A represents the attention loss, H, W represent the height and width of the image input by the generator, A i,j Represents the element value at position [i, j] in the attention mask A. The first term of the attention loss function is the total variation loss, λ is a hyperparameter, and I i represents the original image or the first generated image, P i represents the original image distribution or the first generated image distribution, Express expectations.

10. The method for enhancing facial expression data based on GANimation improvement according to any one of claims 2 to 4, characterized in that: The method for carrying out fusion is: 1 out =(1-A)·C+A·I in , I in represents the generator input, I out Represents the generated image output by the generator.

Images