Unconstrained facial expression synthesis and recognition method and system

Abstract

The application provides an unconstrained face expression synthesis and recognition method and system, comprising: an expression synthesis step: capturing unconstrained style attributes in an unconstrained face dataset image, realizing conditional mapping from a face expression image to a group of style conversion images; a content consistency step: when the face expression image is subjected to style conversion, the content information in the face expression image is kept unchanged, and content consistency is ensured; and an expression recognition step: performing expression classification on the generated style conversion image, feeding the loss generated in the classification to the generator, and then the generator no longer performs blind data enhancement, so as to synthesize images more conducive to expression recognition performance improvement.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for unconstrained facial expression synthesis and recognition. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Facial expression recognition is an important branch of computer vision, playing a crucial role in applications such as medical diagnosis, fatigue monitoring, and human-computer interaction. However, facial expression recognition in natural environments remains a significant challenge because the appearance of a face is highly susceptible to factors such as lighting, occlusion, and unconstrained imaging conditions (e.g., sensor noise, low resolution, and motion blur).

[0004] Traditional facial expression recognition methods mainly rely on designing suitable handcrafted features, such as Local Binary Pattern (LBP), Histogram of Gradients (HOG), and Scale-invariant Feature Transform (SIFT). However, these features are difficult to overcome the interference caused by changes in facial appearance and cannot achieve satisfactory expression recognition performance in natural environments.

[0005] In recent years, deep learning-based methods have achieved remarkable results in the field of computer vision, mainly due to the powerful feature representation capabilities of deep neural networks and the availability of large-scale training data. However, most existing facial expression datasets are relatively small. For example, the Natural Environments dataset (SFEW) contains only 1394 images, which cannot cover the natural variations caused by various unconstrained imaging environmental factors. Furthermore, its data volume is insufficient to support the training of deep neural networks, making it prone to overfitting and hindering the achievement of good facial expression recognition performance.

[0006] One solution to the above problems is to construct a large-scale, unconstrained facial expression dataset. However, collecting a dataset containing tens or even hundreds of thousands of identities is extremely difficult, raising both privacy and labor costs. Another solution is to develop a facial expression image generation model to generate facial expression images with the desired attributes and use these images to augment the expression dataset. In recent years, image synthesis methods based on generative adversarial networks (GANs) have achieved remarkable results in areas such as face synthesis, image translation, and semantic manipulation. Some researchers have proposed using GAN-based methods to design facial expression synthesis models that separate semantic information from the face and control explicit facial attributes (pose, expression, hair color, and skin color, etc.) to generate realistic, high-quality expression images.

[0007] Zhang et al. proposed using facial geometric information to guide the synthesis of facial expression images. It transmits geometric information through a set of facial landmarks, thus clearly separating facial identity features from expression and pose features. This allows for the synthesis of new expression images with different poses and expressions while preserving identity information. StarGAN treats multiple expressions as multiple domains and can perform image-to-image transformations across multiple domains using only a single model, allowing for changes in facial expression categories without altering identity information. Zhang et al. encoded different expressions in a disentangled manner in the latent space, swapping some features in the latent space to exchange the expressions of two facial images, resulting in highly realistic synthesized facial expression images.

[0008] These methods can control explicit facial attributes, thereby generating additional data for the expression training dataset. However, these expression synthesis methods focus on high-quality facial expression synthesis, modifying explicit facial attributes such as pose and expression to synthesize highly realistic facial expression images. But for facial expression recognition in real-life applications, the input facial expression images are often unconstrained, potentially with low resolution, inherent sensor noise, and motion blur, resulting in generally low image quality. Using only these high-quality images to augment the dataset cannot encompass the various natural variations caused by these unconstrained factors, and the expression recognition model trained on such an augmented dataset will not be suitable for complex real-world environments. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, the present invention provides an unconstrained facial expression synthesis and recognition method, which can improve the facial expression recognition performance in natural environments.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0011] Firstly, a method for unconstrained facial expression synthesis and recognition is disclosed, including:

[0012] Expression synthesis steps: Capture unconstrained style attributes from unconstrained face dataset images to achieve conditional mapping from a single facial expression image to a set of style-transferred images;

[0013] Content consistency steps: When performing style transfer on facial expression images, keep the content information in the facial expression images unchanged to ensure content consistency;

[0014] Expression recognition steps: The generated style-transformed image is classified into expressions. The loss from the classification is fed back to the generator. Then, the generator no longer performs blind data augmentation, so as to synthesize images that are more conducive to improving expression recognition performance.

[0015] As a further technical solution, the specific implementation method of the expression synthesis step is as follows:

[0016] After inputting a facial expression image into the generator, the style vector is injected into the generator through adaptive instance normalization to obtain a synthesized expression image.

[0017] The discriminator distinguishes between synthesized facial expression images and unconstrained face images, determining whether they are real or fake. The generator aims to generate facial expression images that can fool the discriminator and be identified as real, while the discriminator aims to distinguish the facial expression images and identify them as fake.

[0018] As a further technical solution, the style vector is used to implement conditional mapping from one image to a set of style-transformed images.

[0019] As a further technical solution, the content consistency step is specifically implemented as follows:

[0020] A style encoder is used to extract style vectors from the input facial expression image and the unconstrained facial image, respectively. The extracted style vectors are then injected into the generator to obtain the reconstructed image.

[0021] As a further technical solution, during the process of obtaining the reconstructed image, reconstruction loss is used to ensure the consistency of the image content.

[0022] The reconstruction loss consists of the perception loss and the mean squared error.

[0023] Perceptual loss uses features extracted from the VGG model to measure the similarity between images.

[0024] As a further technical solution, the discriminator D is utilized. z Make the distribution of Eu match the prior distribution N 0,1;

[0025] The purpose of style vectors is to change the style of facial expression images, ensuring that the synthesized image has similar unconstrained style information to the unconstrained facial image;

[0026] Eu is a style vector extracted from an unconstrained face image.

[0027] As a further technical solution, the facial expression recognition step employs a typical cross-entropy loss function.

[0028] Secondly, an unconstrained facial expression synthesis and recognition system was disclosed, including:

[0029] Expression synthesis module: Captures unconstrained style attributes in unconstrained face dataset images, and realizes conditional mapping from a facial expression image to a set of style-transfer images;

[0030] Content consistency module: When performing style transfer on facial expression images, it ensures that the content information in the facial expression images is not changed, thus guaranteeing content consistency;

[0031] The facial expression recognition module classifies the generated style-transformed images into facial expressions. The loss from the classification is fed back to the generator, which then stops blindly performing data augmentation in order to synthesize images that are more conducive to improving facial expression recognition performance.

[0032] The above one or more technical solutions have the following beneficial effects:

[0033] This invention employs unconstrained facial expression synthesis, focusing on the unconstrained style of images rather than pursuing high quality and realism. Specifically, given a small facial expression dataset X and an additional large-scale unconstrained face dataset U, an unconstrained facial expression synthesis model is trained to discover the unconstrained style attributes of dataset U and incorporate these attributes into dataset X. During generative adversarial training, the unconstrained facial expression synthesis model attempts to capture the attributes of dataset U in its style latent space. After introducing a set of style vectors z ~ N 0,1 as conditions, the facial expression images in dataset X undergo a conditional mapping from one image to a set of style-transformed images, ultimately synthesizing new facial expression images with unconstrained style.

[0034] Simultaneously, a facial expression recognition model is embedded after the facial expression synthesis model to perform end-to-end facial expression synthesis and recognition. The expression synthesis model synthesizes new expression images to augment the dataset, thereby improving the performance of the expression recognition model; the expression recognition model feeds its recognition results back to the expression synthesis model, so that the expression synthesis model no longer blindly performs data augmentation and can synthesize images that are more conducive to improving expression recognition performance. The free parameters in the two models can adapt and cooperate through task-specific losses to promote the performance improvement of each other.

[0035] This invention uses an end-to-end architecture to combine the facial expression synthesis task and the facial expression recognition task, performing them simultaneously in a single model. The two tasks collaborate with each other to improve performance.

[0036] This invention synthesizes unconstrained facial expression images, ensuring that the content information remains unchanged during image synthesis, and classifies and identifies the synthesized images. The synthesized images possess an unconstrained style, encompassing various natural variations caused by unconstrained imaging factors, including different levels of noise, resolution, and motion blur.

[0037] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0038] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0039] Figure 1 This is a schematic diagram illustrating unconstrained facial expression synthesis and recognition according to an embodiment of the present invention; x represents a facial expression dataset image, u represents an unconstrained facial dataset image. G represents the generator, E represents the style encoder, Dimg is the image discriminator, Dz is the style discriminator, and FER represents the facial expression recognition module.

[0040] Figure 2 Images from the RAF-DB dataset, as described in this embodiment of the invention;

[0041] Figure 3 Images from the WiderFace dataset, as described in this embodiment of the invention;

[0042] Figure 4 Synthesized facial expression images; the first column is the image from the facial expression dataset X, and the other columns are the images generated under the influence of style vector z ~ N0,1, where the style vector is the same in each column.

[0043] Figure 5The synthesized facial expression images from the ablation experiment. (a) refers to the image in the facial expression dataset X; (b) is the image generated without using style vectors, denoted by NO_Style; (c) is the image generated without using reconstruction loss and discriminator D. z The image generated at that time is denoted by NO_Rec+Dz; (d) is the image generated without using the discriminator D. z (e) is the image generated when no end-to-end structure is used, denoted by NO_Dz; (f) is the image generated by the method in this paper. Detailed Implementation

[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0046] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0047] Example 1

[0048] This embodiment discloses an unconstrained facial expression synthesis and recognition method, such as... Figure 1 As shown:

[0049] Create two datasets:

[0050] Dataset X = x1, y1, x2, y2, ..., x i ,y i ,…,x n ,y n It is a facial expression dataset, where y i It is x i Facial expression tags. The facial expression dataset X lacks various natural variations caused by unconstrained imaging environmental factors.

[0051] Unconstrained face dataset U = u1, u2, ..., u i ,…,u n X is a large dataset containing a variety of unconstrained imaging factors. It covers natural variations caused by various unconstrained factors, and this paper aims to introduce these natural variations into dataset X.

[0052] Facial expression synthesis steps:

[0053] like Figure 1As shown in (a), after inputting a facial expression image x into the generator G, the style vector z ~ N 0,1 is injected into the generator G through adaptive instance normalization (AdaIN) to obtain the synthesized expression image G x,z. The style vector z serves as a condition here, realizing a conditional mapping from one image x to a set of style-transformed images G x,z. Discriminator D img It distinguishes between Gx,z and unconstrained face images u, determining whether they are real or fake. This creates an adversarial process: the generator G expects to generate Gx,z that can fool the discriminator D. img D img The discriminant is determined to be true; and the discriminator D img The goal is to distinguish Gx and Zz and classify them as false. The adversarial loss function is:

[0054]

[0055]

[0056] Where x represents a facial expression image, X represents a facial expression dataset, u represents an unconstrained facial image, U represents an unconstrained facial dataset, and D img G represents the discriminator, G represents the generator, and z represents the style vector.

[0057] Content consistency processing steps:

[0058] Generator G and discriminator D img Adversarial learning between the two sides ensures that the style information of the synthesized image Gx,z is altered. When the adversarial training reaches equilibrium, D... img It is assumed that the data distribution of the synthesized image Gx,z matches the data distribution of the unconstrained face image u. However, this raises a question: after the style vector z is injected into the generator G, does it only change the style information of the expression image x, and is the content information such as the face identity and expression in the expression image x completely preserved? To solve the above problem, this example adds a style encoder E and a second discriminator Dz, as follows: Figure 1 As shown in (b) and (c) in the figure.

[0059] exist Figure 1In (b) and (c), the style encoder E extracts style vectors Ex and Eu from the input facial expression image x and unconstrained face image u, respectively. Then, Ex and Eu are injected into the generator G to obtain the reconstructed images Gx,Ex and Gu,Eu. To ensure the consistency of image content, this example uses a reconstruction loss. The reconstruction loss consists of perceptual loss (PL) and mean squared error (MSE). Here, the perceptual loss uses features extracted from the VGG model to measure the similarity between images; the perceptual loss function is:

[0060]

[0061] Where F refers to the VGG feature extraction model. Here, x represents a facial expression image, X represents a facial expression dataset, u represents an unconstrained facial image, U represents an unconstrained facial dataset, G represents the generator, F represents the VGG feature extraction model, and E represents the style encoder.

[0062] The mean squared error loss function is:

[0063]

[0064] Where x represents a facial expression image, X represents a facial expression dataset, u represents an unconstrained facial image, U represents an unconstrained facial dataset, G represents a generator, and E represents a style encoder.

[0065] exist Figure 1 In (c), besides the style encoder E, there is also a discriminator Dz, whose function is to ensure that the distribution of Eu matches the prior distribution N0,1. The style vector z ~ N0,1 modifies the style of the facial expression image x, ensuring that the synthesized image Gx,z has similar unconstrained style information to the unconstrained face image u. Eu is precisely the style vector extracted from the unconstrained face image u. If the distribution of the style vector z and the distribution of Eu do not match, it cannot be guaranteed that the synthesized image Gx,z using the style vector z has a similar unconstrained style to the unconstrained face image u.

[0066] This is the second adversarial process in the unconstrained facial expression synthesis and recognition model. The adversarial loss function is as follows:

[0067]

[0068]

[0069] Where u represents an unconstrained face image, U represents an unconstrained face dataset, Dz represents a style discriminator, E represents a style encoder, and z represents a style vector.

[0070] Facial expression recognition steps:

[0071] The facial expression recognition model FER is used to classify facial expression images. In an end-to-end system, the loss from classification is used not only to optimize the parameters of the FER network model but also fed back to the generator G. The facial expression recognition model FER employs a typical cross-entropy loss function (CEL):

[0072]

[0073] Where x represents a facial expression image, y is the expression label of expression image x, X represents the facial expression dataset, z represents the style vector, G represents the generator, and FER represents the expression recognition model.

[0074] In summary, for the generator G, style encoder E, and expression recognition model FER, the overall objective loss function is:

[0075]

[0076] Among them, λ1, λ2, λ3, λ4 and λ5 are weight parameters.

[0077] Unconstrained facial expression synthesis focuses on the unconstrained style of images rather than pursuing high quality and realism. The resulting synthesized facial expression images possess an unconstrained style, encompassing more natural variations and thus improving facial expression recognition performance in natural environments.

[0078] Based on end-to-end facial expression synthesis and recognition, expression synthesis can expand the dataset and improve expression recognition performance, while expression recognition can in turn guide expression synthesis. The two tasks are combined to adapt and cooperate, and improve each other's performance.

[0079] Example 2

[0080] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0081] Example 3

[0082] The purpose of this embodiment is to provide a computer-readable storage medium.

[0083] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0084] Example 4

[0085] The purpose of this embodiment is to provide an unconstrained facial expression synthesis and recognition system, including:

[0086] Expression synthesis module: Captures unconstrained style attributes in unconstrained face dataset images, and realizes conditional mapping from a facial expression image to a set of style-transfer images;

[0087] Content consistency module: When performing style transfer on facial expression images, it ensures that the content information in the facial expression images is not changed, thus guaranteeing content consistency;

[0088] The facial expression recognition module classifies the generated style-transformed images into facial expressions. The loss from the classification is fed back to the generator, which then stops blindly performing data augmentation in order to synthesize images that are more conducive to improving facial expression recognition performance.

[0089] Specifically, the facial expression synthesis module consists of a generator G and a discriminator D. img The main function of the component is to capture the unconstrained style attributes of images u in an unconstrained face dataset, enabling conditional mapping from one image x to a set of style-transformed images G x,z. The content consistency module consists of a style encoder E and a discriminator D. z The goal of this approach is to maintain the integrity of content information in a facial expression image x during style transfer, ensuring consistency. The expression recognition module consists of the expression recognition model FER, which classifies the generated expression images Gx and z. The loss from this classification is fed back to the generator G, which then avoids blind data augmentation and synthesizes images that better enhance expression recognition performance. The unconstrained facial expression synthesis and recognition process is detailed below based on these three modules.

[0090] experiment

[0091] Dataset

[0092] The experiment requires two types of datasets: a facial expression dataset X and an unconstrained face dataset U. The facial expression dataset X uses the RAF-DB dataset, while the unconstrained face dataset U uses the WiderFace dataset.

[0093] (1) RAF-DB dataset

[0094] The RAF-DB dataset is a dataset of facial expressions in natural environments, containing a total of 29,672 images, labeled using basic or compound expressions, such as... Figure 2As shown in the figure, 15,339 images were labeled with seven basic facial expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral. In the experiments presented in this paper, images labeled with these seven basic facial expressions were used, with 12,271 images used for training and 3,068 images used for testing.

[0095] (2) WiderFace dataset

[0096] The WiderFace dataset is a face detection dataset collected by retrieving images with different event keywords from search engines. It contains a more diverse set of faces, with face images containing various levels of noise and resolution, such as... Figure 3 As shown.

[0097] Many faces in the WiderFace dataset still cannot be detected by advanced detection methods. This example only retains detectable faces as training data, approximately 70,000 face images.

[0098] Application example: First, preprocess the facial expression image and resize it to 112×112; then, deploy the model on a machine with a GPU, install the PyTorch toolbox, and input the preprocessed image into the model for training; finally, after training, you can obtain the synthesized expression image and the expression category prediction for the input image.

[0099] Experimental setup

[0100] (1) Network Structure

[0101] The facial expression recognition (FER) module uses a ResNet18 network, but the number of prediction layers in ResNet18 is replaced with the number of facial expression categories. FER is first pre-trained on ImageNet, and then the pre-trained FER is embedded into the facial expression synthesis module.

[0102] The facial expression synthesis module involves a generator G and a discriminator D in its network model. img The content consistency module involves network models including a style encoder (E) and a discriminator (D). z To preserve the geometric structure of the image, the generator G in this example does not involve any downsampling or upsampling. Discriminator D img Discriminator D z The network model structure of style encoder E is shown in Table 1.

[0103] Table 1 Network Structure

[0104]

[0105] K represents the size of the convolution kernel, F represents the number of filters, S represents the stride, and FC represents a fully connected layer.

[0106] (2) Training process

[0107] The method proposed in this example is implemented using PyTorch v1.6. The PyTorch toolbox provides various pre-constructed CNN basic network models and pre-trained network model parameters based on the ImageNet dataset, allowing for direct model loading. This scheme directly loads the ResNet18 model from the PyTorch toolbox as the expression recognition module (FER). Before training begins, an expression image containing 68 facial landmarks is cropped using a face detection algorithm and resized to 112×112. In equation (2-8), the weight parameters λ1=1,λ2=10,λ3=10,λ4=1,λ5=0.2. During model training, the Adam optimizer is used with β1=0.5, β2=0.99, and a learning rate of 0.0001. The model is trained for a total of 150,000 steps, with a mini-batch size of 16.

[0108] Experimental results

[0109] (1) Results of facial expression synthesis

[0110] The result of facial expression synthesis is as follows Figure 4 As shown, the model proposed in this embodiment can synthesize new facial expression images with different blur and lighting styles. Under the influence of different style vectors z ~ N 0,1, the synthesized images have different noise and resolution, encompassing more natural variations and significantly increasing the richness of the facial expression dataset X. Figure 4 In this study, the style of the synthesized images is clearly unconstrained, but the content information is completely preserved; expression, pose, and identity remain unchanged. These synthesized images are neither high-quality nor realistic, but they add many unconstrained imaging environment factors to the facial expression dataset X, thereby making the expression recognition model trained on the dataset X more adaptable to unconstrained imaging environments and achieving better expression recognition performance. Figure 4 In this embodiment, the synthesized image achieves the goal of unconstrained facial expression synthesis.

[0111] (2) Expression recognition results

[0112] The expression recognition results of the proposed model are compared with those of advanced facial expression recognition algorithms, and the results are shown in Table 2. For ResNet18 and VGG16, pre-training is performed on ImageNet first, followed by fine-tuning on the RAF-DB dataset to obtain their expression recognition accuracy. For StarGAN, this approach first trains StarGAN to generate new expression images, then uses these images to augment the expression dataset, and finally uses the augmented expression dataset to train ResNet18 to obtain the expression recognition accuracy.

[0113] As shown in Table 2, the proposed model achieved the highest accuracy of 87.35%, which is 6.91% higher than the VGG16 model and 2.77% higher than ResNet18. Compared with other advanced facial expression recognition algorithms, the proposed method outperforms by 0.39%-4.72%, fully demonstrating its superiority. Compared with StarGAN, the proposed method outperforms by 2.47%, indicating that the model has better expression recognition performance.

[0114] Table 2. Facial Recognition Results

[0115]

[0116]

[0117] ablation experiment

[0118] To verify the effectiveness of each module of the proposed model, ablation experiments were conducted. NO_Style indicates that the style style vector z is not used. Since the style encoder E is used to extract the style style vector Eu from the image, the discriminator D... z It is used to determine whether the distribution of E u matches the style vector z. Therefore, when the style latent vector z is not used, the style encoder E and the discriminator D... z It is no longer used. NO_Rec+Dz indicates that the reconstruction loss and discriminator D are not used. z NO_Dz indicates that the discriminator D is not used. z NO_End_to_End indicates that an end-to-end structure is no longer used, and the two tasks of expression synthesis and expression recognition are performed separately. In the NO_End_to_End experiment, the synthesized expression images are first augmented with a facial expression dataset, and then the augmented dataset is used to train the expression recognition module FER.

[0119] (1) Expression synthesis results

[0120] Facial expression synthesis images from ablation experiments, such as Figure 5 As shown. In Figure 5In the image, (a) is the input facial expression image, and (b) is the synthesized image without using the style vector z. In (b), due to the lack of style vector z injection, for each input facial expression image, only a qualitatively blurred image can be output, which is not as good as... Figure 4 It can output multiple images with different noise and resolution.

[0121] Figure 5 Examples (c) and (d) are for verifying the function of the content consistency module, where (c) and (d) use the same style vector z. In (c), no reconstruction loss or discriminator D is used. z As can be seen, the synthesized image exhibits clear artifacts. In (d), a reconstruction loss was used instead of a discriminator D. z However, the synthesized image still contains noticeable artifacts. This problem arises because, without using the discriminator D... z At that time, the style vector Eu obtained by the style encoder E does not match the prior distribution N0,I. This will cause the style vector z ~ N0,I to not only change the style information of the input image after being injected into the generator G, but also change the content information of the image, thus resulting in artifacts.

[0122] Figure 5 In the diagram, (e) represents the image synthesized without an end-to-end structure, and (f) represents the image synthesized using the full model. Both types of images use the same style vector z and show no obvious artifacts. The comparison between (e) and (f) shows that even with the same style vector z, the facial expression images synthesized by the two models are significantly different. This indicates that in the end-to-end structure, the expression recognition module FER guides the synthesis model, resulting in different images synthesized by the two models after training.

[0123] (2) Expression recognition results

[0124] The facial expression recognition results from the ablation experiment are shown in Table 3. The facial expression recognition accuracy obtained by NO_Style was 84.71%, which is 2.64% lower than the method in this study, and only 0.13% higher than ResNet18. From... Figure 5 (b) As can be seen, without the help of the style vector z, only qualitative blurry images can be synthesized. These synthesized images have a very simple style and cannot cover a variety of noise and lighting conditions, so the improvement on the expression recognition performance is very limited.

[0125] The accuracies of NO_Rec+Dz and NO_Dz were 85.50% and 85.72%, respectively, representing reductions of 1.85% and 1.63% compared to the method presented in this paper. Both NO_Rec+Dz and NO_Dz synthesized images exhibited significant artifacts (such as...). Figure 5 (c) and (d)), and the expression recognition results also show the decline in expression recognition performance, which shows the damage of image artifacts to expression recognition performance and the importance of the content consistency module in the model proposed in this scheme.

[0126] The NO_End_to_End approach does not use an end-to-end structure; the expression synthesis and expression recognition tasks are separated, ultimately achieving an accuracy of 86.51%, a reduction of 0.84% ​​compared to the proposed method. In the end-to-end structure, the expression recognition module can guide the expression synthesis module, and the expression images synthesized by the expression synthesis module, in turn, further improve the performance of the expression recognition module. The experimental results above fully demonstrate the superiority of the end-to-end structure.

[0127] Table 3. Facial expression recognition results from the ablation experiment.

[0128]

[0129] This implementation proposes an unconstrained facial expression synthesis and recognition method. It synthesizes facial expression images with an unconstrained style, encompassing more natural variations caused by unconstrained imaging environment factors. This enriches the facial expression dataset, making the trained facial expression recognition model more applicable to unconstrained natural environments. Simultaneously, the proposed model employs an end-to-end architecture, combining the expression synthesis and recognition tasks. The images synthesized by the expression synthesis task expand the dataset, alleviating the problem of limited data volume. Conversely, the expression recognition task guides the expression synthesis task, synthesizing images more conducive to improving expression recognition performance. The two tasks cooperate to jointly promote performance improvement. Multiple experiments have demonstrated that the proposed method achieves good performance on the RAF-DB expression dataset.

[0130] Experiments on the RAF-DB and WiderFace datasets show that the method of this invention synthesizes a variety of facial expression images with unconstrained styles and achieves good facial expression recognition performance.

[0131] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0132] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0133] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for unconstrained facial expression synthesis and recognition, characterized in that, The method comprises the following steps: An expression synthesis step: capturing unconstrained style attributes in an unconstrained face dataset image, realizing conditional mapping from a single face expression image to a set of style conversion images; the expression synthesis step is specifically implemented as follows: After inputting the face expression image into the generator, the style pattern vector is injected into the generator through adaptive instance normalization to obtain a synthesized expression image; the style pattern vector is used to realize conditional mapping from a single image to a set of style conversion images; The discriminator distinguishes between the synthesized expression image and the unconstrained face image, and discriminates between true and false; wherein the generator expects to generate an expression image that can deceive the discriminator and be judged as true by the discriminator; and the discriminator expects to be able to distinguish the expression image and judge it as false; the discriminator is used to make the distribution of the expression image match the prior distribution of the latent variable . ​ The role of the style pattern vector is to change the style pattern of the face expression image, so as to ensure that the synthesized image has similar unconstrained style information as the unconstrained face image; wherein, is a style pattern vector extracted from an unconstrained face image; A content consistency step: when performing style conversion on the face expression image, the content information in the face expression image is kept unchanged to ensure content consistency; the content consistency step is specifically implemented as follows: The style encoder is used to extract a style pattern vector from the input face expression image and the unconstrained face image respectively, and then the extracted style pattern vector is injected into the generator to obtain a reconstructed image; An expression recognition step: performing expression classification on the generated style conversion image, and feeding the loss generated by the classification back to the generator, so that the generator no longer performs blind data enhancement to synthesize images more conducive to performance improvement of expression recognition.

2. The unconstrained facial expression synthesis and recognition method of claim 1, wherein, In the process of obtaining the reconstructed image, a reconstruction loss is used to ensure the content consistency of the image; The reconstruction loss is composed of a perceptual loss and a mean square error; The perceptual loss uses features extracted from a VGG model to measure the similarity between images.

3. The unconstrained facial expression synthesis and recognition method of claim 1, wherein, The expression recognition step adopts a typical cross-entropy loss function.

4. An unconstrained facial expression synthesis and recognition system characterized by, The method comprises the following steps: An expression synthesis step: capturing unconstrained style attributes in an unconstrained face dataset image, realizing conditional mapping from a single face expression image to a set of style conversion images; the expression synthesis step is specifically implemented as follows: After inputting the face expression image into the generator, the style pattern vector is injected into the generator through adaptive instance normalization to obtain a synthesized expression image; the style pattern vector is used to realize conditional mapping from a single image to a set of style conversion images; The discriminator distinguishes between the synthesized expression image and the unconstrained face image, and discriminates between true and false; wherein the generator expects to generate an expression image that can deceive the discriminator and be judged as true by the discriminator; and the discriminator expects to be able to distinguish the expression image and judge it as false; the discriminator is used to make the distribution of the expression image match the prior distribution . ​ The role of the style pattern vector is to change the style pattern of the face expression image, so as to ensure that the synthesized image has similar unconstrained style information as the unconstrained face image; wherein, is a style pattern vector extracted from an unconstrained face image; A content consistency step: when performing style conversion on the face expression image, the content information in the face expression image is kept unchanged to ensure content consistency; the content consistency step is specifically implemented as follows: The style encoder is used to extract a style pattern vector from the input face expression image and the unconstrained face image respectively, and then the extracted style pattern vector is injected into the generator to obtain a reconstructed image; An expression recognition step: performing expression classification on the generated style conversion image, and feeding the loss generated by the classification back to the generator, so that the generator no longer performs blind data enhancement to synthesize images more conducive to performance improvement of expression recognition.

5. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory and loadable on the processor, characterized in that, The processor executes the program to realize the steps of the method of any one of claims 1-3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to execute the steps of the method of any one of claims 1-3.

Images