Expression recognition method and device

Abstract

The embodiment of the present invention discloses an expression recognition method and device. The expression recognition method includes: according to a preset reference identity selection strategy, migrating facial expression images of various identities to the facial image of the reference identity, and constructing a facial expression image identity normalization module; training the information volume and feature correlation of the image respectively, and constructing an expression attention driving module through a channel attention mechanism; judging the expression category according to the facial expression image identity normalization module and the expression attention driving module. Through the present invention, the problem of low expression recognition accuracy caused by the lack of perception of differences in facial identities in the existing facial expression recognition method based on deep learning network in the related art is solved, and the technical effect of improving the expression recognition accuracy is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology applications, and in particular to an expression recognition method and device. Background Art

[0002] In the facial expression recognition system, removing the difference of face identity is the key to ensure the correct recognition of expression. Traditional expression recognition methods require a lot of time and manpower, and cannot remove the interference caused by face identity on recognition. Facial expression recognition based on deep learning network automatically extracts high-level abstract features to weaken the influence of face identity on expression features, and can achieve higher performance without consuming a lot of resources.

[0003] The related art discloses a method for facial expression recognition based on a deep learning network, which proposes an identity-aware convolutional neural network. First, a novel convolutional neural network is constructed to perceive the identity of the face more deeply; second, an expression-sensitive contrast loss is proposed to measure the similarity of expressions; finally, an identity-sensitive contrast loss algorithm is proposed to learn identity-related information from identity labels to achieve identity-invariant expression recognition. However, this method has the following problems: first, the expression-sensitive contrast loss makes it difficult to obtain the mapping relationship between expression features and labels; second, the identity-sensitive contrast loss reduces the identity difference at the expense of discarding other dimensional features. Therefore, although this method can realize facial expression recognition, it does not fully perceive the identity.

[0004] In the current related technologies, the problem of low accuracy of expression recognition is caused by the insufficient perception of differences in facial identities in existing facial expression recognition methods based on deep learning networks, which has not yet been effectively solved. Summary of the invention

[0005] The embodiments of the present invention provide an expression recognition method and device to at least solve the problem in the related art that the existing facial expression recognition method based on deep learning network has insufficient perception of the difference of facial identity, resulting in low expression recognition accuracy.

[0006] According to one aspect of an embodiment of the present invention, a method for facial expression recognition is provided, comprising: migrating facial expression images of various identities to facial images of a benchmark identity based on a preset benchmark identity selection strategy, and constructing a facial expression image identity normalization module; training the information volume and feature correlation of the image separately, and constructing an expression attention driving module through a channel attention mechanism; and determining the expression category based on the facial expression image identity normalization module and the expression attention driving module.

[0007] Optionally, according to a preset benchmark identity selection strategy, facial expression images of various identities are migrated to the facial image of the benchmark identity, and a facial expression image identity normalization module is constructed, including: through a preset benchmark identity selection strategy, facial expression images of various identities are migrated to the facial image of the benchmark identity, and by filtering identity features that are not related to the facial expression images, a feature space related to the facial expression images is obtained, and a facial expression image identity normalization module is constructed.

[0008] Furthermore, optionally, the method also includes: a generator module based on a neural network, generating various expression images for the facial image of the reference identity, and obtaining the normalized facial identity of the facial expression image to be identified by filtering identity features that are not related to the facial expression image.

[0009] Optionally, the generator of the neural network includes: a first generator subnetwork and a second generator subnetwork, the first generator subnetwork being used for an attention mask; the second generator subnetwork being used for generating a color mask; wherein, the attention mask is obtained by combining the first generator subnetwork with an N-dimensional facial action unit feature vector of an input image and a target expression; the color mask is obtained by combining the second generator subnetwork with an input image and an N-dimensional facial action unit feature vector of a target expression according to the attention mechanism, ignoring background interference in the face area; and the target expression image is generated based on the attention mask and the color mask.

[0010] Optionally, the information content and feature relevance of the image are trained separately, and the expression attention driving module is constructed through the channel attention mechanism, including: the information content and feature relevance of the image are trained separately, the image entropy of the hidden layer feature map is calculated, the image entropy is used as a measurement standard to establish a branch, and the channel attention mechanism is used to retain the hidden layer features to construct the expression attention driving module.

[0011] Optionally, determining the expression category based on the facial expression image identity normalization module and the expression attention driving module includes: calculating the network training loss by combining the weighted loss function of image adversarial loss, attention loss, conditional expression loss and identity loss based on the facial expression image identity normalization module and the expression attention driving module to determine the expression category.

[0012] According to another aspect of an embodiment of the present invention, there is provided an expression recognition device, comprising: a first construction module, for migrating facial expression images of various identities to facial images of a reference identity according to a preset reference identity selection strategy, and constructing a facial expression image identity normalization module; a second construction module, for training the information volume and feature correlation of the image respectively, and constructing an expression attention driving module through a channel attention mechanism; a determination module, for determining the expression category according to the facial expression image identity normalization module and the expression attention driving module.

[0013] Optionally, the first construction module includes: a first construction unit, used to migrate facial expression images of various identities to the facial image of a baseline identity through a preset baseline identity selection strategy, and obtain a feature space related to the facial expression image by filtering identity features that are not related to the facial expression image, and construct a facial expression image identity normalization module.

[0014] Optionally, the second construction module includes: a second construction unit, used to train the information amount and feature correlation of the image separately, calculate the image entropy of the hidden layer feature map, use the image entropy as a measurement standard to establish a branch, and use the channel attention mechanism to retain the hidden layer features to construct an expression attention driving module.

[0015] Optionally, the determination module includes: a determination unit, which is used to calculate the training loss of the network based on the facial expression image identity normalization module and the expression attention driving module, by combining the weighted loss function of image adversarial loss, attention loss, conditional expression loss and identity loss to determine the expression category.

[0016] In the embodiment of the present invention, based on the preset benchmark identity selection strategy, the facial expression images of various identities are transferred to the facial image of the benchmark identity, and a facial expression image identity normalization module is constructed; the information volume and feature correlation of the image are trained separately, and an expression attention driving module is constructed through the channel attention mechanism; the expression category is determined based on the facial expression image identity normalization module and the expression attention driving module. In other words, the embodiment of the present invention can solve the problem of low expression recognition accuracy due to the lack of perception of differences in facial identities in the existing facial expression recognition method based on deep learning networks, thereby achieving the technical effect of improving the expression recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1A flowchart of an expression recognition method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of an expression recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second" and the like in the specification, claims and drawings of the present invention are used to distinguish different objects rather than to limit a specific order.

[0022] According to one aspect of an embodiment of the present invention, a method for facial expression recognition is provided. Figure 1 A flowchart of an expression recognition method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the expression recognition method provided by the embodiment of the present application includes:

[0023] Step S102, according to a preset benchmark identity selection strategy, the facial expression images of various identities are migrated to the facial image of the benchmark identity, and a facial expression image identity normalization module is constructed;

[0024] Optionally, in step S102, facial expression images of various identities are migrated to the facial image of the benchmark identity according to a preset benchmark identity selection strategy, and a facial expression image identity normalization module is constructed, including: migrating facial expression images of various identities to the facial image of the benchmark identity through a preset benchmark identity selection strategy, and obtaining a feature space related to the facial expression image by filtering identity features that are not related to the facial expression image, and constructing a facial expression image identity normalization module.

[0025] Specifically, the embodiment of the present application provides an expression recognition method by constructing a facial expression image identity normalization module, formulating a baseline identity selection strategy, migrating facial expression images of different identities to facial images of the baseline identity, removing identity features irrelevant to facial expressions, and obtaining a feature space related to facial expressions, thereby completing the facial identity normalization of the facial expression images to be identified.

[0026] Further, optionally, the expression recognition method provided by the embodiment of the present application also includes: a generator module based on a neural network, generating various expression images for the facial image of the reference identity, and obtaining the normalized facial identity of the facial expression image to be identified by filtering identity features that are not related to the facial expression image.

[0027] Optionally, the generator of the neural network includes: a first generator subnetwork and a second generator subnetwork, the first generator subnetwork being used for an attention mask; the second generator subnetwork being used for generating a color mask; wherein, the attention mask is obtained by combining the first generator subnetwork with an N-dimensional facial action unit feature vector of an input image and a target expression; the color mask is obtained by combining the second generator subnetwork with an input image and an N-dimensional facial action unit feature vector of a target expression according to the attention mechanism, ignoring background interference in the face area; and the target expression image is generated based on the attention mask and the color mask.

[0028] Specifically, in the embodiment of the present application, according to the preset reference identity selection strategy, the facial expression images of various identities are migrated to the facial image of the reference identity, and the facial expression image identity normalization module is constructed, including:

[0029] The generator module based on the GANimation network generates different expression images for the facial image of the baseline identity, removes identity features that are irrelevant to facial expressions, and completes the facial identity normalization of the facial expression image to be identified.

[0030] GANimation is a generative adversarial conditioning method based on facial action units. Due to the continuity of facial action units, this method can generate continuous facial expressions. The network structure of GANimation consists of two generators and a discriminator. The first generator generates the output image of the target expression from the original input image, and the second generator generates the original input image from the target expression image output by the first generator. The discriminator is used to judge the quality of the generated images and expressions.

[0031] The goal of the expression recognition method provided in the embodiment of the present application is to transfer facial expressions of people with different identities to facial images of people with the same identity. Therefore, in the identity normalization module constructed in the expression recognition method provided in the embodiment of the present application, only the first generator in the GANimation structure is used, which consists of two sub-networks G A (i.e., the first generator subnetwork in the embodiment of the present application) and G I (i.e., the second generator subnetwork in the embodiment of the present application).

[0032] Given an input image and the N-dimensional facial action unit feature vector y of the target expression f, the generator subnetwork G A Used to generate attention mask A, using the attention mechanism to emphasize the face area and ignore background interference; generator subnetwork G I Used to generate the color mask C. The expression of the generated attention mask A is:

[0033]

[0034] The expression of the generated color mask C is:

[0035]

[0036] Finally generate the target expression image

[0037]

[0038] Based on the constructed facial expression image identity normalization module, the facial identity normalization of the facial expression image to be recognized is completed.

[0039] Step S104, training the information content and feature relevance of the image respectively, and constructing an expression attention driving module through a channel attention mechanism;

[0040] Optionally, in step S104, the information content and feature relevance of the image are trained separately, and the expression attention driving module is constructed through the channel attention mechanism, including: the information content and feature relevance of the image are trained separately, the image entropy of the hidden layer feature map is calculated, the image entropy is used as a measurement standard to establish a branch, and the channel attention mechanism is used to retain the hidden layer features to construct the expression attention driving module.

[0041] Specifically, the embodiment of the present application provides an expression recognition method by constructing an expression attention driving module based on image entropy, training the information content and feature correlation of the image separately, calculating the image entropy of the hidden layer feature map and using it as a measurement standard to establish a branch, and utilizing the channel attention mechanism to retain inactive but informative hidden layer features, enrich the expression of hidden layer features, and achieve task differentiation.

[0042] Among them, the expression attention driving module based on image entropy is constructed as follows:

[0043] In the expression attention driving module based on image entropy constructed by the expression recognition method provided in the embodiment of the present application, the image information amount and feature correlation are trained separately, and the image entropy of the hidden layer feature map is calculated and used as a measurement standard to establish a branch, and the channel attention mechanism is used to retain the inactive but informative hidden layer features, thereby enriching the expression of the hidden layer features.

[0044] The expression attention driven module divides the fitting task of generating feature channel weights into two branches: one is to learn and calculate the correlation coefficient between the feature channel and the facial expression, and the other is to calculate the image entropy of each feature map in the hidden layer to obtain the amount of information contained in the feature channel. Finally, the two are superimposed as the standard for measuring the feature channel weight. The calculation formula of image entropy is as follows:

[0045]

[0046] where p i is the probability that the corresponding pixel value i in the image appears in the image.

[0047] Step S106, determining the expression category based on the facial expression image identity normalization module and the expression attention driving module.

[0048] Optionally, in step S106, determining the expression category based on the facial expression image identity normalization module and the expression attention driving module includes: calculating the network training loss by combining the weighted loss function of image adversarial loss, attention loss, conditional expression loss and identity loss based on the facial expression image identity normalization module and the expression attention driving module to determine the expression category.

[0049] Specifically, the embodiment of the present application provides an expression recognition method that combines the facial expression image identity normalization module and the expression attention driving module in step S102, and designs a weighted loss function that combines image adversarial loss, attention loss, conditional expression loss, and identity loss to calculate the network training loss, optimize network performance, and complete the determination of expression categories.

[0050] Among them, the determination of expression categories is as follows:

[0051] The embodiment of the present application provides an expression recognition method that designs a weighted loss function that combines image adversarial loss, attention loss, conditional expression loss, and identity loss to calculate the training loss of the network, optimize network performance, and complete the determination of expression categories.

[0052] The total loss function L designed by the present invention is the weighted sum of the above four losses, and the formula is:

[0053] L=L I +λ y L y +λ A L A +λ idt L idt (5)

[0054] Where L I , L y , L Aand L idt They are image adversarial loss, conditional expression loss, attention loss and identity loss. y , A and λ idt are hyperparameters that control each loss term.

[0055] Since the expression recognition method provided in the embodiment of the present application constructs a facial expression image identity normalization module, formulates a baseline identity selection strategy, and migrates expressions of different identities to the facial image of the baseline identity, thereby reducing the impact of facial identity differences on expression feature extraction; further constructs an expression attention driving module based on image entropy, calculates the image entropy of the hidden layer feature map, utilizes the channel attention mechanism, retains inactive but informative hidden layer features, enriches the expression of hidden layer features, and completes the determination of expression categories based on the joint weighted loss, effectively avoiding the interference of irrelevant factors on expression recognition and improving the reliability of facial expression recognition.

[0056] The expression recognition method provided in the embodiment of the present application has an average recognition rate of 98.9% for six basic facial expressions on the public facial expression dataset CK+, which is significantly superior to the recognition rate of 95.1% in the literature method.

[0057] Since the expression recognition method provided by the embodiment of the present application takes into account identity normalization and expression attention drive, the influence of facial identity differences on expression feature extraction is reduced. At the same time, a channel attention mechanism is constructed to retain inactive but informative hidden layer features, enrich the hidden layer's expression of expressions, and complete the determination of expression categories based on the joint weighted loss, effectively avoiding the interference of irrelevant factors on expression recognition, improving the reliability of facial expression recognition, and achieving efficient recognition of facial expressions.

[0058] In the embodiment of the present invention, based on the preset benchmark identity selection strategy, the facial expression images of various identities are transferred to the facial image of the benchmark identity, and a facial expression image identity normalization module is constructed; the information volume and feature correlation of the image are trained separately, and an expression attention driving module is constructed through the channel attention mechanism; the expression category is determined based on the facial expression image identity normalization module and the expression attention driving module. In other words, the embodiment of the present invention can solve the problem of low expression recognition accuracy due to the lack of perception of differences in facial identities in the existing facial expression recognition method based on deep learning networks, thereby achieving the technical effect of improving the expression recognition accuracy.

[0059] According to another aspect of an embodiment of the present invention, there is provided an expression recognition device, Figure 2 A schematic diagram of an expression recognition device provided by an embodiment of the present invention is shown in FIG. Figure 2As shown, the expression recognition device provided by the embodiment of the present application includes: a first construction module 22, which is used to migrate facial expression images of various identities to the facial image of the benchmark identity according to a preset benchmark identity selection strategy, and construct a facial expression image identity normalization module; a second construction module 24, which is used to train the information volume and feature correlation of the image respectively, and construct an expression attention driving module through a channel attention mechanism; a judgment module 26, which is used to judge the expression category according to the facial expression image identity normalization module and the expression attention driving module.

[0060] Optionally, the first construction module 22 includes: a first construction unit, used to migrate facial expression images of various identities to the facial image of a baseline identity through a preset baseline identity selection strategy, and obtain a feature space related to the facial expression image by filtering identity features that are not related to the facial expression image, and construct a facial expression image identity normalization module.

[0061] Optionally, the second construction module 24 includes: a second construction unit, used to train the information content and feature correlation of the image separately, calculate the image entropy of the hidden layer feature map, use the image entropy as a measurement standard to establish a branch, and use the channel attention mechanism to retain the hidden layer features to construct an expression attention driving module.

[0062] Optionally, the determination module 26 includes: a determination unit, which is used to calculate the training loss of the network based on the facial expression image identity normalization module and the expression attention driving module, by combining the weighted loss function of the image adversarial loss, attention loss, conditional expression loss and identity loss to determine the expression category.

[0063] In the embodiment of the present invention, based on the preset benchmark identity selection strategy, the facial expression images of various identities are transferred to the facial image of the benchmark identity, and a facial expression image identity normalization module is constructed; the information volume and feature correlation of the image are trained separately, and an expression attention driving module is constructed through the channel attention mechanism; the expression category is determined based on the facial expression image identity normalization module and the expression attention driving module. In other words, the embodiment of the present invention can solve the problem of low expression recognition accuracy due to the lack of perception of differences in facial identities in the existing facial expression recognition method based on deep learning networks, thereby achieving the technical effect of improving the expression recognition accuracy.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for facial expression recognition, characterized in that: include: According to the preset benchmark identity selection strategy, the facial expression images of various identities are transferred to the facial image of the benchmark identity, and a facial expression image identity normalization module is constructed; The information content and feature relevance of the image are trained separately, and the expression attention driving module is constructed through the channel attention mechanism; The expression category is determined based on the facial expression image identity normalization module and the expression attention driving module.

2. The method according to claim 1, characterized in that The method of migrating facial expression images of various identities to facial images of the benchmark identity according to the preset benchmark identity selection strategy and constructing a facial expression image identity normalization module includes: Through the preset benchmark identity selection strategy, the facial expression images of the various identities are migrated to the facial image of the benchmark identity, and by filtering the identity features that are not related to the facial expression image, a feature space related to the facial expression image is obtained to construct the facial expression image identity normalization module.

3. The method according to claim 2, characterized in that The method further comprises: A generator module based on a neural network generates various expression images for the facial image of the reference identity, and obtains the normalized facial identity of the facial expression image to be identified by filtering identity features irrelevant to the facial expression image.

4. The method according to claim 3, characterized in that The generator of the neural network includes: a first generator sub-network and a second generator sub-network, wherein the first generator sub-network is used for attention mask; the second generator sub-network is used for generating color mask; wherein, Obtaining the attention mask by combining the first generator subnetwork with the input image and the N-dimensional facial action unit feature vector of the target expression; Ignoring background interference in the face area according to the attention mechanism, the color mask is obtained by inputting the image and the N-dimensional facial action unit feature vector of the target expression and combining the second generator sub-network; A target expression image is generated according to the attention mask and the color mask.

5. The method according to claim 1, characterized in that The information content and feature relevance of the image are trained separately, and the expression attention driving module is constructed through the channel attention mechanism, including: The information content and feature relevance of the image are trained separately, and the image entropy of the hidden layer feature map is calculated, and the image entropy is used as a measurement standard to establish a branch, and the channel attention mechanism is used to retain the hidden layer features to construct the expression attention driving module.

6. The method according to any one of claims 1 to 5, characterized in that The determining of the expression category based on the facial expression image identity normalization module and the expression attention driving module includes: According to the facial expression image identity normalization module and the expression attention driving module, the network training loss is calculated by combining the weighted loss function of image adversarial loss, attention loss, conditional expression loss and identity loss to determine the expression category.

7. An expression recognition device, characterized in that: include: The first construction module is used to migrate the facial expression images of various identities to the facial image of the benchmark identity according to a preset benchmark identity selection strategy, and to construct a facial expression image identity normalization module; The second building module is used to train the information content and feature relevance of the image separately, and build an expression attention driving module through the channel attention mechanism; A determination module is used to determine the expression category based on the facial expression image identity normalization module and the expression attention driving module.

8. The device according to claim 7, characterized in that The first building block comprises: The first construction unit is used to migrate the facial expression images of the various identities to the facial image of the benchmark identity through the preset benchmark identity selection strategy, and obtain the feature space related to the facial expression image by filtering the identity features that are not related to the facial expression image, and construct the facial expression image identity normalization module.

9. The device according to claim 7, characterized in that The second building block comprises: The second construction unit is used to train the information content and feature correlation of the image respectively, calculate the image entropy of the hidden layer feature map, use the image entropy as a measurement standard to establish a branch, and use the channel attention mechanism to retain the hidden layer features to construct the expression attention driving module.

10. The device according to any one of claims 7 to 9, characterized in that The determination module comprises: A determination unit is used to calculate the training loss of the network based on the facial expression image identity normalization module and the expression attention driving module by combining the weighted loss function of image adversarial loss, attention loss, conditional expression loss and identity loss to determine the expression category.

Images