Facial expression recognition method and system based on adaptive feature extraction

Abstract

The invention discloses a facial expression recognition method and system based on adaptive feature extraction, relates to an artificial intelligence technology, and provides a scheme for the problems of feature representation capability and the like in the prior art. The preprocessing module is used for reducing the resolution of the original expression picture and performing preliminary feature extraction to obtain the primary feature of the expression picture; the shared feature extraction module comprises a shared feature extraction network and is used for extracting shared features of the expression pictures; the emotional tendency feature extraction module comprises a router and an emotional tendency feature extraction network and is used for adaptively selecting different feature extraction branches to extract emotional tendency features according to the emotional tendency features of the expression labels; and the classification module is used for fusing the extracted shared features and the extracted emotional tendency features, and performing expression recognition on an originally input expression picture based on a fusion result. The method has the advantages that adaptive feature extraction can be performed on the expression image according to the specific emotional tendency characteristics of the facial expression category.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to a facial expression recognition method and system based on adaptive feature extraction. Background Art

[0002] Facial expressions contain extremely rich emotional information, which can directly convey emotions or intentions to others and serve as a bridge for communication between people. Automatic facial expression recognition is widely used in the real world, such as human-computer interaction, social robots, and medical care. In recent years, many researchers have devoted themselves to facial expression recognition and proposed a large number of FER algorithms. According to the feature extraction method, these methods can be divided into manual design-based and deep learning-based methods. Early work mainly used manual design or shallow learning to extract the overall structural features of facial images. These methods tend to ignore features that have a greater impact on classification when extracting features. With the continuous emergence of large-scale facial expression datasets and the rapid development of GPU computing power, deep learning has been widely used in facial expression recognition tasks and has achieved good performance. The expression features of these methods are automatically extracted by convolutional neural networks, transformers and other neural networks. Compared with the features extracted by manual design-based methods, the extracted features are more comprehensive and detailed.

[0003] Although the facial expression recognition task can achieve good performance with the support of deep learning, there are still problems with weak feature representation ability and insufficient richness. The existing method EfficientFace (Zhao Z, Liu Q, Zhou F. Robust Lightweight Facial Expression Recognition Network with Label Distribution Training [C] / / National Conference on Artificial Intelligence. 2021.) is based on ShuffleNet-V2, introduces label distribution learning to alleviate the problem of label noise, and designs channel space modulators and local feature extractors to extract fine-grained features of expressions to achieve better results. However, in order to meet real-time requirements, this method uses a lightweight model to simplify the structure, which reduces the representation ability and richness of the features. In addition, this method does not make full use of the unique characteristics of facial expressions to extract features of expressions and enrich and improve expression features. Summary of the invention

[0004] The present invention aims to provide a method and system for facial expression recognition based on adaptive feature extraction to solve the problems existing in the above-mentioned prior art.

[0005] The facial expression recognition system based on adaptive feature extraction described in the present invention comprises the following modules:

[0006] Preprocessing module: used to reduce the resolution of the original expression picture and perform preliminary feature extraction to obtain the primary features of the expression picture;

[0007] Shared feature extraction module: including a shared feature extraction network, used to extract shared features of expression images;

[0008] Emotional tendency feature extraction module: including a router and an emotional tendency feature extraction network, which is used to adaptively select different feature extraction branches to extract emotional tendency features according to the emotional tendency characteristics of expression tags;

[0009] Classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion results.

[0010] The invention discloses a facial expression recognition method based on adaptive feature extraction, and utilizes the facial expression recognition system to perform facial expression recognition.

[0011] The method and system for facial expression recognition based on adaptive feature extraction described in the present invention have the advantages that the shared feature extraction network is first pre-trained, and then the shared feature extraction network, the router and the emotional tendency feature extraction network are trained using a double-layer optimization algorithm, thereby obtaining a facial expression recognition model based on adaptive feature extraction. The operation of pre-training first and then double-layer optimization training reduces the overall computational overhead during training, improves training efficiency, reduces the difficulty of model training, and also has good scalability.

[0012] In addition, the router can extract features of facial expressions based on the emotional tendency characteristics of facial expression categories. The emotional tendency feature extraction network enriches and improves the expression features without significantly increasing the amount of calculation. Its multi-branch structure and adaptive routing mechanism can better extract features under different acquisition conditions and improve the robustness of the model.

[0013] The preprocessing module is used to reduce the resolution of the original expression picture and perform preliminary feature extraction to obtain the primary features of the expression picture; the shared feature extraction module is used to extract the shared features in the expression picture; the emotional tendency feature extraction module is used to adaptively select different feature extraction branches to extract emotional tendency features according to the emotional tendency characteristics of the expression label; finally, the classification module is used to fuse the extracted shared features and emotional tendency features, and the original input expression picture is recognized based on the fusion results. The shared features and emotional tendency features are integrated and complemented with each other, which improves the representation ability and richness of the expression features, thereby improving the robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural schematic diagram of the facial expression recognition system described in the present invention.

[0015] Figure 2 It is a schematic diagram of the construction process of the facial expression recognition system described in the present invention.

[0016] Figure 3 It is a structural diagram of the emotion tendency feature extraction module described in the present invention. DETAILED DESCRIPTION

[0017] like Figures 1 to 3 As shown, the facial expression recognition system based on adaptive feature extraction described in the present invention includes the following modules:

[0018] Preprocessing module: used to reduce the resolution of the original expression image and perform preliminary feature extraction to obtain the primary features of the expression image. The original data set is divided into a training set and a validation set, and the images in the data set are shuffled, cropped, flipped, and normalized.

[0019] Shared feature extraction module: includes a shared feature extraction network, which is used to extract shared features of expression pictures. Specifically, the primary feature F1 is extracted from the Conv1 block and input into the channel space modulator and the local feature extractor to extract the fine-grained features of the expression, and label distribution learning is introduced to alleviate the label noise problem. The shared feature extraction network is pre-trained for several epochs to obtain a high-precision and low-overfitting pre-trained network. In order to ensure that the model parameters can be optimized in a positive direction, the shared feature extraction network is first pre-trained for a certain number of epochs. During this pre-training process, the router and the emotional tendency feature extraction network of the emotional tendency feature extraction module do not work, and the parameters will be frozen.

[0020] Emotional tendency feature extraction module: It includes a router and an emotional tendency feature extraction network, which is used to adaptively select different feature extraction branches to extract emotional tendency features according to the emotional tendency characteristics of the expression tag. Specifically, the primary feature F1 obtained by reducing the image resolution and performing preliminary feature extraction is input into the router, and a vector of size 4 is output through the fully connected layer, that is, the score of the expression image belonging to each emotional tendency class.

[0021] Through the study of the emotional tendency characteristics of expression labels, the seven basic expressions are divided into four emotional tendency categories: positive expression, negative expression, neutral expression and surprise. Among them, happiness belongs to positive expression, sadness, fear, disgust, anger belong to negative expression, neutral and surprise are each a category. The reason for classifying surprise separately is that the expression of surprise can contain positive emotional colors or negative emotional colors in human emotional expression. It is a complex emotional response, so it is not suitable to be classified into any of the positive, negative and neutral categories. Therefore, the corresponding emotional tendency label can be obtained according to the label of the expression picture.

[0022] The router is composed of a plurality of stacked residual blocks, and outputs a vector of size 4 through a pooling layer and a fully connected layer, corresponding to the above four emotion tendency classes, thereby realizing adaptive feature extraction of expressions, that is, the router automatically selects the corresponding feature extraction branch to extract the emotion tendency characteristics of the expression according to the emotion tendency characteristics of the expression label.

[0023] In the emotional tendency feature extraction network, each emotional tendency class corresponds to a feature extraction branch, and each feature extraction branch is composed of three depth-wise separable convolutions DWS Conv, BN layer, and ReLU activation function. While making the multi-branch structure as lightweight as possible, the emotional tendency features of the expression are extracted according to the emotional tendency shown by the expression.

[0024] Classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion results.

[0025] After the system pre-training is completed, the router and the remaining network parameters are alternately trained using a two-layer optimization method to continuously optimize the router and network parameters to obtain a facial expression recognition model based on adaptive feature extraction. Specifically, after the pre-training is completed, the network parameters of the shared feature extraction network continue to be trained with the existing optimizer, learning rate and its decay strategy, and the router and emotion tendency feature extraction network use the same optimizer and learning rate decay strategy.

[0026] Since the coupling between the router and the sentiment feature extraction network is complex and cannot be explicitly modeled, router(ψ, φ; D) simulates the coupling of the branches in the router and sentiment feature extraction network. However, when the objective function is strongly dependent on the parameters, training the network by joint training may result in huge computational requirements and low efficiency. The two-layer optimization algorithm provides a hierarchical learning framework with two layers of optimization tasks, where the objectives and variables of the outer layer problem depend on the results of the inner layer optimization. The two-layer optimization algorithm reduces the overall computational overhead and improves the training efficiency by optimizing each layer separately.

[0027]

[0028]

[0029] The parameter θ of the entire model is split into the inner optimization variable φ of the router and the outer optimization variable ψ of the shared feature extraction network and the sentiment feature extraction network, that is, θ = ψ + φ, and represents the training loss of the model, where Represents a set of multiple expression images and their corresponding labels. The two-layer optimization algorithm uses a more efficient alternating optimization algorithm. When optimizing one layer, the variables of the other layer are set to frozen.

[0030]

[0031]

[0032] is the loss function of the shared feature extraction network and the emotional tendency feature extraction network, where N is the number of samples, c is the number of expression categories, is the label distribution of the expression obtained by inputting the i-th expression picture into the pre-trained label distribution generator LDG. is the predicted label distribution calculated from the features extracted by the shared feature extraction network and the sentiment feature extraction network, and Where i and j represent the index of expression samples and expression categories.

[0033]

[0034] is the loss function of the router, Loss CE refers to the cross entropy loss function, It is the output of the router with the primary feature F1 as input, which represents the predicted scores of the four emotion tendency classes corresponding to each expression picture. It is the emotional tendency label obtained by mapping the 7 basic expressions into 4 emotional tendency categories. As shown in the following formula, happiness belongs to the positive expression, sadness, fear, disgust, and anger belong to the negative expression, and neutral and surprise expressions are each a category:

[0035]

[0036] The shared feature extraction network, emotional tendency feature extraction network, and router are trained through the above training strategy and loss function, and the router and other network parameters are continuously optimized to obtain a facial expression recognition model based on adaptive feature extraction.

[0037] The method for facial expression recognition based on adaptive feature extraction described in the present invention is to perform facial expression recognition using the facial expression recognition system.

[0038] The method and system for facial expression recognition based on adaptive feature extraction described in the present invention have at least the following technical advantages over the prior art:

[0039] First, in order to address the problems of insufficient feature representation ability and richness in existing expression recognition methods, a multi-branch structure is designed so that different branches can focus on extracting different dimensions of expression features, thereby improving the diversity and expressiveness of features.

[0040] Second, using the emotional tendency characteristics of expression tags, a router is designed to perform adaptive feature extraction of emotional tendency expression features according to the emotional tendency class to which the expression belongs.

[0041] 3. Considering the complexity of coupling modeling between the router and the emotion tendency feature extraction network, a two-layer optimization technology is used to alleviate the problem of low network training efficiency during joint training. The extracted shared features are then integrated and supplemented with the emotion tendency features to improve the representation ability of expression features.

[0042] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A facial expression recognition system based on adaptive feature extraction, characterized in that: Includes the following modules: Preprocessing module: used to reduce the resolution of the original expression picture and perform preliminary feature extraction to obtain the primary features of the expression picture; Shared feature extraction module: including a shared feature extraction network, used to extract shared features of expression images; Emotional tendency feature extraction module: including a router and an emotional tendency feature extraction network, which is used to adaptively select different feature extraction branches to extract emotional tendency features according to the emotional tendency characteristics of expression tags; Classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion results.

2. A facial expression recognition system based on adaptive feature extraction according to claim 1, characterized in that: In order to ensure that the model parameters can be optimized in a positive direction, the shared feature extraction network is pre-trained for several epochs to obtain a high-precision and low-overfitting pre-trained network.

3. A facial expression recognition system based on adaptive feature extraction according to claim 2, characterized in that: During the training process of the pre-trained network, the router and the emotional tendency feature extraction network of the emotional tendency feature extraction module are inoperative, and the parameters are frozen.

4. A facial expression recognition system based on adaptive feature extraction according to claim 3, characterized in that: The emotional tendency feature extraction module inputs the primary feature F1 obtained by reducing the image resolution and performing preliminary feature extraction into the router, and outputs a vector of size 4 through a fully connected layer to represent the score of the expression image belonging to each emotional tendency class.

5. A facial expression recognition system based on adaptive feature extraction according to claim 4, characterized in that: The seven basic expressions are divided into four emotional tendency categories: positive expression, negative expression, neutral expression and surprise; among them, happiness belongs to positive expression, sadness, fear, disgust and anger belong to negative expression, neutral and surprise are each a category; the corresponding emotional tendency label is obtained according to the label of the expression picture 6. A facial expression recognition system based on adaptive feature extraction according to claim 5, characterized in that: The router is formed by stacking a plurality of residual blocks, and outputs a vector of size 4 through a pooling layer and a fully connected layer, corresponding to the four emotion tendency classes, thereby realizing adaptive feature extraction of expression pictures.

7. A facial expression recognition system based on adaptive feature extraction according to claim 6, characterized in that: In the sentiment tendency feature extraction network, each sentiment tendency class corresponds to a feature extraction branch, and each feature extraction branch is composed of three depth-separable convolutions, a BN layer, and a ReLU activation function.

8. A facial expression recognition system based on adaptive feature extraction according to claim 7, characterized in that: After the system pre-training is completed, a two-layer optimization method is used to alternately train the router and the remaining network parameters, and the router and network parameters are continuously optimized to obtain a facial expression recognition model based on adaptive feature extraction.

9. A facial expression recognition system based on adaptive feature extraction according to claim 8, characterized in that: After the pre-training is completed, the network parameters of the shared feature extraction network continue to be trained with the existing optimizer, learning rate and its decay strategy, and the router and sentiment feature extraction network use the same optimizer and learning rate decay strategy; By l router (ψ, φ; D) simulates the coupling of the router and the branches in the sentiment feature extraction network; provides a hierarchical learning framework with two-layer optimization tasks based on the two-layer optimization algorithm, where the objectives and variables of the outer layer problem depend on the results of the inner layer optimization; the two-layer optimization algorithm reduces the overall computational overhead and improves the training efficiency by optimizing each layer separately; The parameter θ of the entire model is split into the inner optimization variable φ of the router and the outer optimization variable ψ of the shared feature extraction network and the sentiment feature extraction network, that is, θ = ψ + φ, and represents the training loss of the model, where Represents a set of multiple expression images and their corresponding labels. The two-layer optimization algorithm uses a more efficient alternating optimization algorithm. When optimizing one layer, the variables of the other layer are set to be frozen. is the loss function of the shared feature extraction network and the emotional tendency feature extraction network, where N is the number of samples, c is the number of expression categories, is the label distribution of the expression obtained by inputting the i-th expression picture into the pre-trained label distribution generator LDG. is the predicted label distribution calculated from the features extracted by the shared feature extraction network and the sentiment feature extraction network, and Where i and j represent the index of expression samples and expression categories; is the loss function of the router, Loss CE refers to the cross entropy loss function, It is the output of the router with the primary feature F1 as input, which represents the predicted scores of the four emotion tendency classes corresponding to each expression picture. It is the emotional tendency label obtained by mapping the 7 basic expressions into 4 emotional tendency categories; By training the shared feature extraction network, the emotion tendency feature extraction network and the router, and continuously optimizing the parameters of the router and other networks, a facial expression recognition model based on adaptive feature extraction is obtained.

10. A facial expression recognition method based on adaptive feature extraction, characterized in that: Facial expression recognition is performed using the facial expression recognition system described in any one of claims 1 to 9.

Images