A facial expression recognition method and system based on adaptive feature extraction

Abstract

The present invention discloses a facial expression recognition method and system based on adaptive feature extraction, which involves artificial intelligence technology. This solution is proposed to solve the problem of feature representation ability in the existing technology. Pre-processing module: used to reduce the resolution of the original expression picture and perform preliminary feature extraction to obtain the primary features of the expression picture; shared feature extraction module: including a shared feature extraction network, used to extract the shared features of the expression picture; emotional tendency feature extraction module: including a router and an emotional tendency feature extraction network, used to adaptively select different feature extraction branches to extract emotional tendency features according to the emotional tendency characteristics of the expression label; classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion result. The advantage is that it can perform adaptive feature extraction on the expression image according to the emotional tendency characteristics unique to the facial expression category.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to a facial expression recognition method and system based on adaptive feature extraction. Background Art

[0002] Facial expressions contain a wealth of emotional information, directly conveying feelings and intentions to others and serving as a bridge for communication between people. Automatic facial expression recognition has a wide range of real-world applications, such as human-computer interaction, social robotics, and medical care. In recent years, numerous researchers have devoted themselves to facial expression recognition and proposed numerous FER algorithms. These methods can be categorized into those based on handcrafted features and those based on deep learning, depending on the feature extraction method. Early work primarily utilized handcrafted features or shallow learning to extract overall structural features from facial images. These methods tend to overlook features that have a significant impact on classification. With the emergence of large-scale facial expression datasets and the rapid development of GPU computing power, deep learning has been widely used in facial expression recognition tasks, achieving excellent performance. These methods automatically extract expression features using neural networks such as convolutional neural networks and transformers. Compared to features extracted by handcrafted methods, these extracted features are more comprehensive and detailed.

[0003] Although the facial expression recognition task can achieve good performance with the help of deep learning, there are still problems with weak feature representation ability and insufficient richness. The existing method EfficientFace (Zhao Z, Liu Q, Zhou F. Robust Lightweight Facial Expression Recognition Network with Label Distribution Training [C] / / National Conference on Artificial Intelligence. 2021.) is based on ShuffleNet-V2, introduces label distribution learning to alleviate the label noise problem, and designs a channel space modulator and a local feature extractor to extract fine-grained features of expressions to achieve better results. However, in order to meet real-time requirements, this method uses a lightweight model to simplify the structure, which reduces the representation ability and richness of the features. In addition, this method does not make full use of the unique characteristics of facial expressions to extract features of expressions and enrich and improve the expression features. Summary of the Invention

[0004] The present invention aims to provide a facial expression recognition method and system based on adaptive feature extraction to solve the problems existing in the above-mentioned prior art.

[0005] The facial expression recognition system based on adaptive feature extraction described in the present invention includes the following modules:

[0006] Preprocessing module: used to reduce the resolution of the original expression picture and perform preliminary feature extraction to obtain the primary features of the expression picture;

[0007] Shared feature extraction module: includes a shared feature extraction network, which is used to extract shared features of expression images;

[0008] Emotional tendency feature extraction module: includes a router and an emotional tendency feature extraction network, which is used to adaptively select different feature extraction branches to extract emotional tendency features based on the emotional tendency characteristics of the expression tag;

[0009] Classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion results.

[0010] The present invention discloses a facial expression recognition method based on adaptive feature extraction, which utilizes the facial expression recognition system to perform facial expression recognition.

[0011] The method and system for facial expression recognition based on adaptive feature extraction described in the present invention have the advantage of first pre-training a shared feature extraction network, and then training the shared feature extraction network, routers, and emotional tendency feature extraction network using a two-layer optimization algorithm to obtain a facial expression recognition model based on adaptive feature extraction. This pre-training followed by two-layer optimization training reduces the overall computational overhead during training, improves training efficiency, and reduces the difficulty of model training. It also has good scalability.

[0012] Furthermore, the router can adaptively extract features from facial expressions based on the emotional tendencies unique to each facial expression category. The emotional tendency feature extraction network enriches and refines these features without significantly increasing computational complexity. Its multi-branch structure and adaptive routing mechanism enable better feature extraction across diverse acquisition scenarios, enhancing the model's robustness.

[0013] The preprocessing module reduces the resolution of the original expression image and performs preliminary feature extraction to obtain primary features of the expression image. The shared feature extraction module extracts shared features from the expression image. The emotional tendency feature extraction module adaptively selects different feature extraction branches to extract emotional tendency features based on the emotional tendency characteristics of the expression label. Finally, the classification module fuses the extracted shared features with the emotional tendency features, and based on the fusion results, performs expression recognition on the original input expression image. The mutual fusion and complementation of shared features and emotional tendency features enhances the representational power and richness of expression features, thereby improving the robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural diagram of the facial expression recognition system described in the present invention.

[0015] Figure 2 It is a schematic diagram of the construction process of the facial expression recognition system described in the present invention.

[0016] Figure 3 It is a structural diagram of the emotional tendency feature extraction module described in the present invention. DETAILED DESCRIPTION

[0017] like Figures 1 to 3 As shown, the facial expression recognition system based on adaptive feature extraction described in the present invention includes the following modules:

[0018] Preprocessing module: This module is used to reduce the resolution of the original expression images and perform preliminary feature extraction to obtain the primary features of the expression images. The original dataset is divided into a training set and a validation set, and the images in the dataset are preprocessed by shuffling, cropping, flipping, and normalization.

[0019] Shared feature extraction module: This module includes a shared feature extraction network, which is used to extract shared features from expression images. Specifically, the primary feature F1 is extracted from the Conv1 block and input into the channel space modulator and local feature extractor to extract fine-grained features of the expression. Label distribution learning is introduced to alleviate the problem of label noise. The shared feature extraction network is pre-trained for several epochs to obtain a high-precision and low-overfitting pre-trained network. To ensure that the model parameters can be optimized in a positive direction, the shared feature extraction network is first pre-trained for a certain number of epochs. During this pre-training process, the router and the emotional tendency feature extraction network of the emotional tendency feature extraction module are inactive, and their parameters are frozen.

[0020] The emotional tendency feature extraction module includes a router and an emotional tendency feature extraction network. This module adaptively selects different feature extraction branches based on the emotional tendency of the expression tag. Specifically, the primary feature F1, obtained by reducing the image resolution and performing preliminary feature extraction, is input into the router. The fully connected layer then outputs a vector of size 4, representing the score of the expression image belonging to each emotional tendency category.

[0021] Through the study of the emotional tendency characteristics of expression labels, the seven basic expressions are divided into four emotional tendency categories: positive expressions, negative expressions, neutral expressions and surprise. Among them, happiness is a positive expression, sadness, fear, disgust, and anger are negative expressions, and neutral and surprise are each classified as one category. The reason for classifying surprise separately is that the expression of surprise can contain positive or negative emotional colors in human emotional expression. It is a complex emotional response and is therefore not suitable to be classified into any of the positive, negative and neutral categories. Therefore, the corresponding emotional tendency label can be obtained based on the label of the expression picture.

[0022] The router is composed of multiple stacked residual blocks and outputs a vector of size 4 through a pooling layer and a fully connected layer, corresponding to the above four emotional tendency classes, thereby realizing adaptive feature extraction of expressions. That is, the router will automatically select the corresponding feature extraction branch to extract the emotional tendency characteristics of the expression based on the emotional tendency characteristics of the expression label.

[0023] In the emotional tendency feature extraction network, each emotional tendency class corresponds to a feature extraction branch. Each feature extraction branch is composed of three depth-wise separable convolutions (DWS Conv), a BN layer, and a ReLU activation function. While making the multi-branch structure as lightweight as possible, the emotional tendency features of the expression are extracted based on the emotional tendency shown by the expression.

[0024] Classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion results.

[0025] After pre-training, the router and remaining network parameters are alternately trained using a two-layer optimization method. This continuously optimizes the router and network parameters, resulting in a facial expression recognition model based on adaptive feature extraction. Specifically, after pre-training, the shared feature extraction network parameters continue to be trained using the existing optimizer, learning rate, and decay strategy. The router and emotion feature extraction networks use the same optimizer and learning rate decay strategy.

[0026] Since the coupling between the router and the emotion tendency feature extraction network is complex and cannot be modeled explicitly, the l router(ψ, φ; D) simulates the coupling between the router and the branches in the sentiment feature extraction network. However, when the objective function and parameters are strongly dependent, using a joint training approach to train the network can result in high computational requirements and low efficiency. The two-layer optimization algorithm provides a hierarchical learning framework with two optimization tasks, where the objectives and variables of the outer layer depend on the results of the inner layer optimization. By optimizing each layer separately, the two-layer optimization algorithm reduces overall computational overhead and improves training efficiency.

[0027]

[0028]

[0029] The parameter θ of the entire model is split into the inner optimization variable φ of the router and the outer optimization variable ψ of the shared feature extraction network and the sentiment feature extraction network, that is, θ = ψ + φ, and represents the training loss of the model, where Represents a set of multiple expression images and their corresponding labels. The two-layer optimization algorithm uses a more efficient alternating optimization algorithm. When optimizing one layer, the variables of the other layer are set to frozen.

[0030]

[0031]

[0032] is the loss function of the shared feature extraction network and the emotional tendency feature extraction network, where N is the number of samples, c is the number of expression categories, It is the label distribution of the expression obtained by inputting the i-th expression picture into the pre-trained label distribution generator LDG. is the predicted label distribution calculated from the features extracted by the shared feature extraction network and the sentiment feature extraction network, and Where i and j represent the index of expression samples and expression categories.

[0033]

[0034] is the loss function of the router, Loss CE refers to the cross entropy loss function, It is the output of the router with the primary feature F1 as input, which represents the predicted score of the four emotional tendency classes corresponding to each expression picture. The emotional tendency labels are obtained by mapping the seven basic expressions into four emotional tendency categories. Specifically, as shown in the following formula, happiness is a positive expression, sadness, fear, disgust, and anger are negative expressions, and neutral and surprise expressions are each considered a category:

[0035]

[0036] The shared feature extraction network, emotional tendency feature extraction network, and router are trained through the above training strategy and loss function, and the router and other network parameters are continuously optimized to obtain a facial expression recognition model based on adaptive feature extraction.

[0037] The method for facial expression recognition based on adaptive feature extraction described in the present invention utilizes the facial expression recognition system to perform facial expression recognition.

[0038] The method and system for facial expression recognition based on adaptive feature extraction described in the present invention have at least the following technical advantages over the prior art:

[0039] First, to address the problems of insufficient feature representation and richness in existing expression recognition methods, a multi-branch structure is designed so that different branches can focus on extracting different dimensions of expression features, thereby improving the diversity and expressiveness of features.

[0040] 2. Using the emotional tendency characteristics of expression tags, a router is designed to perform adaptive feature extraction of emotional tendency expression features according to the emotional tendency class to which the expression belongs.

[0041] 3. Considering the complexity of coupling modeling between the router and the emotional tendency feature extraction network, a two-layer optimization technology is used to alleviate the problem of low network training efficiency during joint training. The extracted shared features are then integrated and supplemented with the emotional tendency features to improve the representation ability of expression features.

[0042] Those skilled in the art can make various other corresponding changes and deformations based on the technical solutions and concepts described above, and all of these changes and deformations should fall within the scope of protection of the claims of the present invention.

Claims

1. A facial expression recognition system based on adaptive feature extraction, characterized in that: Includes the following modules: Preprocessing module: used to reduce the resolution of the original expression picture and perform preliminary feature extraction to obtain the primary features of the expression picture; Shared feature extraction module: includes a shared feature extraction network, which is used to extract shared features of expression images; The emotional tendency feature extraction module includes a router and an emotional tendency feature extraction network, which is used to adaptively select different feature extraction branches to extract emotional tendency features based on the emotional tendency characteristics of the expression tag. The emotional tendency feature extraction module inputs the primary feature F1 obtained by reducing the image resolution and performing preliminary feature extraction into the router, and outputs a vector of size 4 through a fully connected layer to represent the score of the expression image belonging to each emotional tendency class. Classification module: used to fuse the extracted shared features and emotional tendency features, and perform expression recognition on the original input expression picture based on the fusion results.

2. A facial expression recognition system based on adaptive feature extraction according to claim 1, characterized in that: In order to ensure that the model parameters can be optimized in a positive direction, the shared feature extraction network is pre-trained for several epochs to obtain a high-precision and low-overfitting pre-trained network.

3. A facial expression recognition system based on adaptive feature extraction according to claim 2, characterized in that: During the training process of the pre-trained network, the router and the emotional tendency feature extraction network of the emotional tendency feature extraction module do not work, and the parameters are frozen.

4. A facial expression recognition system based on adaptive feature extraction according to claim 3, characterized in that: The seven basic expressions are divided into four emotional tendency categories: positive expression, negative expression, neutral expression and surprise; among them, happiness is a positive expression, sadness, fear, disgust, and anger are negative expressions, and neutral and surprise are each a category; the corresponding emotional tendency label is obtained according to the label of the expression picture 5. A facial expression recognition system based on adaptive feature extraction according to claim 4, characterized in that: The router is composed of a stack of multiple residual blocks, and outputs a vector of size 4 through a pooling layer and a fully connected layer, corresponding to the four emotional tendency classes, thereby realizing adaptive feature extraction of expression pictures.

6. A facial expression recognition system based on adaptive feature extraction according to claim 5, characterized in that: In the emotional tendency feature extraction network, each emotional tendency class corresponds to a feature extraction branch, and each feature extraction branch is composed of three depth-wise separable convolutions, a BN layer, and a ReLU activation function.

7. A facial expression recognition system based on adaptive feature extraction according to claim 6, characterized in that: After the system pre-training is completed, a two-layer optimization method is used to alternately train the router and the remaining network parameters, and the router and network parameters are continuously optimized to obtain a facial expression recognition model based on adaptive feature extraction.

8. A facial expression recognition system based on adaptive feature extraction according to claim 7, characterized in that: After pre-training, the network parameters of the shared feature extraction network continue to be trained with the existing optimizer, learning rate, and decay strategy. The router and sentiment feature extraction networks use the same optimizer and learning rate decay strategy. By l router (ψ, φ; D) simulates the coupling of routers and branches in the sentiment feature extraction network. A hierarchical learning framework with two optimization tasks is provided based on a two-layer optimization algorithm, where the objectives and variables of the outer layer depend on the results of the inner layer optimization. The two-layer optimization algorithm reduces the overall computational overhead and improves training efficiency by optimizing each layer separately. The parameter θ of the entire model is split into the inner optimization variable φ of the router and the outer optimization variable ψ of the shared feature extraction network and the sentiment feature extraction network, that is, θ = ψ + φ, where Represents a set of multiple expression images and their corresponding labels. The two-layer optimization algorithm uses a more efficient alternating optimization algorithm. When optimizing one layer, the variables of the other layer are set to frozen. l(ψ,φ * (ψ); D) is the loss function of the shared feature extraction network and the emotional tendency feature extraction network, where N is the number of samples, c is the number of expression categories, It is the label distribution of the expression obtained by inputting the i-th expression picture into the pre-trained label distribution generator LDG. is the predicted label distribution calculated from the features extracted by the shared feature extraction network and the sentiment feature extraction network, and Where i and j represent the index of expression samples and expression categories; is the loss function of the router, Loss CE refers to the cross entropy loss function, It is the output of the router with the primary feature F1 as input, which represents the predicted score of the four emotional tendency classes corresponding to each expression picture. It is the emotional tendency label obtained by mapping the 7 basic expressions into 4 emotional tendency classes; By training the shared feature extraction network, the emotional tendency feature extraction network, and the router, and continuously optimizing the router and other network parameters, a facial expression recognition model based on adaptive feature extraction is obtained.

9. A facial expression recognition method based on adaptive feature extraction, characterized in that: Facial expression recognition is performed using the facial expression recognition system described in any one of claims 1 to 8.

Images