A dual-branch hybrid residual connection method for facial expression recognition

Abstract

The present invention provides an expression recognition method with dual-branch mixed residual connection. Specifically, it includes: taking ResNet18 as the backbone feature extraction network, designing a dual-branch multi-residual connection method on this basis to improve feature extraction; secondly, comprehensively analyzing the problems of network depth and width, fusing deep separable convolution, and building a lightweight expression recognition network; finally, according to the characteristics of facial expression information, in order to obtain more accurate and complete feature information, adding an adaptive feature fusion module ASFF, and analyzing and verifying the fused information, selecting the most superior hierarchical output, and performing the final expression classification prediction. The present invention combines a perfect feature extraction mechanism with an adaptive feature fusion method, and replaces ordinary convolution with deep separable convolution, which can solve the recognition accuracy of expression recognition under uncontrolled conditions and the lightweight problem of the model.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to an expression recognition method using a dual-branch mixed residual connection. The method combines adaptive feature fusion to construct a robust feature extraction network to obtain better facial expression recognition. Background Art

[0002] Facial expressions play a large role in interpersonal communication. Facial expression recognition is a technology in which computers try to understand human emotions by analyzing facial information. It has become a hot topic in the field of computer vision. With the development and deepening of facial expression research, the methods of expression recognition are becoming more and more diverse. To improve the accuracy and speed of expression recognition and to deal with the various influencing factors in recognition, the development of computer technology has affected the development of computer technology to a certain extent. With the rapid growth of chip processing power and the careful design of network architecture, research in various fields has begun to turn to the use of deep learning to solve various problems and has also achieved recognition results far exceeding previous methods. Therefore, the development space of facial expression recognition technology in various aspects has also been further developed. The task of facial expression recognition has gradually shifted from the controlled environment of the laboratory to the challenging real-world environment. Driven by deep learning, neural networks can automatically learn feature information and are gradually applied to automatic facial expression recognition tasks.

[0003] As a branch of face recognition, the research of expression recognition has great research value in psychology and behavior. The most critical technology in facial expression analysis is the facial feature extraction algorithm. Due to the similarity between classes and the difference within classes of expression feature information, it is necessary to extract sufficiently complete and subtle feature information. However, if the deep neural network is used directly for feature extraction, overfitting is prone to occur, and the recognition accuracy we need cannot be achieved. Therefore, this design proposes a hybrid residual connection method, on which a dual-branch feature extraction network is designed to improve feature extraction; secondly, the network depth and width are comprehensively analyzed, and the deep separable convolution is integrated to construct a lightweight expression recognition network; finally, according to the characteristics of facial expression information, in order to obtain more accurate and complete feature information, an adaptive feature fusion module is added, and the fused information is analyzed and verified, and the hierarchical output with the best performance is selected for the final expression classification prediction. Applying deep learning to expression recognition, while ensuring recognition accuracy, to improve applicability, and lightweight models are easy to deploy to other embedded devices, is also the current research focus. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the current expression recognition task, such as the small number of reliable expression recognition databases, the lack of robustness of recognition under uncontrolled conditions, the long training time of deep learning expression recognition models, and the large size of the models. By training and testing the uncontrolled expression recognition data set FER2013, a better recognition rate than the existing algorithm is obtained by the method using a single network training and without taking additional training data.

[0005] In order to achieve the above-mentioned purpose, the present invention provides an expression recognition method with a dual-branch mixed residual connection to solve the problems of recognition accuracy of expression recognition under uncontrolled conditions and lightweight of the model. The method mainly includes five parts. The first part is basic image data enhancement; the second part is to construct a dual-branch mixed residual network for feature extraction; the third part is to add an adaptive feature fusion module; the fourth part is to fuse deep separable convolution to reduce the number of model parameters; the fifth part is network training and testing to output the final expression classification prediction.

[0006] The first part consists of two steps:

[0007] Step 1: Randomly scale and crop the image, and then horizontally flip the image data to increase the image database;

[0008] Step 2: Use mixup image fusion and label_smoothing label smoothing to further enhance the image data so that the trained model can have better generalization ability;

[0009] The second part consists of two steps:

[0010] Step 3, the main branch is to modify the ResNet18 structure: first, the input image data corresponds to the number of channels × image height × image width, and the corresponding size is 1×40×40. Due to the small size of the image, the input data is first subjected to a basic 3×3 convolution and then input to each layer for feature extraction. After that, it is passed to the fully connected layer after global average pooling for 7-category prediction of expression;

[0011] Step 4: First, the connection mode of the original ResNet18 network is simply modified in the secondary branch: the residual connection block of the main branch remains unchanged; from the perspective of multi-scale, a 1×1 convolution kernel is used to extract the features of the secondary branch, which can retain the information from the original input image to the maximum extent and have a positive impact on the final classification result;

[0012] The third part consists of two steps:

[0013] Step 5, add the adaptive feature fusion module ASFF to fuse the large amount of multi-scale expression information obtained by the feature extraction part, and fuse the features of different layers together by learning weight parameters to ensure the completeness of feature information and remove the generated feature redundancy to a certain extent;

[0014] Step 6: Output the results of each level of ASFF separately and make the final expression classification prediction;

[0015] The fourth part consists of a step:

[0016] Step 7: Replace the ordinary convolution with depth-wise separable convolution to reduce the huge amount of parameters caused by deepening the network.

[0017] Part 5 consists of two steps:

[0018] Step 8, debug the network structure hyperparameters from step 2 to step 7, set the network model parameters, where the main hyperparameters are set to 0.1 for the learning rate, 300 for the Epochs, and 64 for the Bach size, use the SGD optimizer, and obtain the final training model;

[0019] Step 9: Download the uncontrolled expression recognition dataset FER2013, and perform training and testing according to the trained model set in step 8.

[0020] The present invention provides a method for facial expression recognition using a dual-branch mixed residual connection. Aiming at the characteristics of inter-class similarity and intra-class difference in facial expression feature information, the method designs a dual-branch mixed residual connection mode to ensure the complete and accurate extraction of features; secondly, comprehensively analyzes the problems of network depth and width, fuses deep separable convolution, and constructs a lightweight facial expression recognition network; finally, according to the characteristics of facial expression information, in order to obtain more accurate and complete feature information, an adaptive feature fusion module is added, and the fused information is analyzed and verified, and the hierarchical output with the best performance is selected for the final facial expression classification prediction. The present invention can solve the problems of recognition accuracy of facial expression recognition under uncontrolled conditions and lightweight model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall framework diagram of the network of the present invention;

[0022] Figure 2 The double-branch mixed residual feature extraction network structure diagram of the present invention is integrated with the depth separable convolution;

[0023] Figure 3 This is the structural diagram of the adaptive feature fusion module ASFF of the present invention. DETAILED DESCRIPTION

[0024] In order to better understand the present invention, the following is a more detailed description of the feature extraction network combining adaptive feature fusion and dual-branch hybrid residual connection of the present invention in conjunction with specific embodiments. In the following description, the detailed description of the current prior art may dilute the subject content of the present invention, and these descriptions will be ignored here.

[0025] Figure 1 It is a specific network model diagram of the present invention combining adaptive feature fusion and dual-branch mixed residual connection feature extraction network. The MRS module is a dual-branch mixed residual module designed to integrate depth-separable convolution. In this embodiment, the following steps are followed:

[0026] Step 1: Randomly scale and crop the image, and then horizontally flip the image data to increase the image database;

[0027] Step 2: Use mixup image fusion and label_smoothing to further enhance the image data, so that the trained model can have better generalization ability. The specific implementation is as follows:

[0028] (1) The input size of the training data is 48×48, so the size is set to 40×40 for random cropping, and then the image is horizontally flipped to increase the training data;

[0029] (2) The mixup operation mainly fuses the two input images. By setting the value of α, the fusion ratio of the two images is controlled, and the fusion factor value is set to 1.0.

[0030] Step 3, build Figure 2 The dual-branch hybrid residual feature extraction network with fused depth-separable convolution shown in the figure has a main branch that modifies the ResNet18 structure: first, the input image data corresponds to the number of channels × image height × image width, and the corresponding size is 1×40×40. Due to the small size of the image, the input data is first subjected to a basic 3×3 convolution and then input into each layer for feature extraction. The convolution kernel dimension of each convolution layer is 3×3, and each convolution layer is nonlinearized by the ReLU activation function. Finally, after global average pooling, it is passed to the fully connected layer for 7-category prediction of expression.

[0031] Step 4: First, the secondary branch makes a simple modification to the connection mode of the original ResNet18 network: the residual connection block of the main branch remains unchanged; from the perspective of multi-scale, a 1×1 convolution kernel is used to extract the features of the secondary branch, which can retain the information from the original input image to the maximum extent and have a positive impact on the final classification result. The specific implementation is as follows:

[0032] (1) The input image data is downsampled from the other branch using a 1×1 convolution kernel size, and after fusing the feature information of the corresponding residual connection block, it is used as the input of the next 1×1 convolution until it is fused with the last layer of residual connection blocks;

[0033] (2) The feature information after the mixed residual operation is not sent to the main branch network for feature extraction, but is output as scale 1 feature separately, and finally connected with the residual information of the main residual block. Finally, the feature fusion of each scale information obtained by the main branch network and the scale 1 branch is performed, and the global feature is fused through the fully connected layer for expression classification;

[0034] Step 5: Build Figure 3 The adaptive feature fusion module ASFF shown in the figure adds the adaptive feature fusion module ASFF to fuse the features of different layers together by learning weight parameters to ensure the perfection of feature information and remove the generated feature redundancy to a certain extent;

[0035] Step 6: Combine the adaptive feature fusion module in step 5 with the outputs of different scale levels in step 4, output the results of each level of ASFF respectively, and make the final expression classification prediction. The specific implementation is as follows:

[0036] (1) First, extract three layers of different sizes from the feature extraction backbone network and select one of the layers Levell ( l ∈[0,2]), then for the other two levels Leveln ( n ∈[0,2], n≠l ) to upsample or downsample to the same size as Levell The same, then merge, and get the result after fusion ASFFl , select the first level result as the output, that is, the corresponding ASFF 0 output, the three levels of fusion are selected Level 0 corresponds to a size of 512×5×5, Level 1 corresponds to a size of 256×10×10, Level 2 corresponds to the size of 128×20×20;

[0037] (2) The three levels are adaptively fused through the ASFF module, and adaptive feature weights are generated for the feature maps of the three levels. The value range is controlled in [0, 1] through softmax. The final result is globally averaged and pooled for the final expression prediction:

[0038] The three-level feature maps are multiplied by the adaptive weights and then added to form ASFF 0.ASFF 1 、ASFF 2. The formula is as follows:

[0039]

[0040] in Representative from Leveln The layer features are resized to Levell After layer ( i, j ), Refers to the three differences of network adaptive learning Level arrive Levell The spatial importance weights of the feature maps ， ASFF 0 、ASFF 1 、ASFF 2 is the feature map output after spatial scale fusion. The final level output selection of the present invention is set to ASFF2 So choose ASFF 2. Give specific instructions: ASFF 2 is to choose from Level 0. Level 1. Feature map resize to Level 2 same size, where represent Level 0 feature map after reset, represent Level 1 Feature map after reset, Representatives from Level 2 feature map.

[0041] in To correspond to ASFF The weight parameters of each feature map in 2 are generated by first resizing the resized Level0_resized , Level1_resized ,as well as Level 2. Perform 1×1 convolution to reduce the channel dimension from 128 to 16, then concatenate the three feature maps into 48×H×W channels (H represents the height of the feature map, W represents the width of the feature map), and then perform 1×1 convolution to reduce the dimension to 3×H×W. Use 1×1 convolution to calculate the channels from Level0_ resized , Level1_resized , Level2 of Finally, softmax is performed in the channel dimension and Level0_ resized , Level1_resized , Level2 Multiplying the corresponding numbers gives:

[0042]

[0043] Step 7: Replace the ordinary convolution with depthwise separable convolution to reduce the huge amount of parameters caused by the deepening of the network. The specific implementation is as follows:

[0044] Since the number of channels has reached 512 after multiple downsampling, considering that an overly deep network will cause overfitting of expression recognition, and the depth-separable convolution is divided into channel convolution and point convolution, which deepens the number of network layers to a certain extent, and excessive use will damage the network accuracy, the present invention only replaces the last layer of convolution in the layer 4 of ResNet18 with the depth-separable convolution to reduce parameters;

[0045] Step 8, debug the network structure hyperparameters from step 2 to step 7, set the network model parameters, where the main hyperparameters are set to 0.1 for the learning rate, 300 for the Epochs, and 64 for the Bach size, use the SGD optimizer, and obtain the final training model;

[0046] Step 9, download the uncontrolled expression recognition dataset FER2013, and train and test it according to the trained model set in step 8. The recognition accuracy of the model without replacing the depthwise separable convolution is 73.36%. After replacing the depthwise separable convolution, the recognition accuracy is only reduced by 0.02%, but the model parameters will be reduced by nearly 5M.

[0047] The present invention is based on a deep neural network and provides an adaptive hybrid residual connection lightweight network for expression recognition. In order to improve the expression feature information extraction mechanism, the method designs a dual-branch multi-residual connection mode: the main branch is built according to the ResNet18 architecture, and the deep separable idea in the Xception architecture is learned. According to the training task characteristics of the present invention, the deep separable convolution is integrated as the feature extraction backbone network; the secondary branch starts from different scales, and the input part is multi-residual connected to ensure that the extracted feature information is complete and accurate. The 1×1 convolution kernel is selected so that the feature information can retain the information from the original input picture to the maximum extent, and the subsequent category prediction is assisted to ensure the integrity of the feature information. Secondly, the network depth and width are comprehensively analyzed, the deep separable convolution is integrated, and a lightweight expression recognition network is constructed. Finally, the adaptive feature fusion module ASFF is connected, the feature information of different levels is integrated, and a complete expression recognition network is constructed. After the three-level output is verified and analyzed, the level output with better response to the category feature is selected for the final expression classification. The present invention realizes an adaptive hybrid residual connection lightweight network for expression recognition based on deep learning, which has a strong application value.

[0048] Although the above describes an illustrative specific embodiment of the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiment. For ordinary technicians in this technical field, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A facial expression recognition method using a dual-branch mixed residual connection, characterized in that: A dual-branch feature extraction network is introduced, a hybrid residual connection method is used to improve the feature extraction mechanism, and an adaptive feature fusion module ASFF is added to combine the extracted feature information, and finally the model is lightweight; it includes five parts: basic image data enhancement, dual-branch hybrid feature extraction, adaptive feature fusion ASFF, model lightweight and experimental testing: The first part consists of two steps: Step 1: Randomly scale and crop the image, and then horizontally flip the image data to increase the image database; Step 2: Use mixup image fusion and label_smoothing label smoothing to further enhance the image data, so that the trained model has better generalization ability; The second part consists of two steps: Step 3, the main branch is to modify the ResNet18 structure: first, the input image data corresponds to the number of channels × image height × image width, and the corresponding size is 1×40×40. Due to the small size of the image, the input data is first subjected to a basic 3×3 convolution, and then input to each layer for feature extraction, and then passed to the fully connected layer after global average pooling for 7-category prediction of expression; Step 4: First, the connection method of the original ResNet18 network is simply modified in the secondary branch: the residual connection block of the main branch remains unchanged; from the multi-scale aspect, a 1×1 convolution kernel is used to extract the features of the secondary branch. The specific implementation is as follows: (1) The input image data is downsampled from the other branch using a 1×1 convolution kernel size, and after fusing the feature information of the corresponding residual connection block, it is used as the input of the next 1×1 convolution until it is fused with the last layer of residual connection blocks; (2) The feature information after the mixed residual operation is not sent to the main branch network for feature extraction, but is output as scale 1 feature separately, and finally connected with the residual information of the main residual block; finally, the feature information of each scale obtained by the main branch network and the scale 1 branch is fused, and the global feature is fused through the fully connected layer for expression classification; The third part consists of two steps: Step 5, add the adaptive feature fusion module ASFF to extract the multi-scale expression information obtained by the feature extraction part, and fuse the features of different layers together by learning weight parameters to ensure the complete feature information; Step 6: Output the results of each level of ASFF separately and make the final expression classification prediction; the specific implementation is as follows: (1) First, three layers of different sizes are extracted from the feature extraction backbone network. After selecting one of the layers, the other two layers are upsampled or downsampled to make the sizes of the three layers consistent. Then, the first layer is selected as the output, which is the corresponding ASFF0 output. The three fused layers are Level_0 with a size of 512×5×5, Level_1 with a size of 256×10×10, and Level_2 with a size of 128×20×20. (2) The three layers are adaptively fused through the ASFF module, and adaptive feature weights are generated for the feature maps of the three layers. The value range is controlled in [0, 1] through softmax. The final result is subjected to global average pooling for the final expression prediction. The fourth part consists of a step: Step 7: Replace the ordinary convolution with depthwise separable convolution to reduce the huge amount of parameters caused by the deepening of the network. The specific implementation is as follows: Since the number of channels has reached 512 after multiple downsampling, the last convolution layer in layer 4 of ResNet18 is replaced with a depth-wise separable convolution to reduce parameters; Part 5 consists of two steps: Step 8, debug the network structure hyperparameters from step 2 to step 7, set the network model parameters, where the main hyperparameters learning rate is set to 0.1, Epochs is set to 300, Bach size is set to 64, SGD optimizer is used, and the final training model is obtained; Step 9: Download the uncontrolled expression recognition dataset FER2013, and perform training and testing according to the trained model set in step 8.

2. The expression recognition method of a dual-branch hybrid residual connection according to claim 1 is characterized in that: Step 4 uses a dual-branch hybrid residual connection network for feature extraction.

3. The expression recognition method of a dual-branch hybrid residual connection according to claim 1 is characterized in that: Step 6 uses the adaptive feature fusion module ASFF to fuse the different scale information of the feature extraction network and generate adaptive weights.

4. The expression recognition method of a dual-branch hybrid residual connection according to claim 1 is characterized in that: In step 7, the last convolution layer in layer 4 of ResNet18 is replaced with a depth-wise separable convolution to reduce parameters and make the model lightweight.

Images