A facial expression recognition method based on feature restoration and self-enhancement
Through the de-whitening, feature recombination and feature sharing modules in the feature optimization network, the problems of filling erosion and small differences between classes in facial expression recognition are solved, and higher recognition accuracy and recognition rate are achieved.
Patent Information
- Application Number
- CN202210132610.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-14
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-02-14
AI Technical Summary
Existing facial expression recognition algorithms are difficult to improve the recognition rate on a limited data set, and the filling of convolutional neural networks leads to erosion of feature information, affecting the recognition effect.
A facial expression recognition method based on feature repair and self-enhancement is proposed. By building a feature optimization network, using de-whitening module, feature recombination module and feature sharing module, remove external information, gather feature pixels with the same degree of erosion, and decompose expressions The characteristics are two parts: face and expression, which simplifies the expression of features.
It effectively reduces the impact of filling erosion on feature information, improves the accuracy of recognition of differences between expressions, and improves the recognition rate, especially on small-scale data sets.
Smart Images

Figure CN114495237B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of facial expression recognition, and in particular relates to a facial expression recognition method based on feature restoration and self-enhancement. Background Art
[0002] In recent years, deep learning has made great progress and has gradually become the main research method for facial expression recognition. Researchers have proposed many algorithms based on classic convolutional neural networks to overcome various challenges in the field of facial expression recognition. However, due to the limitation of the size of the data set, these algorithms cannot fully utilize the advantages of neural networks. Building a larger data set requires expensive annotation costs. How to improve the recognition rate on a limited data set has become the most important problem at present. To solve this problem, this paper conducts an in-depth study of facial expression recognition from two aspects: data and network.
[0003] There are 1,000 objects in the ImageNet dataset, and the differences between different categories are usually quite obvious. However, the objects of facial expression recognition are all face images, and the feature differences between different expressions are small. Even for the same expression, different people's facial expressions are different, which makes it very difficult to accurately classify each category of expression.
[0004] Early convolutional neural networks were not perfect, so Hinton and others proposed using padding to assist convolution calculations. On the one hand, padding can expand the area of the feature map, solving the problem of feature map shrinkage caused by convolution, allowing the convolutional neural network to be further deepened; on the other hand, padding makes the originally peripheral pixels relatively move inward, balancing the perceived frequency between pixels. Padding has the above advantages, but it also brings new problems; in the image field, pixel values have color meanings. Although researchers usually default to using 0 as the padding value, the black information represented by 0 is obviously also external information; these external information penetrates into the new features from the outside to the inside through convolution calculations, and as the feature information becomes more refined, the proportion of external information gradually increases. These problems make the erosion effect of padding on feature information not to be ignored. Summary of the invention
[0005] To solve the above-mentioned problems, it is necessary not only to consider the particularity of facial expression data, but also to solve the negative impact of filling erosion. The present invention proposes a facial expression recognition method based on feature restoration and self-enhancement, which considers and solves the negative impact caused by convolution filling, and uses a new facial expression representation learning strategy to solve the problem of small inter-class differences.
[0006] The method for facial expression recognition based on feature restoration and self-enhancement described in the present invention is characterized in that the steps of the method are:
[0007] S1. Build a feature optimization network and use the convolutional layer of ResNet-18 to extract 7*7*512 high-dimensional features;
[0008] S2. Extract representation information from high-dimensional features using a feature optimizer, wherein the feature optimizer includes a dewhitening module, a feature recombination module, and a feature sharing module; perform recombination preprocessing on the high-dimensional features extracted by ResNet-18 using the feature recombination module (the preprocessing is to enhance the dewhitening effect), and then generate pure high-dimensional features through the dewhitening module after recombination, and then extract representation information through the feature sharing module;
[0009] S3. Finally, the high-dimensional representation processed by the three modules is flattened, and the expression category is predicted using a fully connected layer.
[0010] Furthermore, the feature reorganization module gathers feature pixels with the same erosion degree and reorganizes the multi-channel features in a single channel. The method comprises the steps of:
[0011] S2-1-1. If the high-dimensional features satisfy the common multiple of 4 channels, divide the H*W*C features into Group H*W*4-dimensional features, where H, W, and C are the height, width, and number of channels of the feature map matrix respectively;
[0012] S2-1-2, clustering and reorganizing the pixels distributed in the four channels at each position of a single group of features into H*W 2*2 dimensional feature clusters;
[0013] H*W 2*2 single-channel feature clusters are reorganized into (H*2)*(W*2)*1-dimensional features according to the original feature pixel positions in a single channel;
[0014] S2-1-3, The (H*2)*(W*2)*1-dimensional features are concatenated along the channel dimension Dimensional features;
[0015] S2-1-4, repeat S2-2-1 to S2-2-3 until the number of channels of the high-dimensional feature does not meet the multiple of 4 and the iteration cannot continue;
[0016] After being processed by the feature recombination module, the final 7*7*512 high-dimensional features are reorganized into 112*112*2 high-dimensional features.
[0017] Furthermore, the de-whitening module filters out extraneous information from the whitening features and extracts pure representations, and the steps are as follows:
[0018] S2-2-1. Perform erosion analysis on the high-dimensional features extracted by the convolutional layer of ResNet-18;
[0019] S2-2-2, using perceptual bias to assign weights to feature pixels at different locations, reducing the weights of peripheral eroded pixels and increasing the weights of central key pixels;
[0020] S2-2-3. The ideal result of the feature reorganization module should be that all feature pixels are arranged in a single channel and processed by sliding the dewhitening convolution kernel of the single channel. Although the final reorganization actually obtains the feature map of two channels, the status of the two channels should be regarded as equal. Therefore, the dewhitening convolution kernel of a single channel is used to slide the two channels in turn, and the average value of the feature map after convolution processing is taken as the final output of the dewhitening module.
[0021] Furthermore, the feature sharing module decomposes the expression features into two parts: the common features of facial components F generic and the difference between the expression components F differential :
[0022] F expression =F generic +F differential (1)
[0023] The first batch F during training init Extract the average feature as the common feature initial value, and the subsequent batch F batch Use exponential moving average to slowly update common features:
[0024]
[0025] F generic =λF batch +(1-λ)F generic (3)
[0026] Among them, F batch and F' generic Represents the average feature of the current batch and the common feature F after the previous batch iteration generic , the smoothing factor λ between 0 and 1 indicates the degree of weight reduction.
[0027] The beneficial effects described in the present invention are as follows: the method described in the present invention uses a feature optimizer composed of a dewhitening module, a feature recombination module and a feature sharing module. In the representation extraction stage, the dewhitening module filters external information from the whitened features to extract pure representations; without changing the relative positions, the feature recombination module gathers pixels with the same erosion degree, which is beneficial to enhancing the effect of the dewhitening module; the feature sharing module splits the expression features into two parts, namely, a face and an expression, which not only simplifies the feature expression, but also enables the network to focus on the subtle differences between different expressions, effectively coping with the problem of small differences between classes. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The end-to-end framework diagram of the facial expression recognition method based on feature restoration and self-enhancement according to an embodiment of the present invention is shown;
[0029] Figure 2 This is a schematic diagram of the principle of a de-whitening module described in an embodiment of the present invention;
[0030] Figure 3 This is a schematic diagram of the principle of a part-feature recombination module according to an embodiment of the present invention;
[0031] Figure 4 The experimental results of the embodiments of the present invention on Raf-DB;
[0032] Figure 5 These are the experimental results of the embodiments of the present invention on AffectNet. DETAILED DESCRIPTION
[0033] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.
[0034] Combination Figure 1 As shown in the figure, the proposed feature optimization network includes the following three parts: the convolution layer part of the ResNet-18 network, the feature optimization network and the fully connected layer, and the ImageNet dataset is used to initialize the weights of the remaining network parameters of ResNet-18. The detailed information of the network is shown in Table 1, which is a schematic diagram of the specific parameters of the network structure of an embodiment of the present invention, where ResNet-18* refers to the network before the pooling layer of ResNet-18, and c, k, and s represent the number of channels, size, and step size of the convolution kernel, respectively.
[0035] Table 1: Parameters of each module of the feature optimization network
[0036]
[0037] The feature optimization network first uses the convolutional layer of ResNet-18 to extract high-dimensional features, then uses three modules to eliminate external information interference from the whitened features and extract pure representation information, and finally uses the fully connected layer to predict the expression category.
[0038] 1. Feature Extractor
[0039] The feature extraction part of ResNet-18 has 18 layers of convolution, and padding is used in each layer of convolution. Padding adds zeros to the periphery of the feature map, but the convolution calculation does not consider the impact of the information introduced by padding, which causes external information to interfere with the new features. When calculating the value of a new pixel, if the receptive field of the convolution kernel contains padded pixels, the new pixel is said to be eroded by padding, and the greater the proportion of padded pixels in the receptive field, the more severe the erosion. This erosion effect exists in all layers, and the previous influences continue to accumulate and propagate backward, seriously eroding the final high-dimensional features. Due to the influence of padding erosion, it is also called whitening feature.
[0040] 2. Feature Recombination Module
[0041] From a spatial perspective, the more pixels are eroded, the more serious they are. Since the spatial area of high-dimensional features is often compact, the erosion changes from the outside to the inside are different, making it difficult to process them uniformly. From a channel perspective, pixels at the same position in each channel are eroded to the same extent, so pixels with the same degree of erosion are evenly distributed in each channel. The scattered distribution is not conducive to uniform processing. The dewhitening module achieves weight allocation by adjusting the size of the convolution kernel, and the small spatial dimension of high-dimensional features makes it impossible to fine-tune the adjustment. The small spatial dimension not only limits the adjustment space of dewhitening, but also easily over-adjusts, resulting in damage to the core information. Therefore, pixels with the same degree of erosion are clustered in a single channel to expand the spatial dimension of the feature map, which is conducive to fine-tuning and finding the optimal value.
[0042] The feature reorganization module gathers feature pixels with the same erosion degree and reorganizes multi-channel features in a single channel. The method comprises the steps of:
[0043] (1) If the feature satisfies the common multiple of 4 channels, the features of H*W*C are divided into Group H*W*4-dimensional features, where H, W, and C are the height, width, and number of channels of the feature map matrix respectively;
[0044] (2) Figure 3 As shown, the pixels distributed in the four channels at each position of a single set of features are clustered and reorganized into H*W 2*2 dimensional feature clusters; Figure 3 In , pixels at the same position in each channel are marked with the same color, and different colors represent different feature clusters;
[0045] H*W 2*2 single-channel feature clusters are reorganized into (H*2)*(W*2)*1-dimensional features according to the original feature pixel positions in a single channel;
[0046] (3) The (H*2)*(W*2)*1-dimensional features are concatenated along the channel dimension Dimensional features;
[0047] (4) Repeat (1)(2)(3) until the number of channels of the high-dimensional feature is no longer a multiple of 4. The output feature of ResNet-18 is 7x7x512. According to the principle of feature recombination, the number of channels is not a multiple of 4 at this time, so only a two-channel feature map can be obtained. As the number of channels of the feature map is reduced to 2, the feature map space is correspondingly increased to 112x112, and finally a 112x112x2 dimensional recombined feature is obtained. The point set composed of feature pixels at the same position of each channel in the original feature map is called a feature cluster, with a size of 16x16x1.
[0048] 3De-whitening module
[0049] Erosion analysis of high-dimensional features extracted by ResNet-18: Peripheral pixels are greatly affected by external information during convolution calculations. For example, when calculating the value of the pixel in the upper left corner of the feature, the number of padded pixels in the receptive field of the convolution kernel is close to the image pixels; internal pixels are less affected by external information, for example, when calculating the pixel value at the center of the feature, there are no padded pixels in the receptive field.
[0050] The perceptual bias is used to allocate weights to feature pixels at different positions: reducing the weight of peripheral eroded pixels and increasing the weight of central key pixels, which can effectively solve the impact of eroded whitening features on representation.
[0051] Perceptual bias refers to the fact that when no padding is used, the convolution kernel perceives different positions within a feature at different times: the edge of the feature is perceived less frequently than the center. Figure 2 As shown in the figure, the application diagram of perceptual bias in 7x7 feature map, where the convolution kernel size and step size are 3 and 1 respectively. Each grid in the figure represents a pixel, and the number in the pixel indicates the number of times the pixel is perceived by the convolution kernel. The number of perceptions is positively correlated with the size of the weight, so perceptual bias can realize weight allocation for different positions. Adjusting the size and step size of the convolution kernel can achieve qualitative adjustment of perceptual bias.
[0052] If the convolution kernel and feature cluster are of the same size, a single feature cluster will dominate the range of the convolution kernel, reducing diversity. Therefore, the convolution kernel size is set to 32x32x1, which is much larger than the feature cluster; at the same time, the convolution kernel step size is set to half the size of the feature cluster, that is, 8.
[0053] Although the reorganized feature map is not in a single channel as expected, its two channels should still be considered equal. Therefore, the de-whitening module should use a single-channel convolution kernel to perform convolution on the two channels in turn.
[0054] The de-whitened features are batch normalized, and finally the channel dimension average is used as the output of the de-whitening module.
[0055] 4 Feature Sharing Module
[0056] During forward propagation, traditional convolutional neural networks usually need to learn to express complete image features. However, expression data are all based on images of human faces, and there are only slight differences between different expressions. The task of expressing complete expression features while taking into account subtle differences makes this task very challenging. Therefore, in order to achieve better results, a large number of iterative training is usually required. By analyzing facial expression data, we proposed a feature learning strategy with "inherited thinking", which uses a new expression representation method to reduce the learning difficulty and accelerate the convergence speed.
[0057] Facial expression data is based on face images. The feature sharing module decomposes the expression features into two parts: the common features of facial components F generic and the difference between the expression components F differential :
[0058] F expression =F generic +F differential (1)
[0059] By maintaining a set of universal facial features, the network does not need to learn to express complete facial features, but only needs to focus on the subtle differences between different categories. This method reduces the difficulty of feature expression, improves the training convergence speed, and enables the classic network to perform better on small-scale data sets.
[0060] During training, the first batch extracts the average feature as the initial value of the common feature, and the subsequent batches use the exponential moving average (EMA) to slowly update the common features:
[0061]
[0062]
[0063] Among them, F batch and F' generic Represents the average feature of the current batch and the common feature F after the previous batch iteration generic The smoothing factor λ between 0 and 1 indicates the degree of weight reduction, and a larger λ will cover old observations faster. The smoothing factor λ depends on the dataset. By default, it is initialized to 0.3 and updated according to the gradient descent trend.
[0064] 5 Fully connected layers
[0065] The parameters of the fully connected layer are determined according to the outputs of the three modules of the feature extractor and the number of expression categories.
[0066] To optimize the network parameters, the Adam optimizer with a small batch size of 256 was used. The parameter settings for the Adam optimizer during training were as follows: the initial value of the learning rate was 0.001, and a decay learning rate strategy of 0.9 was used.
[0067] In summary, the present invention proposes a feature optimization network to solve the potential filling erosion problem and simplifies the expression representation learning by using expression feature decomposition. Figure 4 , 5 The recognition confusion matrices of this embodiment on RAF-DB and AffectNet are respectively, where the abbreviations of the expression categories are: SU-surprise, FE-fear, DI-disgust, HA-happiness, SA-sadness, AN-anger and NE-neutral. Each row in the confusion matrix represents the recognition distribution of samples of this type, and the diagonal of the confusion matrix is the recognition accuracy of each expression category. Combined with the experimental results on the public datasets RAF-DB and AffectNet shown in Table 2, it is shown that the method proposed in the present invention achieves excellent recognition performance.
[0068] Table 2: Recognition accuracy of feature optimization network implementation example (%)
[0069]
[0070] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.
Claims
1. A facial expression recognition method based on feature restoration and self-enhancement, characterized in that: The steps of the method are: S1. Build a feature optimization network and use the convolutional layer of ResNet-18 to extract 7*7*512 high-dimensional features; S2. Extract representation information from high-dimensional features using a feature optimizer, wherein the feature optimizer includes a dewhitening module, a feature recombination module, and a feature sharing module; the high-dimensional features extracted by ResNet-18 are reorganized and preprocessed using the feature recombination module, and after the recombination, pure high-dimensional features are generated by the dewhitening module, and then the representation information is extracted by processing with the feature sharing module; S3, finally flatten the high-dimensional representation processed by the three modules in sequence, and use the fully connected layer to predict the expression category; The feature reorganization module gathers feature pixels with the same erosion degree and reorganizes multi-channel features in a single channel. The method comprises the steps of: S2-1-1. If the high-dimensional features satisfy the common multiple of 4 channels, divide the H*W*C features into Group H*W*4-dimensional features, where H, W, and C are the height, width, and number of channels of the feature map matrix respectively; S2-1-2, clustering and reorganizing the pixels distributed in the four channels at each position of a single group of features into H*W 2*2 dimensional feature clusters; H*W 2*2 single-channel feature clusters are reorganized into (H*2)*(W*2)*1-dimensional features according to the original feature pixel positions in a single channel; S2-1-3, The (H*2)*(W*2)*1-dimensional features are concatenated along the channel dimension Dimensional features; S2-1-4, repeat S2-2-1 to S2-2-3 until the number of channels of the high-dimensional feature does not meet the multiple of 4 and the iteration cannot continue; After being processed by the feature reorganization module, the final 7*7*512 high-dimensional features are reorganized into 112*112*2 high-dimensional features; The de-whitening module filters out extraneous information from the whitened features and extracts pure representations, and the steps are as follows: S2-2-1. Perform erosion analysis on the high-dimensional features extracted by the convolutional layer of ResNet-18; S2-2-2. Use the perceptual bias of convolution to assign weights to feature pixels at different positions, reduce the weights of peripheral eroded pixels, and increase the weights of central key pixels; S2-2-3. The final reorganization results in a two-channel feature map. The two channels are considered to be of equal status. A single-channel dewhitening convolution kernel is used to slide the two channels in turn, and the channel average value of the feature map after convolution is taken as the final output of the dewhitening module.
2. A facial expression recognition method based on feature restoration and self-enhancement according to claim 1, characterized in that: The feature sharing module transforms the expression feature F expression Decomposed into two parts: the general features F of facial components generic and the difference between the expression components F differential : F expression =F generic +F differential (1) The first batch F during training init Extract the average feature as the initial value of the common feature, and then the batch feature F batch Use exponential moving average to slowly update common features: F generic =λF batch +(1-λ)F' generic (3) f i F is the pure feature of the facial expression image after feature reorganization and de-whitening module. batch and F' generic Represents the average feature of the current batch and the common feature F after the previous batch iteration generic , the smoothing factor λ between 0 and 1 indicates the degree of weight reduction.
Citation Information
Patent Citations
Depth convolution wavelet neural network expression identification method based on auxiliary task
CN107292256A
Lightweight unconstrained facial expression recognition method and system embedded with high-order information
CN112801040A
Facial expression recognition method based on multi-channel fusion and lightweight neural network
CN113989890A
Efficient image classification method based on structured pruning
US20210049423A1
Face recognition method and apparatus, device, and storage medium
WO2021073418A1
