A method and device for recognizing facial expression

Abstract

This application relates to a facial expression recognition method and apparatus, the specific steps of which include: preprocessing a facial image; extracting features from the preprocessed facial image using an improved octave convolutional neural network to obtain a depth feature map of the facial image; inputting the depth feature map into an adaptive pooling layer for downsampling to obtain an average pooling feature map; and sequentially inputting the average pooling feature map into a fully connected layer and a softmax classification layer for classification and recognition to obtain an expression recognition result. The facial expression recognition method and apparatus described in this application, without increasing the number of parameters, reduces low-frequency redundancy and computational load, and improves feature extraction capability; the dynamic activation function used can adaptively generate optimal coefficients based on the input feature map, effectively improving the model's expressive power.

Description

Technical Field

[0001] The present application relates to the field of facial expression recognition, and in particular to a method and device for facial expression recognition. Background Art

[0002] Facial expressions are the most effective and common information for conveying emotional states. In recent years, the field of artificial intelligence has developed rapidly, and facial expression recognition has a wide range of application scenarios, such as in human-computer interaction, smart classrooms, and smart driving. Therefore, the research on expression recognition is of great significance.

[0003] At present, facial expression recognition methods are generally divided into two categories: traditional methods and deep learning methods. Among them, most traditional methods use manual features or shallow learning, such as local binary patterns (LBP), principal component analysis, non-negative matrix decomposition, Gabor wavelet transform, Histogram of Oriented Gradient (HOG), etc. These methods can achieve good experimental results in certain specific situations, but are easily affected by other factors, such as lighting, angle, posture factors, and are not very robust in real scenes.

[0004] With the development of deep learning technology and the gradual enrichment of face data sets, deep learning methods are gradually applied to facial expression recognition research. Convolutional Neural Networks (CNN) can solve the problem of subjective independence in facial expressions and the problem of facial expression position and angle in pictures; regional CNN (R-CNN) can learn facial expression features; fusing Visual Geometry Group Network (VGG) and Residual Network (ResNet) can improve model performance; however, these methods all have problems such as complex network model structure, poor feature extraction ability, many parameters, and large amount of calculation. Summary of the invention

[0005] Based on this, the purpose of this application is to provide a facial expression recognition method and device to reduce low-frequency redundancy, reduce the amount of calculation, improve feature extraction capabilities, and enhance the nonlinear expression capabilities of the model.

[0006] A method for recognizing facial expressions, comprising the following steps:

[0007] Preprocessing of face images;

[0008] Using an improved octave convolutional neural network to extract features from the preprocessed face image to obtain a deep feature map of the face image; wherein the improved octave convolutional neural network includes an improved octave convolution block and a maximum pooling layer connected in sequence, the improved octave convolution block includes two octave convolution layers, and each of the eight convolution layers includes an improved octave convolution module, batch normalization, and a dynamic activation function;

[0009] The improved octave convolution module divides the image input to the improved octave convolution module into a high-frequency part image and a low-frequency part image, the resolution of the high-frequency part image remains unchanged, and the low-frequency part image is input into the maximum pooling layer with a pooling kernel size of 2×2 and a step size of 2 in the improved octave convolution module for downsampling to obtain a low-frequency image with a resolution half of the input image; the improved octave convolution module processes the high-frequency part image and the low-frequency image obtained by downsampling according to the following formula, respectively inputs the high-frequency part image and the low-frequency image into different convolution kernel channels of the improved octave convolution module, performs convolution operation according to the convolution weights obtained by training to obtain a high-frequency convolution feature map and a low-frequency convolution feature map, performs nearest neighbor interpolation upsampling on the low-frequency convolution feature map to obtain a low-frequency upsampled feature map with the same resolution as the high-frequency convolution feature map, and splices the high-frequency convolution feature map and the low-frequency upsampled feature map on different convolution kernel channels of the improved octave convolution module to output a fused feature map;

[0010]

[0011] Among them, X l represents the image input to the improved octave convolution module, X l+1 represents the output fusion feature map, (i, j) represents the pixel of the image (i, j), An image representing a high-frequency portion of an image input into the improved octave convolution module; Represents the low-frequency part of the image input to the improved octave convolution module, which is X l Obtained by downsampling through the maximum pooling layer in the improved octave convolution module; represents the convolution weight of the high-frequency part of the image, represents the convolution weight of the low-frequency image, a represents the component ratio of the low-frequency part of the image, upsimple represents upsampling, represents the convolution operation, Indicates a splicing operation; X l+1 (i, j) where (i, j)∈{1, 2, ...L l+1}, L l+1 For X l+1 Size, L l For Xl The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size;

[0012] The batch normalization adjusts the fused feature map output by the improved octave convolution module to a standard Gaussian distribution;

[0013] The dynamic activation function adaptively generates the optimal coefficient according to the batch-normalized fusion feature map, performs nonlinear mapping on the batch-normalized fusion feature map, and obtains a nonlinear mapping feature map; the dynamic activation function is a DyReLU function, which is shared across space and not shared across channels, and its calculation formula is:

[0014]

[0015] Among them, y i Represents the nonlinear mapping feature map of the i-th channel output, x i a represents the batch-normalized fused feature map of the i-th channel input; i and b i is the coefficient corresponding to the i-th channel, which is calculated and output by the auxiliary function θ(x);

[0016] The maximum pooling layer downsamples the nonlinear mapping feature map output by the dynamic activation function to obtain the depth feature map;

[0017] Input the depth feature map into the adaptive pooling layer for downsampling to obtain an average pooling feature map;

[0018] The average pooled feature map is sequentially input into the fully connected layer and the softmax classification layer for classification and recognition to obtain the expression recognition result.

[0019] A facial expression recognition method described in the present application uses an improved octave convolution module to convolve the preprocessed facial image, reduces low-frequency redundancy and calculation amount, and improves feature extraction capability without increasing the number of parameters; the dynamic activation function used can adaptively generate the optimal coefficient according to the batch-normalized fusion feature map, effectively improving the expression ability of the model.

[0020] Furthermore, the preprocessing of the face image includes grayscale, scale adjustment, cropping and mirroring.

[0021] Furthermore, the improved octave convolutional neural network includes four improved octave convolution blocks and four maximum pooling layers, which are respectively the first octave convolution block, the first maximum pooling layer, the second octave convolution block, the second maximum pooling layer, the third octave convolution block, the third maximum pooling layer, the fourth octave convolution block and the fourth maximum pooling layer connected in sequence.

[0022] Furthermore, the number of convolution kernels of the octave convolution module of the first octave convolution block is 64, the convolution kernel size is 3*3, the convolution step is 1, and the boundary padding is 1; the number of convolution kernels of the octave convolution modules of the second, third, and fourth octave convolution blocks are 128, 256, and 256 respectively, and other parameters are consistent with the first octave convolution block.

[0023] Furthermore, the pooling kernel sizes of the four maximum pooling layers are all 2*2, and the convolution step sizes are all 2.

[0024] Furthermore, before the average pooling feature map is input into the fully connected layer, the method further includes inputting the average pooling feature map into a Flatten layer for flattening to obtain a one-dimensional vector which is input into the fully connected layer.

[0025] The present application also provides a facial expression recognition device, characterized in that it includes:

[0026] A preprocessing unit, used for preprocessing the face image;

[0027] An octave convolutional neural network unit, used to extract features from a preprocessed face image using an improved octave convolutional neural network to obtain a depth feature map of the face image; wherein the improved octave convolutional neural network comprises an improved octave convolution block and a maximum pooling layer connected in sequence, the improved octave convolution block comprises two octave convolutional layers, and each of the eight convolutional layers comprises an improved octave convolution module, batch normalization, and a dynamic activation function;

[0028] The improved octave convolution module divides the image input to the improved octave convolution module into a high-frequency part image and a low-frequency part image, the resolution of the high-frequency part image remains unchanged, and the low-frequency part image is input into the maximum pooling layer with a pooling kernel size of 2×2 and a step size of 2 in the improved octave convolution module for downsampling to obtain a low-frequency image with a resolution half of the input image; the improved octave convolution module processes the high-frequency part image and the low-frequency image obtained by downsampling according to the following formula, respectively inputs the high-frequency part image and the low-frequency image into different convolution kernel channels of the improved octave convolution module, performs convolution operation according to the convolution weights obtained by training to obtain a high-frequency convolution feature map and a low-frequency convolution feature map, performs nearest neighbor interpolation upsampling on the low-frequency convolution feature map to obtain a low-frequency upsampled feature map with the same resolution as the high-frequency convolution feature map, and splices the high-frequency convolution feature map and the low-frequency upsampled feature map on different convolution kernel channels of the improved octave convolution module to output a fused feature map;

[0029]

[0030] Among them, X lrepresents the image input to the improved octave convolution module, X l+1 represents the output fusion feature map, (i, j) represents the pixel of the image (i, j), An image representing a high-frequency portion of an image input into the improved octave convolution module; Represents the low-frequency part of the image input to the improved octave convolution module, which is X l Obtained by downsampling through the maximum pooling layer in the improved octave convolution module; represents the convolution weight of the high-frequency part of the image, represents the convolution weight of the low-frequency image, a represents the component ratio of the low-frequency part of the image, upsimple represents upsampling, represents the convolution operation, Indicates a splicing operation; X l+1 (i, j) where (i, j)∈{1, 2, ...L l+1}, L l+1 For X l+1 Size, L l For X l The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size;

[0031] The batch normalization adjusts the fused feature map output by the improved octave convolution module to a standard Gaussian distribution;

[0032] The dynamic activation function adaptively generates the optimal coefficient according to the batch-normalized fusion feature map, performs nonlinear mapping on the batch-normalized fusion feature map, and obtains a nonlinear mapping feature map; the dynamic activation function is a DyReLU function, which is shared across space and not shared across channels, and its calculation formula is:

[0033]

[0034] Among them, y i Represents the nonlinear mapping feature map of the i-th channel output, x i a represents the batch-normalized fused feature map of the i-th channel input; i and b i is the coefficient corresponding to the i-th channel, which is calculated and output by the auxiliary function θ(x);

[0035] The maximum pooling layer downsamples the nonlinear mapping feature map output by the dynamic activation function to obtain the depth feature map;

[0036] An average pooling layer unit, used for inputting the depth feature map into the adaptive pooling layer for downsampling to obtain an average pooling feature map;

[0037] The fully connected layer and the softmax classification layer unit are used to input the average pooling feature map into the fully connected layer and the softmax classification layer in sequence for classification and recognition to obtain the expression recognition result.

[0038] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a facial expression recognition method according to an embodiment of the present invention;

[0040] Figure 2 Octave convolution block model structure diagram of the facial expression recognition method provided in the embodiment of the present application;

[0041] Figure 3 A model structure diagram of an octave convolution module of a facial expression recognition method provided in an embodiment of the present application;

[0042] Figure 4 It is the traditional ReLU static activation function image;

[0043] Figure 5 The DyReLU dynamic activation function image of the facial expression recognition method provided in the embodiment of the present application;

[0044] Figure 6 A model structure diagram of a facial expression recognition method provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of the structure of a facial expression recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application is further described below in conjunction with the accompanying drawings and examples. The implementation methods of the present application include but are not limited to the following examples.

[0047] like Figure 1 As shown, a facial expression recognition method, the specific steps are as follows:

[0048] S1. Preprocess the face image.

[0049] The preprocessing of the face image includes graying, scaling, cropping and mirroring.

[0050] Specifically, the grayscale conversion is to convert a color face image into a grayscale face image, which can reduce the dimension of the input matrix, improve the subsequent running speed, and retain the gradient information; the grayscale face image is scaled, and then part of the face image is cropped out; the cropped image is mirrored.

[0051] After the grayscale face image is scaled, a portion of the face image may be cropped from at least one of the upper left corner, the upper right corner, the center, the lower left corner, and the lower right corner of the face image.

[0052] The mirroring method of the mirroring process can be to mirror the image on the left side of the horizontal center line of the image to the right side of the horizontal center line, or to mirror the image on the right side of the horizontal center line of the image to the left side of the horizontal center line, horizontal flipping, etc.

[0053] S2. Use the improved Octave convolutional neural network to extract features from the preprocessed face image to obtain a deep feature map of the face image.

[0054] The improved octave convolutional neural network includes an improved octave convolutional block and a maximum pooling layer connected in sequence.

[0055] like Figure 2 As shown, the improved octave convolution block (Octave Block) includes two octave convolution layers, each of which includes an improved octave convolution module (Octave Module), batch normalization (Batch Normalization, BN) and a dynamic activation function (DyReLU).

[0056] The core of the octave convolution module is the convolution operation. The calculation formula of the traditional convolution operation on the feature map is as follows:

[0057]

[0058] Among them, X l represents the input image, X l+1 represents the output feature map, (i, j) represents the pixel of the image at (i, j), W l represents the k×k convolution kernel, Represents the convolution operation. X l+1 (i, j) where (i, j)∈{1, 2, ...L l+1}, L l+1 For X l+1 Size, L l For X l The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size.

[0059] like Figure 3As shown, the improved octave convolution module described in the present application divides the image input to the improved octave convolution module into a high-frequency part image and a low-frequency part image, the resolution of the high-frequency part image remains unchanged, and the low-frequency part image is input into the improved octave convolution module with a pooling kernel size of 2×2 and a maximum pooling layer with a step size of 2 for downsampling to obtain a low-frequency image with a resolution of half the input image; the improved octave convolution module processes the high-frequency part image and the low-frequency image obtained by downsampling according to the following formula, and respectively inputs the high-frequency part image and the low-frequency image into different convolution kernel channels of the improved octave convolution module, performs convolution operation according to the convolution weights obtained by training to obtain a high-frequency convolution feature map and a low-frequency convolution feature map, performs nearest neighbor interpolation upsampling on the low-frequency convolution feature map to obtain a low-frequency upsampled feature map with the same resolution as the high-frequency convolution feature map, and splices the high-frequency convolution feature map and the low-frequency upsampled feature map on different convolution kernel channels of the improved octave convolution module to output a fused feature map.

[0060]

[0061] Among them, X l represents the image input to the improved octave convolution module, X l+1 represents the output fusion feature map, (i, j) represents the pixel of the image (i, j), An image representing a high-frequency portion of an image input into the improved octave convolution module; Represents the low-frequency part of the image input to the improved octave convolution module, which is X l Obtained by downsampling through the maximum pooling layer in the improved octave convolution module; represents the convolution weight of the high-frequency part of the image, represents the convolution weight of the low-frequency image, a represents the component ratio of the low-frequency part of the image, upsimple represents upsampling, represents the convolution operation, Indicates a splicing operation; X l+1 (i, j) where (i, j)∈{1, 2, ...L l+1}, L l+1 For X l+1 Size, L l For X l The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size.

[0062] The high-frequency part of the image includes detailed information of the image, such as edges and contours; while the low-frequency part includes the overall information of the image, which is redundant. Feature maps of different scales are used to represent the high and low frequency parts of the image, that is, the high frequency part is represented by the original resolution, and the low frequency part is represented by low resolution, which can reduce spatial redundancy and computational complexity; its octave convolution module contains four channels: high frequency to high frequency, low frequency to low frequency, low frequency to high frequency, and high frequency to low frequency, which leads to many parameters of the octave convolution module and a relatively complex structure. Because the traditional convolution input and output have only one scale, it is necessary to construct multiple octave convolution modules when replacing the traditional convolutional neural network.

[0063] To address this problem, the present application improves the traditional octave convolution module. After the improved octave convolution module performs convolution operations on the high-frequency image and the low-frequency image obtained by downsampling, the output is a fusion feature map after the high-frequency and low-frequency parts are spliced. Therefore, the input of the next layer of the network has both low-frequency and high-frequency parts, so that the high-frequency and low-frequency communicate with each other and the input and output feature map sizes are of only one size. The number of parameters is the same as that of the traditional convolution, the structure is simpler, and the traditional convolution network can be directly replaced.

[0064] The batch normalization adjusts the feature map output by the improved octave convolution module to a standard Gaussian distribution, accelerates network convergence, and improves accuracy.

[0065] The dynamic activation function adaptively generates the optimal coefficient according to the batch-normalized fusion feature map, performs nonlinear mapping on the batch-normalized fusion feature map, and obtains a nonlinear mapping feature map; the dynamic activation function is a DyReLU function, which is shared across space and not shared across channels, and its calculation formula is:

[0066]

[0067] Among them, y i Represents the nonlinear mapping feature map of the i-th channel output, x i a represents the batch-normalized fused feature map of the i-th channel input; i and b i is the coefficient corresponding to the i-th channel, and the output is calculated by the auxiliary function θ(x).

[0068] like Figure 4 As shown in FIG. 1 , the traditional ReLU static activation function formula is: y=max{x,0}, where y represents the output of the function and x represents the input of the function.

[0069] When the input x of the function is a negative value, the output y of the function is 0, and the neuron will not be activated; when the input x of the function is a non-negative value, the output y of the function is equal to the input x, and the neuron will be activated; therefore, only some of the neurons of this function will be activated at the same time, making the neural network very sparse and improving computational efficiency, but the traditional ReLU activation function is static and the calculations for all input samples are indiscriminate.

[0070] like Figure 5 As shown, the DyReLU dynamic activation function used in this application includes an auxiliary function θ(x) and an activation function f θ(x) (x), where the auxiliary function θ(x) is used to calculate the coefficient a for the activation function i and b i , the activation function f θ(x) (x) is used to calculate the value of the nonlinear mapping feature map.

[0071] The auxiliary function θ(x) can adaptively generate the optimal coefficient according to the batch-normalized fused feature map to determine a suitable dynamic activation function, and then perform nonlinear mapping on the batch-normalized fused feature map, which can effectively improve the expressiveness of the model.

[0072] The maximum pooling layer downsamples the nonlinear mapping feature map output by the dynamic activation function to obtain the depth feature map.

[0073] In an optional embodiment, the improved octave convolutional neural network includes four improved octave convolution blocks and four maximum pooling layers, which are respectively a first octave convolution block, a first maximum pooling layer, a second octave convolution block, a second maximum pooling layer, a third octave convolution block, a third maximum pooling layer, a fourth octave convolution block and a fourth maximum pooling layer connected in sequence.

[0074] S3. Input the depth feature map into the adaptive pooling layer for downsampling to obtain an average pooling feature map.

[0075] Specifically, the adaptive pooling layer downsampling is an operation of taking the average of the input deep feature map. It has a global receptive field and no parameters, so it can achieve dimensionality reduction of the deep feature map without increasing parameters, and can also reduce the parameters of the fully connected layer to prevent model overfitting.

[0076] S4. Input the average pooled feature map into the fully connected layer and the softmax classification layer in sequence for classification and recognition to obtain the expression recognition result.

[0077] Specifically, the neurons in the fully connected layer map the average pooling feature map output by the adaptive pooling layer into a one-dimensional vector. The specific calculation formula of the fully connected layer for the average pooling feature map is:

[0078] y=xA T +b

[0079] Wherein, y represents the output vector of the fully connected layer, x represents the average pooled feature map input to the fully connected layer, A represents the weight vector of the fully connected layer, and b represents the bias vector. The neurons in the fully connected layer reduce the dimension of the average pooled feature map and output a one-dimensional vector including 7 scores.

[0080] Specifically, the softmax classification layer uses 7 neurons to fully connect the neurons of the fully connected layer, maps the one-dimensional vector output by the fully connected layer, and outputs 7 values ​​between 0 and 1. The sum of the 7 values ​​is 1. These values ​​represent the probability that the input face image belongs to this type of expression. Finally, the expression category corresponding to the neuron with the largest output value is selected as the result of the expression classification of the face image. The specific calculation formula of the softmax classification layer for the one-dimensional vector output by the fully connected layer is:

[0081] i=1,2,...,7

[0082] Where i represents the total number of categories of expression classification, which is 7, such as anger, disgust, fear, happiness, sadness, surprise and neutral; z i is the output of the softmax classification layer, indicating the probability that the input face image belongs to the i-th expression category; y is the input of the softmax classification layer, that is, the one-dimensional vector output by the fully connected layer.

[0083] In a preferred embodiment, before the average pooling feature map is input into the fully connected layer, the average pooling feature map is also input into the Flatten layer for flattening to obtain a one-dimensional vector; the one-dimensional vector is then input into the fully connected layer for mapping to obtain a one-dimensional vector including 7 scores; finally, the softmax classification layer is used for mapping to output 7 values ​​between 0 and 1, and the expression category corresponding to the maximum value is selected as the result of the expression classification of the facial image.

[0084] In an optional embodiment, if Figure 6 As shown, Figure 6 This is a model structure diagram of the facial expression recognition method provided in this embodiment.

[0085] In step S1, the face image of the input model is grayed and resized to obtain a face image with a pixel size of 100*100. Then, a 90*90 image is cropped from the 100*100 face image, and one of the five positions of the upper left corner, upper right corner, center, lower left corner and lower right corner of the face image is selected for cropping. Finally, the cropped face image is mirrored.

[0086] In step S2, the face image with 90*90 pixels obtained by S1 preprocessing is input into the improved octave convolution neural network. As shown in Table 1, the number of convolution kernels of the octave convolution module of the first octave convolution block is 64, the convolution kernel size is 3*3, the convolution step is 1, and the boundary padding is 1; the number of convolution kernels of the octave convolution modules of the second, third, and fourth octave convolution blocks are 128, 256, and 256 respectively, and the other parameters are consistent with the first octave convolution block; the pooling kernel size of the four maximum pooling layers is 2*2, and the convolution step is 2. The face image with 90*90 pixels obtained by S1 preprocessing is convolved by the four improved octave convolution blocks and downsampled by the four maximum pooling layers, and finally a depth feature map with 256 pixels of 5*5 is obtained.

[0087] Table 1 Model hierarchy

[0088] Network Layer Output Convolution kernel size Convolution stride Border Fill enter 90*90*1 - - - First octave convolutional block 90*90*64 3*3 1 1 First max pooling layer 45*45*64 2*2 2 0 First octave convolutional block 45*45*128 3*3 1 1 First max pooling layer 22*22*128 2*2 2 0 First octave convolutional block 22*22*256 3*3 1 1 First max pooling layer 11*11*256 2*2 2 0 First octave convolutional block 11*11*256 3*3 1 1 First max pooling layer 5*5*256 2*2 2 0 Adaptive Pooling Layer 2*2*256 - - - Flatten layer 1*1*1024 - - - Fully connected layer 1*1*7 - - -

[0089] In step S3, the depth feature map with 256 pixels of 5*5 obtained in S2 is input into the adaptive pooling layer for down-sampling, and an average pooling feature map with 256 pixels of 2*2 can be obtained.

[0090] Before step S4, the process also includes inputting the 256-pixel 2*2 average pooling feature map obtained in step S3 into the Flatten layer for flattening, and outputting a one-dimensional vector containing 1024 elements.

[0091] In step S4, the one-dimensional vector containing 1024 elements is input into the fully connected layer for mapping to obtain a one-dimensional vector containing 7 elements. Finally, the one-dimensional vector containing 7 elements is input into the softmax classification layer for mapping, and 7 values ​​between 0 and 1 are output, and the sum of the 7 values ​​is 1. These values ​​represent the probability that the input face image belongs to this type of expression, and the expression category corresponding to the neuron with the largest output value is selected as the result of the expression classification of the face image.

[0092] like Figure 7 As shown, Figure 7 A schematic diagram of the structure of a facial expression recognition device provided in an embodiment of the present application. The facial expression recognition device comprises:

[0093] A preprocessing unit 10, used for preprocessing the face image;

[0094] An octave convolutional neural network unit 11 is used to extract features from a preprocessed face image using an improved octave convolutional neural network to obtain a depth feature map of the face image; wherein the improved octave convolutional neural network comprises an improved octave convolution block and a maximum pooling layer connected in sequence, the improved octave convolution block comprises two octave convolution layers, and each of the eight convolution layers comprises an improved octave convolution module, batch normalization, and a dynamic activation function;

[0095] The improved octave convolution module divides the image input to the improved octave convolution module into a high-frequency part image and a low-frequency part image, the resolution of the high-frequency part image remains unchanged, and the low-frequency part image is input into the maximum pooling layer with a pooling kernel size of 2×2 and a step size of 2 in the improved octave convolution module for downsampling to obtain a low-frequency image with a resolution half of the input image; the improved octave convolution module processes the high-frequency part image and the low-frequency image obtained by downsampling according to the following formula, respectively inputs the high-frequency part image and the low-frequency image into different convolution kernel channels of the improved octave convolution module, performs convolution operation according to the convolution weights obtained by training to obtain a high-frequency convolution feature map and a low-frequency convolution feature map, performs nearest neighbor interpolation upsampling on the low-frequency convolution feature map to obtain a low-frequency upsampled feature map with the same resolution as the high-frequency convolution feature map, and splices the high-frequency convolution feature map and the low-frequency upsampled feature map on different convolution kernel channels of the improved octave convolution module to output a fused feature map;

[0096]

[0097] Among them, X l represents the image input to the improved octave convolution module, X l+1 represents the output fusion feature map, (i, j) represents the pixel of the image (i, j), An image representing a high-frequency portion of an image input into the improved octave convolution module; Represents the low-frequency part of the image input to the improved octave convolution module, which is X l Obtained by downsampling through the maximum pooling layer in the improved octave convolution module; represents the convolution weight of the high-frequency part of the image, represents the convolution weight of the low-frequency image, a represents the component ratio of the low-frequency part of the image, upsimple represents upsampling, represents the convolution operation, Indicates a splicing operation; X l+1 (i, j) where (i, j)∈{1, 2, ...Ll+1}, L l+1 For X l+1 Size, L l For X l The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size;

[0098] The batch normalization adjusts the fused feature map output by the improved octave convolution module to a standard Gaussian distribution;

[0099] The dynamic activation function adaptively generates the optimal coefficient according to the batch-normalized fusion feature map, performs nonlinear mapping on the batch-normalized fusion feature map, and obtains a nonlinear mapping feature map; the dynamic activation function is a DyReLU function, which is shared across space and not shared across channels, and its calculation formula is:

[0100]

[0101] Among them, y i Represents the nonlinear mapping feature map of the i-th channel output, x i a represents the batch-normalized fused feature map of the i-th channel input; i and b i is the coefficient corresponding to the i-th channel, which is calculated and output by the auxiliary function θ(x);

[0102] The maximum pooling layer downsamples the nonlinear mapping feature map output by the dynamic activation function to obtain the depth feature map;

[0103] The average pooling layer unit 12 is used to input the depth feature map into the adaptive pooling layer for downsampling to obtain an average pooling feature map;

[0104] The fully connected layer and softmax classification layer unit 13 is used to input the average pooling feature map into the fully connected layer and the softmax classification layer in sequence for classification and recognition to obtain the expression recognition result.

[0105] It should be noted that, when the facial expression recognition device provided in the above embodiment executes the facial expression recognition method, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the facial expression recognition device and the facial expression recognition method provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0106] This application proposes a method and device for facial expression recognition. The improved octave convolution module is used to convolve the preprocessed facial image, which reduces low-frequency redundancy and calculation without increasing the number of parameters, and improves the feature extraction capability; the DyReLU dynamic activation function is used to enhance the learning and expression capabilities of the model; the model is pre-trained on a large data set to further improve the model performance.

[0107] At present, many models in the field of facial expression recognition are improved based on classic network models. The experimental comparison results of the model of this application and the classic model are shown in Table 2. It can be found that our model is superior to the classic model in terms of both computational complexity and parameter quantity. The comparison with the current facial expression recognition method is shown in Table 3. The model of this application is pre-trained on a large data set: experiments are conducted on FER2013, FERPlus, and RAF-DB data sets. The weight of the trained model is only 10.6MB, and the accuracy rates are 73.20%, 89.58%, and 88.46%, respectively. Compared with many current methods, it has higher accuracy and lower computing resource consumption, which fully proves the effectiveness of the method of this application.

[0108] Table 2 Comparison with classic models

[0109]

[0110]

[0111] Table 3 compares current facial expression recognition methods

[0112] FER2013 FERPlus RAF-DB VGG+SVM 66.31 VGG16+Focal Loss 72.49 RAN+ResNet-18 88.55 86.90 SCN+ResNet-18 88.01 88.14 Proposed 73.20 89.56 88.46

[0113] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, and the present application is also intended to include such modifications and modifications.

Claims

1. A method for recognizing facial expressions, characterized in that: The following steps are involved: Preprocessing of face images; Using an improved octave convolutional neural network to extract features from the preprocessed face image to obtain a deep feature map of the face image; wherein the improved octave convolutional neural network includes an improved octave convolution block and a maximum pooling layer connected in sequence, the improved octave convolution block includes two octave convolution layers, and each of the eight convolution layers includes an improved octave convolution module, batch normalization, and a dynamic activation function; The improved octave convolution module divides the image input to the improved octave convolution module into a high-frequency part image and a low-frequency part image, the resolution of the high-frequency part image remains unchanged, and the low-frequency part image is input into the maximum pooling layer with a pooling kernel size of 2×2 and a step size of 2 in the improved octave convolution module for downsampling to obtain a low-frequency image with a resolution half of the input image; the improved octave convolution module processes the high-frequency part image and the low-frequency image obtained by downsampling according to the following formula, respectively inputs the high-frequency part image and the low-frequency image into different convolution kernel channels of the improved octave convolution module, performs convolution operation according to the convolution weights obtained by training to obtain a high-frequency convolution feature map and a low-frequency convolution feature map, performs nearest neighbor interpolation upsampling on the low-frequency convolution feature map to obtain a low-frequency upsampled feature map with the same resolution as the high-frequency convolution feature map, and splices the high-frequency convolution feature map and the low-frequency upsampled feature map on different convolution kernel channels of the improved octave convolution module to output a fused feature map; Among them, X l represents the image input to the improved octave convolution module, X l+1 represents the output fusion feature map, (i, j) represents the pixel of the image (i, j), An image representing a high-frequency portion of an image input into the improved octave convolution module; Represents the low-frequency part of the image input to the improved octave convolution module, which is X l The maximum pooling layer in the improved octave convolution module is downsampled; W l H Represents the convolution weight of the high-frequency part of the image, W l L represents the convolution weight of the low-frequency image, a represents the component ratio of the low-frequency part of the image, upsimple represents upsampling, represents the convolution operation, Indicates a splicing operation; X l+1 (i, j) where (i, j)∈{1, 2, ...L l+1 }, L l+1 For X l+1 Size, L l For X l The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size; The batch normalization adjusts the fused feature map output by the improved octave convolution module to a standard Gaussian distribution; The dynamic activation function adaptively generates the optimal coefficient according to the batch-normalized fusion feature map, performs nonlinear mapping on the batch-normalized fusion feature map, and obtains a nonlinear mapping feature map; the dynamic activation function is a DyReLU function, which is shared across space and not shared across channels, and its calculation formula is: Among them, y i Represents the nonlinear mapping feature map of the i-th channel output, x i a represents the batch-normalized fused feature map of the i-th channel input; i and b i is the coefficient corresponding to the i-th channel, which is calculated and output by the auxiliary function θ(x); The maximum pooling layer downsamples the nonlinear mapping feature map output by the dynamic activation function to obtain the depth feature map; Input the depth feature map into the adaptive pooling layer for downsampling to obtain an average pooling feature map; The average pooled feature map is sequentially input into the fully connected layer and the softmax classification layer for classification and recognition to obtain the expression recognition result.

2. A facial expression recognition method according to claim 1, characterized in that: The preprocessing of the face image includes graying, scaling, cropping and mirroring.

3. A facial expression recognition method according to claim 1, characterized in that: The improved octave convolutional neural network includes four improved octave convolution blocks and four maximum pooling layers, which are respectively the first octave convolution block, the first maximum pooling layer, the second octave convolution block, the second maximum pooling layer, the third octave convolution block, the third maximum pooling layer, the fourth octave convolution block and the fourth maximum pooling layer connected in sequence.

4. A facial expression recognition method according to claim 3, characterized in that: The number of convolution kernels of the octave convolution module of the first octave convolution block is 64, the convolution kernel size is 3*3, the convolution step is 1, and the boundary padding is 1; the number of convolution kernels of the octave convolution modules of the second, third, and fourth octave convolution blocks are 128, 256, and 256 respectively, and other parameters are consistent with the first octave convolution block.

5. A facial expression recognition method according to claim 3, characterized in that: The pooling kernel sizes of the four maximum pooling layers are all 2*2, and the convolution step sizes are all 2.

6. A facial expression recognition method according to claim 1, characterized in that: Before the average pooling feature map is input into the fully connected layer, the average pooling feature map is also input into the Flatten layer for flattening to obtain a one-dimensional vector and input into the fully connected layer.

7. A facial expression recognition device, characterized in that: include: A preprocessing unit, used for preprocessing the face image; An octave convolutional neural network unit, used to extract features from a preprocessed face image using an improved octave convolutional neural network to obtain a depth feature map of the face image; wherein the improved octave convolutional neural network comprises an improved octave convolution block and a maximum pooling layer connected in sequence, the improved octave convolution block comprises two octave convolutional layers, and each of the eight convolutional layers comprises an improved octave convolution module, batch normalization, and a dynamic activation function; The improved octave convolution module divides the image input to the improved octave convolution module into a high-frequency part image and a low-frequency part image, the resolution of the high-frequency part image remains unchanged, and the low-frequency part image is input into the maximum pooling layer with a pooling kernel size of 2×2 and a step size of 2 in the improved octave convolution module for downsampling to obtain a low-frequency image with a resolution half of the input image; the improved octave convolution module processes the high-frequency part image and the low-frequency image obtained by downsampling according to the following formula, respectively inputs the high-frequency part image and the low-frequency image into different convolution kernel channels of the improved octave convolution module, performs convolution operation according to the convolution weights obtained by training to obtain a high-frequency convolution feature map and a low-frequency convolution feature map, performs nearest neighbor interpolation upsampling on the low-frequency convolution feature map to obtain a low-frequency upsampled feature map with the same resolution as the high-frequency convolution feature map, and splices the high-frequency convolution feature map and the low-frequency upsampled feature map on different convolution kernel channels of the improved octave convolution module to output a fused feature map; Among them, X l represents the image input to the improved octave convolution module, X l+1 represents the output fusion feature map, (i, j) represents the pixel of the image (i, j), An image representing a high-frequency portion of an image input into the improved octave convolution module; Represents the low-frequency part of the image input to the improved octave convolution module, which is X l The maximum pooling layer in the improved octave convolution module is downsampled; W l H Represents the convolution weight of the high-frequency part of the image, W l L represents the convolution weight of the low-frequency image, a represents the component ratio of the low-frequency part of the image, upsimple represents upsampling, represents the convolution operation, Indicates a splicing operation; X l+1 (i, j) where (i, j)∈{1, 2, ...L l+1 }, L l+1 For X l+1 Size, L l For X l The size of , p is the border padding, f is the convolution kernel size, and s is the convolution step size; The batch normalization adjusts the fused feature map output by the improved octave convolution module to a standard Gaussian distribution; The dynamic activation function adaptively generates the optimal coefficient according to the batch-normalized fusion feature map, performs nonlinear mapping on the batch-normalized fusion feature map, and obtains a nonlinear mapping feature map; the dynamic activation function is a DyReLU function, which is shared across space and not shared across channels, and its calculation formula is: Among them, y i Represents the nonlinear mapping feature map of the i-th channel output, x i a represents the batch-normalized fused feature map of the i-th channel input; i and b i is the coefficient corresponding to the i-th channel, which is calculated and output by the auxiliary function θ(x); The maximum pooling layer downsamples the nonlinear mapping feature map output by the dynamic activation function to obtain the depth feature map; An average pooling layer unit, used for inputting the depth feature map into the adaptive pooling layer for downsampling to obtain an average pooling feature map; The fully connected layer and the softmax classification layer unit are used to input the average pooling feature map into the fully connected layer and the softmax classification layer in sequence for classification and recognition to obtain the expression recognition result.

Images