Face Expression Recognition Method Based on Broad Attention and Multi-Scale Fusion Mechanism

Abstract

The present invention belongs to the technical field of image processing, and particularly relates to a facial expression recognition method based on broad attention and multi-scale fusion mechanism. The recognition method includes: Step 1, preprocessing the obtained facial expression data set; Step 2, establishing a facial expression recognition model composed of a combination of a ResNet network and a capsule network; Step 3, establishing a broad attention and multi-scale fusion mechanism, embedding it into the ResNet network in Step 2, and using it as a feature extraction network; using the capsule network as a classification network; Step 4, inputting the facial expression image in Step 1 into the network in Step 3 to obtain an expression classification result. The present invention takes into account the background and texture information of the image, can capture more comprehensive feature information of the image; at the same time, it can blend the feature maps obtained from each layer of the ResNet to realize the interaction of feature information between the upper layer and the lower layer, so that the network can more fully extract facial expression features; the recognition accuracy is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a face expression recognition method based on broad attention and multi-scale fusion mechanism. Background Art

[0002] In the process of communication between people, facial expressions are the strongest signals for expressing inner emotions. In recent years, face expression recognition technology has been widely applied in the fields of public security, judicial interrogation, clinical diagnosis, human-computer interaction, etc., and has made great contributions to improving the quality of human life and the happiness index.

[0003] With the popularity of deep learning technology, face expression recognition technology has developed vigorously. However, although most methods use attention and feature fusion mechanisms, traditional attention fails to extract complete features of face expression images, and the fusion mechanisms used do not take into account the information blending between shallow features and deep features, resulting in a low accuracy of face expression recognition.

[0004] In the patent document with the application number "202310493454.6", a face expression recognition method and system based on feature fusion and attention mechanism are disclosed. The feature fusion method used is to achieve fusion by splicing the feature maps output by two intermediate layers, and the attention mechanism used is the multi-head attention mechanism Transformer. Its defect is that this feature fusion method only considers intermediate layer features and ignores the fusion of shallow and deep features. The multi-head attention mechanism used has problems such as consuming a large amount of memory and having poor processing ability for short-range context information, resulting in incomplete extraction of face expression features. Summary of the Invention

[0005] The present invention provides a face expression recognition method based on broad attention and multi-scale fusion mechanism to solve the problem in the prior art that the extraction and fusion of face expression features are not comprehensive, resulting in low accuracy of face expression recognition.

[0006] To achieve the above object, the technical solution of the present invention is: a face expression recognition method based on broad attention and multi-scale fusion mechanism, characterized by including the following steps:

[0007] Step 1: Preprocess the obtained face expression data set; divide the data set into a training set and a test set;

[0008] Step 2: Establish a face expression recognition model composed of a combination of a ResNet network and a capsule network;

[0009] Step 3: Establish a broad attention and multi-scale fusion mechanism, embed it into the ResNet network in Step 2, and use it as the feature extraction network; use the capsule network as the classification network;

[0010] Step 4: Input the facial expression image in Step 1 into the feature extraction network in Step 3 for feature extraction, and input the extracted feature map into the classification network to obtain the expression classification result.

[0011] Furthermore, the preprocessing of the facial expression dataset in the above Step 1 includes cropping the facial expression image to remove non-face regions; adjusting the image size to 224×224; and normalizing the adjusted image.

[0012] Furthermore, the broad attention in the above Step 3 includes four steps. First, divide the image along the spatial axis, perform average pooling and max pooling separately in the horizontal and vertical axis directions. Second, concatenate the results of max pooling and average pooling in the horizontal axis direction, and concatenate the results of max pooling and average pooling in the vertical axis direction. Third, finally concatenate the results obtained in the horizontal and vertical directions to obtain the weights of each pixel point in the image, and finally multiply it with the input feature map to obtain the final output feature map.

[0013] Furthermore, the above multi-scale fusion mechanism includes: First, connect the output feature map of the first layer of ResNet with the input, perform 1×1 convolution to ensure that the output feature map maintains its original size and number of channels. For the second layer, its input passes through a 7×7 convolution to expand its receptive field, and then merges with the output and convolution of the second layer, maintaining 128 output channels. Then, the output feature map of the second layer is short-circuited with the intermediate input of the third layer, so that the number of input channels of the third layer reaches 256. In the third layer, the input feature map passes through a 5×5 convolution, and then is connected to the output feature map, maintaining the number of channels at 256. The output feature map of the third layer is short-circuited with the intermediate output of the fourth layer, and the number of input channels of the fourth layer is 512. Finally, the input of the fourth layer passes through a 3×3 convolution, and then is connected and convolved with the output feature map to ensure that the number of channels of the final output feature map remains at 512.

[0014] Furthermore, in the above Step 4, the feature map extracted by the feature extraction network is input into the classification network for classification. After passing through the primary capsule layer in the capsule network, an 8-dimensional capsule vector of 32×6×6 is generated, and then a 16-dimensional capsule vector is obtained through dynamic routing. Finally, the probabilities of 7 types of expressions are output through the L2 norm.

[0015] Among them, the 7 types of expressions are anger, contempt, disgust, happiness, neutral, sadness, and surprise, a total of seven expressions.

[0016] Compared with the prior art, the advantages of the present invention are:

[0017] 1. The present invention designs an attention method that can simultaneously focus on the background and texture information of an image - broad attention. In this broad attention, worst case pooling and average pooling are used for the horizontal and vertical directions of the image at the same time, and the two are spliced. Unlike the existing attention method, it can extract from both the background and texture of the image in the horizontal and vertical directions, thereby enhancing the feature extraction capability of the network, taking into account the background and texture information of the image, and capturing more comprehensive feature information of the image.

[0018] 2. The present invention also designs a multi-scale fusion mechanism that can fuse the features extracted from each layer of ResNet. In the multi-scale fusion mechanism, an operation of fusing the features extracted from the shallow layer with the features extracted from the deep layer is adopted. The difference from the existing method is that the feature maps of the deep layer and the shallow layer are fused. Therefore, the feature maps obtained from each layer of ResNet can be fused to achieve the purpose of interaction between the feature information of the upper layer and the lower layer, so that the network can more fully extract the facial expression features.

[0019] 3. The present invention embeds broad attention and multi-scale fusion mechanisms into ResNet respectively. Broad attention is embedded into each residual block to ensure that each residual block can extract features to the greatest extent; the multi-scale fusion mechanism is embedded at the connection between every two layers to ensure that the deep and shallow layers of the network have the ability to interact with each other. The final classification of expressions is performed with the help of the viewpoint invariance of the capsule network. The combination of these two mechanisms improves ResNet's ability to extract facial expression features and integrate feature information, enhances the expression feature extraction stage, and ultimately improves the accuracy of the model in facial expression recognition.

[0020] 4. To verify the high efficiency of the present invention, it was verified on existing public datasets, and a recognition accuracy rate of 98.98% was obtained on the CK+ dataset, and a recognition accuracy rate of 98.46% was obtained on the Jaffe dataset, both of which are improvements compared to other traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the process of Example 1 of the present invention;

[0022] Figure 2 This is a schematic diagram of the overall network structure of Embodiment 1 of the present invention;

[0023] Figure 3 Schematic diagram of the structure of the broad attention mechanism proposed in the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of the multi-scale fusion mechanism proposed in the present invention;

[0025] Figure 5 Schematic diagram of the recognition accuracy of the present invention on the CK+ dataset;

[0026] Figure 6 Schematic diagram of the recognition accuracy of the present invention on the Jaffe dataset. Specific implementation manners

[0027] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0028] Embodiment 1. In this embodiment, the public face expression datasets CK+ and Jaffe are used. The CK+ dataset contains 327 images labeled with expression categories. Each image is a grayscale image with a fixed size of 256×256. Three peak frames are selected from the annotation sequence, and finally 981 facial expression images are obtained, including seven emotions: anger, contempt, disgust, fear, happiness, sadness, and surprise. The Jaffe dataset contains 213 facial expression pictures in 7 emotional states.

[0029] See Figure 1 , a face expression recognition method based on a wide attention and multi-scale fusion mechanism provided by the present invention, includes the following steps:

[0030] Step 1. Preprocess the obtained face expression dataset: adjust the face expression images to a size of 224×224, perform normalization processing on the images, with the mean value of [0.485, 0.456, 0.406] and the standard deviation of [0.228, 0.224, 0.225]; obtain the normalized images; after preprocessing, the interference of irrelevant noises can be removed and useful features can be enhanced.

[0031] Step 2. Construct a network model composed of a combination of a ResNet network and a capsule network:

[0032] Step 3. Establish a wide attention and multi-scale fusion mechanism, incorporate it into the ResNet network in Step 2, use it as a feature extraction network, and use the capsule network as a classification network;

[0033] See Figure 3 , the wide attention divides the image along the spatial axis, performs average pooling and max pooling respectively in the horizontal and vertical axis directions, splices the results of max pooling and average pooling in the horizontal axis direction, splices the results of max pooling and average pooling in the vertical axis direction, finally splices the results obtained in the horizontal and vertical directions to obtain the weights of each pixel point in the image, and multiplies it with the input feature map to obtain the final output feature map.

[0034] Specifically, the input of the wide attention module is:

[0035] X(N×C×H×W)(1)

[0036] The first step of processing in the horizontal and vertical directions of the spatial coordinate axes of the image is

[0037]

[0038] The second step of processing for the obtained horizontal and vertical directions is

[0039]

[0040] The third step of processing for the obtained horizontal and vertical directions is

[0041]

[0042] The feature maps obtained in the horizontal and vertical directions can be expressed as

[0043]

[0044] The final output can be expressed as

[0045] Output = attention_h × attention_w × Input(6)

[0046] Among them, N represents the Batch Size of the image input into the network, C represents the number of channels, and H and W respectively represent the height and width of the image.

[0047] See Figure 4 , the multi-scale fusion mechanism described in the present invention includes:

[0048] First, the output feature map of the first layer of ResNet is connected to the input, and 1×1 convolution is performed to ensure that the output feature map maintains its original size and number of channels. For the second layer, its input undergoes 7×7 convolution to expand its receptive field, and then it is merged with the output and convolution of the second layer, maintaining an output channel number of 128.

[0049] Then, the output feature map of the second layer is short-circuited with the intermediate input of the third layer, so that the input channel number of the third layer reaches 256. In the third layer, the input feature map undergoes 5×5 convolution, and then it is connected to the output feature map, maintaining the channel number of 256. The output feature map of the third layer is short-circuited with the intermediate output of the fourth layer, and the input channel number of the fourth layer is 512. Finally, the input of the fourth layer undergoes 3×3 convolution, and then it is connected and convolved with the output feature map to ensure that the channel number of the final output feature map remains at 512, thereby realizing feature fusion at different scales and different levels.

[0050] Step 4: Input the facial expression image in Step 1 into the feature extraction network in Step 3 for feature extraction. Input the feature map extracted by the feature extraction network into the classification network for classification. Generate an 8-dimensional capsule vector of 32×6×6 through the primary capsule layer in the capsule network, obtain a 16-dimensional capsule vector through dynamic routing, and obtain the probabilities of 7 types of expressions through the nonlinear function squash. Finally, take the expression category with the highest probability as the recognition result.

[0051] Among them, the squash function is:

[0052]

[0053] Among them, ||s j || is the modulus of the 16-dimensional vector s j When ||s j || is relatively large, the value of V i tends to 1, and vice versa tends to 0.

[0054] This step utilizes the view-point invariance of the capsule network to accurately distinguish whether the image is a normal facial expression image. Finally, calculate the modulus length of the obtained result vector and take the category with the longest length as the predicted expression category, that is, the classification result of the expression.

[0055] Verify the expression classification results obtained in the embodiment on the CK+ and Jaffe datasets. As Figure 5 、 Figure 6 shown, the recognition results obtained in this embodiment are 98.98% and 98.46% respectively, which are 6.34% and 6.15% higher than the recognition rates of directly using the ResNet combined capsule network on the CK+ dataset and the Jaffe dataset.

[0056] The above is only a preferred embodiment of the present invention, and does not impose any form of limitation on the present invention. Any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A facial expression recognition method based on a broad attention and multi-scale fusion mechanism, characterized in that, It includes the following steps: Step 1: Preprocess the obtained facial expression dataset; divide the dataset into a training set and a test set; Step 2: Establish a facial expression recognition model composed of a combination of a ResNet network and a capsule network; Step 3: Establish a broad attention and multi-scale fusion mechanism, embed it into the ResNet network in Step 2, and use it as a feature extraction network; use the capsule network as a classification network; Step 4: Input the facial expression images in Step 1 into the feature extraction network in Step 3 for feature extraction, and input the extracted feature maps into the classification network to obtain the expression classification result; The broad attention in Step 3 includes four steps. First, divide the image along the spatial axis, perform average pooling and max pooling respectively in the horizontal and vertical axis directions. Second, concatenate the results of max pooling and average pooling in the horizontal axis direction, and concatenate the results of max pooling and average pooling in the vertical axis direction. Third, finally concatenate the results obtained in the horizontal and vertical directions to obtain the weights of each pixel point in the image, and finally multiply it with the input feature map to obtain the final output feature map; The multi-scale fusion mechanism in Step 3 includes: First, connect the output feature map of the first layer of ResNet with the input, perform 1×1 convolution to ensure that the output feature map maintains its original size and number of channels; for the second layer, its input undergoes 7×7 convolution to expand its receptive field, and then merges with the output and convolution of the second layer, maintaining 128 output channels; then, the output feature map of the second layer is short-circuited with the intermediate input of the third layer, so that the input channel number of the third layer reaches 256; in the third layer, the input feature map undergoes 5×5 convolution, and then is connected with the output feature map, maintaining 256 channels; the output feature map of the third layer is short-circuited with the intermediate output of the fourth layer, and the input channel number of the fourth layer is 512; finally, the input of the fourth layer undergoes 3×3 convolution, and then is connected and convolved with the output feature map to ensure that the number of channels of the final output feature map remains at 512.

2. The face expression recognition method based on broad attention and multi-scale fusion mechanism according to claim 1, wherein: The preprocessing of the facial expression dataset in Step 1 includes cropping the facial expression images to remove non-face regions; adjusting the image size to 224×224; Normalize the adjusted images.

3. The face expression recognition method based on a broad attention and multi-scale fusion mechanism according to claim 1 or 2, characterized in that: In Step 4, input the feature maps extracted by the feature extraction network into the classification network for classification. After passing through the primary capsule layer in the capsule network, generate an 8-dimensional capsule vector of 32×6×6, then obtain a 16-dimensional capsule vector through dynamic routing, and then obtain the probabilities of 7 types of expressions through the nonlinear function squash. Finally, take the expression category with the maximum probability as the recognition result; Among them, the squash function is: where ||s j || is the norm of the 16-dimensional vector s j .

4. The face expression recognition method based on a broad attention and multi-scale fusion mechanism according to claim 3, characterized in that: The 7 types of expression categories are anger, contempt, disgust, happiness, neutral, sadness, and surprise.

Images