A facial expression recognition method based on the fusion of visual Transformer and convolutional network

Abstract

This invention belongs to the field of image classification in computer vision, specifically relating to a facial expression recognition method based on the fusion of visual Transformer and convolutional network. The method has the following features and includes the following steps: Step 1, preprocessing the image to be trained to obtain a preprocessed image; Step 2, inputting the preprocessed image into a model based on the fusion of visual Transformer and convolutional network for training, thereby obtaining the model's weight file. This model includes a convolutional module, an encoder, and an attention mechanism. The convolutional layer includes associating the positional information of image features; the encoder includes multiple residual modules, which use the encoder's input and final output as the encoder's final output. The encoder consists of multiple residual modules, and these outputs are used as inputs for the fusion attention mechanism; the fusion attention mechanism uses the output of the pooling layer as input to the attention mechanism module, and uses an adaptive attention mechanism to find different weight responses in the features of the input feature map, finally inputting it into the visual Transformer for training; Step 3, loading the model weight file, inputting the test facial expression image into the model to obtain the expression prediction result. Furthermore, the facial expression recognition cutting model of the present invention can better separate features between different categories, thereby improving the accuracy of the expression recognition model prediction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image classification of computer vision, and particularly relates to a facial expression recognition method based on fusion of visual Transformer and convolutional network. BACKGROUND

[0002] With the increasing development of computer technology, more and more technologies are applied to the society, and the country is advancing towards the information age, and informationization is driving the development of technology. Artificial intelligence is the product of this era, and artificial intelligence has begun to fully penetrate into people's daily life. Through the various conveniences brought by artificial intelligence, people can let computers replace human operation in some fields, reduce the time cost of people, and can more efficiently handle affairs, for example, facial recognition technology makes the society more information-based, and people will no longer have the trouble of carrying an ID card everywhere. Moreover, the most popular automatic driving field nowadays makes people have a deeper understanding of the concept of driving. Although this technology still has many deficiencies and needs to be improved, the application of this technology represents the transition of computer vision from the theoretical level to real life. It is the application of this technology to practical scenarios that has given rise to more and more computer vision technologies in real life.

[0003] Facial expression plays a leading role in people's daily life. Compared with text or action expression, expression can directly and effectively express human emotion, and most people also convey their emotions at the moment through changes in their own expressions. It can almost cover most people's emotional state in different scenarios. Therefore, expression is usually the main basis for judging emotion. Society is full of tragedies caused by emotions, such as people ending their lives because of too much pressure from life and work, and some people causing irreparable harm to others because they cannot bear the humiliation. These events are often preventable by calculating the emotional state of humans, quickly making judgments, and preventing dangerous events from occurring. The main purpose of expression recognition is to identify the expression of a face in a short time through some available information. The input required by expression recognition can be divided into static images and image sequences, and various expressions of humans are separated from these pictures. When computers reach the level of human understanding and judgment of expressions, computers can replace humans in judging expressions, and humans and machines can achieve better cooperation. With the rapid development of expression recognition technology, it can be applied to a very wide range of scenarios in the future. Many fields have already applied expression recognition technology to real-life scenarios. Artificial intelligence can capture customers' expression changes in real time through remote video monitoring and quickly make judgments about whether they are in a dangerous state. Expression recognition is also applied in school scenarios. Online education platforms identify students' expressions to analyze whether students are paying attention in class, and feedback the results to the teacher, so that the teacher can better manage the students. SUMMARY

[0004] The technical problem solved by the present application is that the number of expression categories in the training process of the facial expression recognition model is too small, resulting in a low recognition rate of a certain expression category after the network training is completed, and the feature map only contains local information without global information during the training process. The technical solution provided by the present application is as follows:

[0005] A facial expression recognition method based on the fusion of visual Transformer and convolutional network, comprising the following steps:

[0006] Obtain facial expression picture input samples and corresponding expression category labels; perform face detection and face alignment on the facial expression pictures;

[0007] Establish a network model that fuses visual Transformer and convolutional network, and input the facial expression pictures into the network model to obtain the facial expression recognition result;

[0008] Sheet

[0009] The facial expression features are extracted by inputting the picture into the model, and the expression features of the picture are obtained;

[0010] The proportion of the number of each category picture in the network model in all pictures is calculated;

[0011] The expression features are input into the classifier for classification, and the classification result is output;

[0012] According to the proportion of each expression category in the entire data set, the corresponding loss value is calculated;

[0013] Further, the facial expression picture sample and the corresponding category label specifically include:

[0014] The face position in the facial expression picture is obtained, and the face in the facial expression picture is detected to obtain the specific position of the face in the picture.

[0015] The face position obtained above is mapped to obtain the facial expression picture after face alignment.

[0016] Further, the network model combining the visual Transformer and the convolutional network is established, the facial expression picture is input into the model for training, and then the picture feature is obtained, which specifically includes the following:

[0017] A module combining the visual Transformer and the convolutional network is created, 4 modules are created, and the optimizer, learning rate, convolution kernel and other parameters in the model are initialized;

[0018] The expression picture is input into the network model combining the visual Transformer and the convolutional network, and the one-dimensional feature sequence output by the last layer of the visual Transformer is taken as the last feature sequence;

[0019] Further, the self-attention mechanism, namely the CBAM module, is added in the visual Transformer, and the calculation formula is as follows:

[0020]

[0021]

[0022] Where F is the input feature map, M c (·) is the channel attention function, is the multiplication of the corresponding element positions of the feature maps, M s (·) is the spatial attention function, and F″ is the output of the CBAM module, which is the feature map combining channel attention and spatial attention;

[0023] The channel attention calculation formula is as follows:

[0024] M c (F) = σ (MLP (AvgPool (F)) + MLP (MaxPool (F))),

[0025] where AvgPool (·) is to average pool the picture, MaxPool (·) is to maximum pool the picture, MLP (·) is a fully connected operation on the input feature map, and σ (·) is a sigmoid function;

[0026] The spatial attention calculation formula is as follows:

[0027] M s (F) = σ (f ([AvgPool (F) ; MaxPool (F)])),

[0028] where f (·) is a convolution operation on the feature map, and [AvgPool (·) ; MaxPool (·)] is to splice the feature maps output by maximum pooling and average pooling;

[0029] Further, the calculation input into the network model of the number of each category picture in all pictures accounts for a specific proportion, which includes the following:

[0030] The proportion of the number of each expression category in all pictures is different, which will directly affect the gradient calculation of the network model during training of a certain category, resulting in that some expression categories are not given the same or more attention during training, and finally leading to that the network has different tendencies to different categories after training.

[0031] Further, the expression feature is input into the classifier for classification, and the classification result is output. Its feature is that the expression feature obtained by the last layer of the Transformer is input into the Softmax function to obtain the class prediction confidence of the network for the picture, which represents the probability of the network for the picture belonging to each expression category.

[0032] Further, the corresponding loss value is calculated according to the proportion of each expression category in the entire data set, which includes the following:

[0033] The proportion of the expression category in all categories refers to the proportion in quantity, which solves the problem that the gradient of the expression category picture with a small proportion in the network training process has a small influence. The calculation formula is as follows:

[0034]

[0035] where N is the number of expression picture samples in each network training, M is the number of expression categories, and pij is the output value of the last layer of the network model for the i-th sample and the j-th expression category, W j is the proportion of the j-th expression category in the entire training set. The Loss value can effectively help the network to solve the problem of insufficient training of a small number of expression categories during the training process. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of the facial expression recognition method based on the fusion of the visual Transformer and the convolutional network constructed in the present application is shown.

[0037] Figure 2 A whole structure diagram of the expression recognition model based on the fusion of the visual Transformer and the convolutional network constructed in the present application is shown.

[0038] Figure 3 A PCS-Transformer structure diagram in the present application is shown.

[0039] Figure 4 A CS-Transformer structure diagram in the present application is shown.

[0040] Figure 5 A PCS-MHSA module structure diagram in the present application is shown.

[0041] Figure 6 A CBAM module whole structure diagram in the present application is shown. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and in detail in combination with the drawings in the embodiments of the present application. The described embodiments are only some of the embodiments of the present application.

[0043] The technical solutions of the present application to solve the above technical problems are:

[0044] Embodiment 1:

[0045] As shown in the expression recognition method of the present application, the steps include: Figure 1

[0046] Step 1: All samples are classified according to the file name, and then each expression picture is preprocessed, which is divided into face detection and face alignment. The specific method is to use the MTCNN network to detect the 5 key points of the face picture, obtain the positions of the 5 key points of the face, and then use the similarity transformation matrix algorithm to map the 5 key points to the center position area in the picture, so as to achieve the purpose of face alignment.

[0047] ​Step 2: Calculate the category proportion of the pictures obtained in step 1, which is the proportion of the number of pictures in each category to the total number of pictures in all categories. This proportion is used to calculate the loss value of the network model in the future;

[0048] Step 3: Create a network model based on visual Transformer and convolutional network, and then put the processed expression pictures into the network for training. The specific steps are as follows:

[0049] (1) Standardize the facial expression pictures after face alignment in step 1, set all expression pictures to 224x224, and normalize all pictures;

[0050] (2) Divide the processed data set into multiple sets, each set containing different categories of expression pictures, and convert all data in the set to tensor format for training the initialized network.

[0051] (3) Create a visual Transformer and convolutional network model, and load the network model parameters trained on a large data set into the network model to facilitate faster training in the future;

[0052] (4) Input the tensor set into the network model to extract facial expression features, and get the response expression features. The i-th expression feature sample gets the feature value x i =(x i1 ,x i2 ,…,x in ), where n=512, and each sample gets a 1x512 feature sequence;

[0053] (5) Input the expression features obtained in the previous step into the last fully connected layer of the neural network to get the classification information P i =(p i1 ,p i2 ,…,p iM ), where M is the number of categories, and each sample corresponds to seven values representing the confidence of belonging to seven categories;

[0054] Step 4: Calculate the seven category confidences obtained in step 3 and the category proportions obtained in step 2 respectively according to the following formula:

[0055]

[0056] The obtained loss value is fed back to the network model, and finally the weight file of the model is obtained.

Claims

1. A facial expression recognition method based on the fusion of visual Transformer and convolutional network, characterized in that, The method comprises the following steps: (1) obtaining face expression picture input samples and corresponding expression category labels; (2) face detection and face alignment are performed according to the face expression picture; (3) a network model combining visual Transformer and convolutional network is established, the face expression picture is input into the model to extract face expression features, and the expression features of the picture are obtained; wherein the network model comprises a visual Transformer module and a convolutional module, and a self-attention mechanism (CBAM module) is added in the visual Transformer module, and the specific calculation formula is as follows: (Q, K, V) = (XW q , XW k , XW v ), Wherein Q, K and V represent query, key and value tensors in multi-head self-attention; the further calculation formula is as follows: I i-1 = Seq2Image(X att ), P i = CBAM(I i-1 ), (Q, K, V) = (XW q , P i W k , P i W v ), wherein X att is the feature sequence output by the i-1th layer visual Transformer, Seq2Image(·) is a one-dimensional feature sequence to a two-dimensional feature map, I i-1 is a two-dimensional feature map, CBAM(·) is a self-attention mechanism for input feature map, which performs adaptive weight response on the channel and spatial dimensions of the feature map, P i is a one-dimensional sequence feature tensor obtained by the CBAM module from I i-1 ; (4) the proportion of the number of each category picture in the network model in all pictures is calculated; (5) the expression features are input into a classifier for classification, and a classification result is output; (6) the loss value corresponding to each expression category in the whole data set is calculated.

2. The method according to claim 1, wherein, The face expression picture samples and corresponding category labels specifically comprise: The face position in the face expression picture is obtained, the face in the face expression picture is detected to obtain the specific position of the face in the picture; The face position obtained above is mapped to obtain the face expression picture after face alignment.

3. The method according to claim 1, wherein, The network model combining visual Transformer and convolutional network is established, the face expression picture is input into the model for training, and then the picture features are obtained, which specifically comprise the following: A module combining visual Transformer and convolutional network is created, a total of 4 modules are created, and the optimizer, learning rate, convolution kernel and other parameters in the model are initialized; The expression picture is input into the network model combining visual Transformer and convolutional network, and a one-dimensional feature sequence output by the last layer of visual Transformer is taken as the final feature sequence.

4. The face expression recognition method based on the combination of visual Transformer and convolutional network according to claim 1 Emotion recognition method, characterized in that The self-attention mechanism in the visual Transformer, namely the CBAM module, has the following calculation formula: where F is the input feature map, M C (·) is a channel attention function, is the multiplication of corresponding element positions between feature maps, M S (·) is a spatial attention function, F" is the output of the CBAM module, which is the feature map combining channel attention and spatial attention. The channel attention calculation formula is as follows: Mc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))), Wherein AvgPool(·) is to average pool the picture, MaxPool(·) is to maximum pool the picture, MLP(·) is to perform full connection operation on the input feature map, and sigma(·) is a sigmoid function; The spatial attention calculation formula is as follows: Ms(F)=σ(f([AvgPool(F);MaxPool(F)])), Wherein f(·) is to perform convolution operation on the feature map, and [AvgPool(·);MaxPool(·)] is to splice the feature maps output by maximum pooling and average pooling.

5. The facial expression recognition method based on the fusion of visual Transformer and convolutional network according to claim 1, wherein the number proportion of each category picture in all pictures in the input network model is calculated, and specifically comprises the following: the number proportion of each expression category in all pictures is different, which directly affects the gradient calculation of the network model during the training process of a certain category, causing some expression categories to not receive the same or more attention during the training, and finally causing the network to have different tendencies for different categories after training. The expression features obtained by the last layer of Transformer are input into the Softmax function to obtain the class prediction confidence of the network for the picture, which represents the probability of the network for the picture belonging to each expression category.

6. The method according to claim 3, characterized in that 7. The facial expression recognition method based on the fusion of visual Transformer and convolutional network according to claim 1, wherein the loss value corresponding to each expression category is calculated according to the proportion of each expression category in the entire data set, and specifically comprises the following: the proportion of the expression category in all categories refers to the number proportion, which solves the problem that the gradient of the expression category picture with a small proportion has a small influence on the network during the training process; the calculation formula is as follows: ​ ​ where N is the number of facial expression image samples in each network training, M is the number of facial expression categories, p ij is the output value of the last layer of the network model for the i-th sample of the j-th facial expression category, W j is the proportion of the j-th facial expression category in the entire training set; the Loss value can effectively help the network to solve the problem of insufficient training of a small number of facial expression categories in the training process.

Images